On Representation Learning Under Class Imbalance
Authors: Ravid Shwartz-Ziv
Micah Goldblum
Yucen Lily Li
C. Bayan Bruss
Andrew Gordon Wilson
Venue: Submitted to ICLR 2023 (2023)
Keywords: Class Imbalance
Neural Networks
Representation Learning
Flatness
Self-Supervised Learning
Bayesian Learning

Abstract: Unlike carefully curated academic benchmarks, real-world datasets are often highly class-imbalanced, involving training and test sets which contain few examples from certain minority classes. While there is a common understanding that neural network generalization is negatively impacted by imbalance, the source of this problem and its resolution are unclear.  Through extensive empirical investigation, we study foundational learning behaviors for various models such as neural networks, gradient-boosted decision trees, and SVMs across a range of domains and find that (1) contrary to conventional wisdom, re-balancing the training set to include a higher proportion of minority samples degrades performance on imbalanced test sets; (2) minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization.  Motivated by the observation that re-balancing class-imbalanced training data is ineffective, we show that several existing techniques for improving representation learning are effective in this setting: (3) self-supervised pre-training is insensitive to imbalance and can be used for feature learning before fine-tuning on labels; (4) Bayesian inference is effective because neural networks are especially underspecified under class imbalance; (5) flatness-seeking regularization pulls decision boundaries away from minority samples, especially when we seek minima that are particularly flat on the minority samples’ loss.

Under review as a conference paper at ICLR 2023
ON REPRESENTATION LEARNING UNDER CLASS IMBALANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Unlike carefully curated academic benchmarks, real-world datasets are often highly class-imbalanced, involving training and test sets which contain few exam- ples from certain minority classes. While there is a common understanding that neural network generalization negatively impacted by imbalance, the source of this problem and its resolution are unclear. Through extensive empirical investiga- tion, we study foundational learning behaviors for various models such as neural networks, gradient-booster decision trees, and SVMs across range of domains and find that (1) contrary to conventional wisdom, re- balancing the training set to include higher proportion of minority samples degrades performance on imbal- anced test sets; (2) minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization. Motivated by the observa- tion that re- re-balancing ilass-imbalanced training data is ineffective, we show that several existing techniques for improving representation learning are effective this setting: (3) self-supervised pre-training is insensitive to imbalance and can be used for feature learning before fine-tuning on labels; (4) Bayesian inference is effective because neural networks are especially underspecified under class im- balance; (5) flatness-seeking regularization pulls decision boundaries away from minority samples, especially when we seek minima that are particularly flat the minority samples' loss.
INTRODUCTION
In real-world data collection scenarios, some events are common while others are exceedingly rare. For example, only miniscule proportion of credit card transactions are fraudulent, and most cancer screenings come back negative, As a result of this property, machine learning systems are routinely trained and deployed on class-imbalanced data where relatively few samples are associated with certain minority classes, while majority classes dominate the datasets. Nonetheless, the vast majority of works exclusively consider class balanced benchmarks (LeCun, 1998; Krizhevsky, 2009; Deng et al., 2009), including both foundational literature which seeks to understand how and why machine learning algorithms operate as well as applied methodological literature.
In this work, we conduct an exploration, on various machine learning approaches including neural networks, gradient-boosted decision trees, and SVMs, of what makes learning under class imbalance so difficult and the associated implications for best practices in such scenarios. Many of the widely referenced methods for remedying class-imbalance problems rely on modifying how the training data is sampled, such as oversampling or SMOTE (Chawla et al., 2002) and have been shown to be ineffective for neural networks (Buda al., 2018).
A common assumption underpinning these sampling methods is that lcarning under class imbalance pathological, perhaps even defaulting to predicting only the majority class on all inputs when imbalance is sufficiently severe, so we must intervene by simulating balanced training. An effect of sampling from minority classes disproportionately more often is that more signal from those classes is injected into model updates, helping to fit the otherwise rarely seen minority samples. To tease out exactly why oversampling is ineffective, we begin by studying the relationship between imbalances seen at train and test time, and we investigate whether poor generalization under really be explained by failures of optimization. We find that while minority samples are hard to fit, this optimization phenomenon has little explanatory power regarding generalization as fitting them
can

---

Under review as a conference paper at ICLR 2023
does not affect performance. Additionally, find that the one hand, re-balancing training data to include more minority samples can negatively impact generalization under imbalanced testing, but on the other hand, gathering more majority samples to increase the size of the dataset degrades generalization as well.
Following our investigation into the role of dataset imbalances in generalization, we show why self-supervised learning (SSL), Bayesian inference, and flatness-seeking regularizers are particu- larly well-suited for deep learning in class-imbalanced settings. Self-supervised learning algorithms are less sensitive to the proportion of samples in various classes as they do not make use of label information, so we can learn better feature representations via SSL before fine-tuning even on the same data but with labels. Previous works have found that the number of high singular values of the Hessian related to the number of classes (Sagun et al., 2017; Papyan, 2020), but these works were conducted strictly on balanced data. In examining such properties on imbalanced datasets, we observe that neural networks trained in such settings are significantly more underdetermined by the data. In cases where many parameter settings, and induced functions, are compatible with the data, Bayesian Neural Networks (BNNs) can represent our uncertainty for improved accuracy (Wilson & Izmailov, 2020; Shwartz-Ziv et al., 2022), and we find that this capability of BNNs is especially advantageous in the flass-mbalanced regime. Finally, whereas neural network decision bound- aries tend to hug minority samples in order expand the margins from majority data points which occur more frequently in training data, we can counteract this behavior with Sharpness- imization (SAM) (Foret et al., 2020), and we can further improve margins with respect to minority samples by increasing flatness on their corresponding loss functions. In summary, our work ques- tions the motivation of orthodox sampling methods and proposes new directions by which improved representation learning can benefit classifiers in class- cass-imbalancee settings.
Min-
2 RELATED WORKS
A long line of research has been conducted on imbalanced classification. There are several general approaches to address this problem: (1) Re-sampling the data- - In early ensemble learning studies, boosting and bagging algorithms were adjusted to take account of imbalanced data by re- re-sampling. Traditionally, re-sampling involves oversampling minority class samples by simply copying them (Guo & Viktor, 2004; Chawla et al., 2002; Han et al., 2005), or undersampling majority classes by removing samples (Drummond et al., 2003; Hu et al., 2020; Ando & Huang, 2017; Buda et al., 2018), so that minority and majority class samples appear equally frequently in the training process. (2) Loss re-weighting: Loss re-weighting assigns different weights to majority and minority classes, thus reducing optimization difficulty under class imbalance (Cui et al., 2019; Huang et al., 2019a). For instance, one may scale the loss by inverse class frequency He & Garcia (2009) or re-weight it using the effective number of samples Cui et al. (2019). As an alternative approach, one may focus on hard examples by down-weighing the loss of well-classified examples (Lin et al., 2017) or dynamically re-scaling the cross-entropy loss based on the difficulty of classifying a sample (Ryou et al., 2019). Bertsimas et al. (2018) propose to encourage larger margins for rare classes, while Goh & Sim (2010) learn robust features for classify minority classes using class- caaussuncerrainty information which approximates Bayesian methods. 3) Two-stage fine-tuning and meta-learning approaches: Two-stage methods separate the training process into representation learning and classifier learning (Liu et al., 2019; Ouyang et al., 2016; Kang al., 2019; Bansal al., 2021). In the first stage, the data is unmodified, and no re-sampling or re-weighting is used to train good representations. In the second stage, the classifier balanced by freezing the backbone and fine-tuning the last layers with re-sampling techniques or by learning to debias the confidence. These methods assume that the bias towards majority classes exists only in the classifier layer or that tweaking the classifier layer can correct the underlying biases.
Several works have also inspected representations learned under class imbalance. Kang et al. (2019) find that representations learned on iaaalaimall training data via supervised learning perform better when the linear head is fine-tuned on balanced samples. Yang & Xu (2020) instead examine the effect of self- and semi-supervised training on imbalanced data and conclude that imbalanced la- bels are significantly more useful when accompanied by auxiliary data for semi cumi--sperriieed learning. Kotar et al. (2021); Yang& & Xu (2020); Liu et al. (2021) make the observation that self- pre- pre-trainee training is insensitive to imbalance in the upstream training data. These works study SSL pre- training for the purpose of transfer learning, sometimes using linear probes to evaluate the quality of
2

---

Under review as a conference paper at ICLR 2023
representations. Inspired by their observations, find that SSL pre-training and then fine-tuning on the same exact class- caaas-imbalanced dataset can significantly improve generalization.
3 EXPERIMENTAL, SETUP
Class-imbalance ratio. Let r denote the ratio between the number of samples in the rarest class to the number of samples in the most frequent class. In the case where 1, the dataset is fully balanced, whereas r = 0.1 indicates that there are 10 times as many samples in the majority class as in the minority class. In this paper, will construct and investigate both rrain and fest sets with varying imbalance ratios. It is important to note that if the train dataset ratio is fixed and the test ratio is varied, then we may train a single model but evaluate it on a variety of test sets. On the other hand, the test ratio is fixed and the training ratio is varied, we must train many models. We will vary our setup along both these axes.
Datasets. For experiments with neural networks, we use CIFAR-I0 (Krizhevsky, 2009) as well as a binary variant which we simply use two of the CIFAR-10 classes. For tabular data, we use the Adult dataset and Forest Cover dataset from the UCI Machine Learning Repository (Dua & Graff, 2017). We adopt the imbalanced datasets of Liu et al. (2019) which imbalance classes via exponential distribution, closely reflecting natural long-tailed class distributions.
Models. We use ResNet-34 (He et al., 2016) on CIFAR-10. We use XGBoost (Chen & Guestrin, 2016) and SVM on tabular datasets. For supervised pre-training, we follow the standard protocol of (Kang al., 2020). For self-supervised pre-training, we use SimCLR (Chen et al., 2020) and fine-tune as in (Kotar et al., 2021). For each evaluation in experiments, we run five seeds and report the mean along with one standard error. Appendix A. contains additional details.
4 THE ROLE OF IMBALANCED DATA IN GENERALIZATION AND OPTIMIZATION
In this section, first explore the impact of training set imbalances generalization across var- ious imbalanced testing scenarios, finding that re-balancing training data actually harms accuracy under imbalanced testing, across multiple ML models. We then find that severe class imbalance makes fitting minority samples difficult, but solving this problem is irrelevant to generalization in neural networks. Re-balancing the training data and fitting minority samples are both primary ef- fects of oversampling, so these observations will inspire us to look past sampling algorithms in our subsequent search for improved training methods for neural networks.
4.1 THE RELATIONSHIP BETWEEN TRAIN-TIME AND TEST-TIME IMBALANCE
The literature on training routines for class imbalance in machine learning is filled with methods designed for scenarios in which training data is highly imbalanced, but testing data is balanced. However, in industrial applications of machine learning, data encountered during deployment is typically also imbalanced. Therefore, we disentangle training and testing balances and investigate how sensitive models are to discrepancies between the two. This study may be particularly impor- tant if one is considering collecting training data for downstream application. Should we gather training data with the same balance as we anticipate during testing? If the data we encounter during deployment more or less balanced than the training data we gathered, how worried should we be?
A priori, the optimal train set balance is unclear. On the one hand, one might want to train on the same distribution as they anticipate testing on. On the other hand, previous studies have suggested that imbalanced training data impedes representation learning and have suggested sampling proce- dures for simulating more balanced training (Gosain & Sardana, 2017). To answer this question, we train on datasets with wide range of imbalance ratios and evaluate each trained model on a variety of testing ratios. We illustrate three scenarios in Figure 1: (1) identical training and testing ratios, (2) balanced training, and (3) the training ratio with the lowest test error (optimal training ratio). We see that training on data with the same imbalance as the testing data is typically superior to training on balanced data, and the two strategies only approach equal performance when the testing data
3

---

Under review as a conference paper at ICLR 2023
Equal Train/Tor Ratio
Balanced Training
Best Train Ratio
10
10'
1044
14
10
16
Test Laror
10
Test Eavor
10'
10"
1g
10'
10
16
10
10
10'
10
10"
10'
10"
10"
10"
Test Ratio
Test Ratio
Test Ratio
(a) CIFAR- 10 Dataset
(b) XGBoost on Adult Dataset (c) SVM on Forest Cover Dataset
Figure I: Training on imbalanced data is optimal for imbalanced testing scenarios. Test accu- racy as a function of the test dataset ratio for different training setups. Error bars correspond to one standard error over trials.
becomes balanced. Moreover, training on data with the same imbalance as the testing data nearly optimal, so there is very little to gain by balancing the training set.
To determine the optimal training distribution, plot the best train ratio versus the best test ratio in Figure 2. In this figure, if there is a perfect match between the training and testing distributions, we would expect points lic on the diagonal. Indeed, the points are close to the diagonal, indicating that it is best to train with a very similar ratio to that of the test dataset, especially for highly imbalanced testing scenarios. Interestingly, when the optimal training distribution is not exactly the same as the testing distribution, it is usually very close to it and generally more balanced than the testing distribution. Even cases where the best ratio for training is more balanced, there minimal difference in test error between the best ratio for training and the test-equivalent ratio for training. This phenomena is consistent across different training methods (including oversampling training Drummond et al. (2003) and loss re- ne-weighting Cui et al. (2019)). This observation calls into question oversampling routines which attempt to improve performance by re-balancing the training data.
Training with a train set class imbalance similar to that of the test set is near optimal across the ML models and datasets we consider.
Takeaway:
Ratio
10
Best Train Ratio
40"
Bost Train Ratio
10
0s0s00eeis Saieso
0*0...a
10:2
30
10
.*...
Stadard Reweighting
10
10
XGBoost
10
MGBoost
Ovensampling
10
10
10"
10
10"
10"
10
10"
10"
Test Ratio
Test Ratio
'ess Ratio
(a) CIFAR-10 Dataset
(b) Adult Dataset
(c) Forest Cover Dataset
Figure 2: The optimal train dataset ratio is very close to the test dataset ratio. Optimal train imbalance ratio function of test imbalance ratio for various datasets and models.
4

---

Under review as a conference paper at ICLR 2023
Naral Network
XGBBoss SVM
92
3610
40
Best Number of
Majority Samples
l0"
Test Eroor
20
g
10
XGBooot
ee
10'
1n"
10'
10' Number of Minority Samples
Number of Majority Samples
Number of Majority Samples
(a) Binary CIFAR-10
(b) Adult Dataset
(c) Adult Dataset
Figure 3: The potentially destructive effects of adding majority class data. In (a) and (b), we fix the number of minority samples to be 500 and vary the number of majority samples. In (c), we plot the number of majority samples that gives us the lowest test error against the number of minority samples. We that increasing the number of majority samples degrades performance. Error reported on balanced test set.
4.2 WHEN MORE DATA DEGRADES PERFORMANCE
In the previous section, fixed the total number of training samples and saw that models per- form best when trained on similar data distribution to their testing data. However, in practice, a practitioner likely will not have precise control over the data they collect. Will collecting addi- tional samples always help performance? If not, practitioners must be cautious when collecting new data that its balance matches their testing data or else the additional data could result in even worse models. Instead of fixing the total number of samples and varying their class ratio, we now fix the number samples from the minority class and vary the number of others.
In Figure 3, we see that increasing the number of samples from the majority class, both for neural networks (Figure 3a) and XGBoost (Figure 3b), initially boosts performance on a balanced test set. Nevertheless, in both cases, the performance reaches an optimum before the ever increasing imbalance in training data eventually degrades test accuracy. Thus, adding training data can be helpful, even without considering the balance of the additional data, but if we add enough samples, we need to be careful not to cause too sharp a mismatch between training and testing distributions. Notably, the optimal training set ratio is nearly balanced, matching the test set, allowed to gather extra samples from one class without having to forego samples from another.
when
10a
10
Majority Class
Minority
Train Error
Train Error
10
XGBoost Majority Class
XGBoost Minority Class
Majority
10
SVM Minority Class
10
0
10
10"
10"
Train Ratio
Train Ratio
(a) ResNet on CIFAR-10
(b) XGBoost and SVM on Adult
Figure 4: Standard training routines fail to fit minority samples. Error bars correspond to one standard error over trials

---

Under review as a conference paper at ICLR 2023
10
10'6
Test Error
Test Error
10.
10
10
Regular Oversampling
10th
Overfitting
10
10
10"
10
10 2
10 -1
Train Ratio
Train Ratio
(a) ResNet Test Error on Binary CIFAR-10
(b) XGBoost Test Error on Adult
Figure 5: Overfitting with a low learning rate or oversampling does improve neural net- work generalization. Oversampling is helpful for XGBoost. Error reported on a balanced test set.
4.3 MINORITY DATA IS HARD TO FIT, BUT FITTING IT DOES NOT HELP
We investigate optimization imbalanced data. Does the classifier correctly label minority training points? Our goal is to determine if low performance under class imbalance can be attributed poor optimization. This question will inform future exploration of methods for training on imbalanced data as will tell us what underlying problem needs solving.
In imbalanced training, the vast majority of the gradient signal during training comes from majority class samples since they are seen much more often, making it hard to fit minority samples as shown in Figure 4. In order to re-balance imbalanced training data, practitioners employ various methods, including oversampling and undersampling. These methods sample minority data more often, thus boosting their contribution to the gradient and helping fit these minority samples. We confirm in Appendix A.2 that such sampling methods indeed help to fit minority samples across models. But the inability to minority samples culprit responsible for worse test accuracy?
To answer the above question, we train our model using two additional methods: (1) oversampling minority class, and (2) overfitting the training examples using large number of epochs and small learning rate. Both methods successfully fit minority and majority classes. Nonetheless, they fail to improve neural network test performance in contrast to XGBoost which benefits from in Figure 5. We include additional details and results for neural networks and XGBoost, as well as results on imbalanced test sets, in Appendix A.2. light of this study, while minority samples are hard to fit, will search elsewhere for improving neural network generaliza- tion subsequent sections.
oversampling,
Takeaway: Since minority samples are rarely seen in training, they hard to fit. Nonethe- less, strategies such as oversampling which fit these samples do not improve generalization.
LEARNING WITH SUPERVISED FINE-TUNING FOR ROBUST FEATURE LEARNING
COMBINING
Self-supervised learning (SSL) has recently achieved great success for representation learning in computer vision, NLP, and tabular data (Chen et al., 2020; Kenton & Toutanova, 2019; Somepalli et al., 2021). SSL pre-trained networks often exhibit even more transferable representations than their supervised counterparts (Grill et al., 2020). Liu et al. (2021) notes that self-supervised pre- training training strategies for transfer learning are more robust to upstream imbalance than super- vised pre-training However, many use-cases for deep learning pre- pre-training training datasets. We thus propose two-step procedure in which we first perform SSL pre- training and then supervised fine-tuning, all on the same imbalanced dataset. By first learning
accompanied by massive

---

Under review as a conference paper at ICLR 2023
feature extractor via SSL, which is insensitive to class imbalance, improve the quality of features and as a result generalization to
We use SimCLR (Chen et al., 2020)
10"
for self-supervised pre-training and try two fine-tuning routines: (1) fine-tune all layers in an end-to-end fashion and (2) train only fully-
Minority Test Error
connected layer on top of the fixed SSL feature extractor. We train on CIFAR- 10 using wide range of class-imbalance ratios reporting ac- curacy on balanced test set, and we compare to three baselines cluding standard supervised learn- ing, oversampling, and SimCLR Im- ageNet pre- traininngg The latter base- line uses an extra one million training samples for pre-training, which may not be available for a particular data domain in practice, but will serve as an aspiration for representation learn- ing. Figure 6, we see that our step procedure which uses SSL pre-training on the small imbalanced CIFAR-10 dataset achieves almost as high performance as ImageNet pre-training and far superior performance standard su- pervised learning and oversampling, even when the training set is relatively balanced. See Appendix Appendix A.2 for an identical experiment but where testing data has the same imbalance as training data. We similar trend there, namely that SSL pre-training improves generalization significantly.
10'
Regular Training
Over- Over-Sample
SSL ImageNet
SSL Train Data
SST. Linear Fvaluation
10
10
10
10
Test Ratio
Figure 6: SSL pre-training and end-to-end fine-tuning improves representation learning. Test error reported on balanced test sets for neural networks binary CIFAR-IO. Error bars correspond to one standard error over 5 trials
Takeaway: A two-step process of (a) SSL pre-training on the imbalanced training set and (b) supervised fine-tuning on the same data can boost accuracy across imbalance ratios.
UNDERSPECIFICATION AND BAYESIAN INFERENCE
Expressive models such as neural networks are often capable of representing numerous functions which are each compatible with the training data. Standard neural network training protocols only choose one of these solutions. However, we can better represent this uncertainty using an ensemble. Bayesian Neural Networks (BNNs) approach this problem by estimating the average prediction over all parameters weighted by their posterior probabilities conditioned on the training data, p(y|2, D) p(yla (z, w u)p(w] dd:. This approach particularly effective when the model underspecified by the data, in other words when there are many solutions that each achieve a high likelihood (Wilson & Izmailov, 2020). In this section, we see that underspecification particularly strong on class-imbalanced data so that BNNs are especially useful in this setting.
FLATNESS-SEEKING REGULARIZATION PULLS DECISION BOUNDARIES AWAY FROM MINORITY SAMPLES
number of works have found that the number of high singular values of the loss function Hessian is equal to the number of classes (Sagun et al., 2017; Papyan, 2020). Papyan (2020) notes that the theoretical underpinnings this connection may depend on the training data being class balanced. We thus measure the 10 leading singular values of the Hessian models trained balanced or highly imbalanced CIFAR-10 10 data, where the loss function is evaluated on the models' respective training data. We observe in Figure 7a that imbalanced data leads to lower singular values. We also perturb the parameter vector in random directions Figure 7b using filter nommaaiamaiinnionn (Huang al., 2019b) normalize distance measurements, and that the loss increases slightly slower

---

Under review as a conference paper at ICLR 2023
10
50/50 Ratio
Singular Value
20
99/1 Ratio
Change in Loss
10.5
15
10
10
5
0
-0.2
0.2
n-th Largest
Perturbation (Filter Normalized)
(a) Ten largest singular values of the Hessian.
(b) Loss minima are flatter under imbalanced data.
Figure 7: Loss minima are flatter under imbalanced data. Models trained on CIFAR-10.
on imbalanced data. Note that the volume of basin scales with its radius raised to power equal to the dimension of the space. As a neural network contains millions of parameters, a very small increase in basin radius will therefore lead massive increase in volume (Huang al., 2019b). Flat minima have the property that their parameters can be perturbed without unfitting the data, indicating that there are many solutions, even locally, which are all consistent with the training samples. But which of these solutions should we choose? By using Bayesian inference methods, we need not commit to only single one.
We now try two such Bayesian inference proce- dures. The Laplace approximation (Daxberger et al., 2021) fits a Gaussian distribution with diagonal covariance to the posterior so that lower curvature, which we observed above imbalanced datasets, will induce higher vari- ance. SGHMC (Chen al., 2014) is Markov chain Monte Carlo sampler which instead in- troduces noise to the training procedure so that parameters spend more time in places with higher probability under the posterior and sam- ples checkpoints periodically. We see in Fig- 8 that such methods confer especially large boosts in accuracy on class-imbalanee training data with virtually no additional training cost.
2.8 10
Laplace
Regular
SGHMC
Test Error
2 10'
2x10'
10"
10"
Train Ratio
Figure 8: Bayesian inference improves perfor- mance, especially on imblanced training data. Error reported on balanced CIFAR-10 test set.
Takeaway: Under class imbalance, models are particularly underspecified by the data. We can reflect uncertainty over solutions with Bayesian inference, leading to increased accuracy.
Sharpness-Aware Minimization (SAM) (Foret al., 2020) is optimizer for finding flat minima of the loss function which often generalize better than those found by SGD. To this end, SAM entails an inner ascent step followed by descent step which finds parameters such that ascent step minimally increases the loss. Huang et al. (2019b) connect flat minima to wide margin decision boundaries. By plotting the decision boundaries of small multi-layer perceptron on a toy 2D dataset in Figure 9, we see small margins surrounding minority class samples, and SAM expands these margins supporting the intuition of Huang et al. (2019h). Following this observation, we employ SAM on our CIFAR-10 10 setup and find that SAM especially improves generalization on inaaaiimbllaceed training data in Figure 10. In light of this result, we now further increase flatness specifically minority class loss terms by increasing the ascent step size in SAM's 's inner loop. We see Figure 10 that this adaptation can yield even greater performance boosts. See Liu et al. (2021) for adaptation of SAM which, rather than increasing the size of ascent steps minority sample loss, instead performs superviee pre- pre-traime training where the ascent step size is related to the density of examples estimated via kernel density estimation.
8

---

Under review as a conference paper at ICLR 2023
Decision boundaries after regular training
(b) Decision boundaries after SAM training
Figure 9: Flatness seeking regularization pulls decision boundaries away from minority sam- ples. Experiments conducted on toy 2D dataset paired with MLP architecture.
DISCUSSION
In this work, we first examined the effects of class-imbalanced data on optimization and gen- cralization. The results of these experiments indicated that oversampling solves exactly the wrong problems, namely re-balancing the train- ing data is not helpful, and fitting minority sam- ples does not improve generalization. The ob- servation that adding additional majority class samples actually hurts performance raises critical open problem: how can we best har- ness extra majority class samples? Follow- ing the above study, we suggest three existing class-
2,8 10
Test Error
2,4 10
Regular
10
SAM
Asymnmetric SAM
2
Train Ratio
methods for improving performance imbalanced data without ad-hoc interventions specific imbalance: (1) since SSL is insensi- tive to imbalance, pre-tree vee-training followed by end- to-end fine-tuning can result enhanced repre- sentations; (2) due to amplified underspecifica- tion, Bayesian inference is particularly effective on class- -inasaleallneeed problems; (3) sharpness- minimization can pull decision boundaries away from minority samples. Given the pervasiveness imbalanced data in real-world applications, the success of these simple existing methods should redirect our attention away from re-sampling meth- ods and towards better feature representations.
Figure 10: Sharpness-Aware Mininization (SAM) is particularly effective on imbalanced training data. "Asymmetrie SAM" denotes the version where the inner loop ascent step size is in- creased for minority samples. Error reported on balanced CIFAR-10 10 test set.
REFERENCES
Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 770-785. Springer, 2017.
Arpit Bansal, Micah Goldblum, Valeriia Cherepanova, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Metabalance: High fiigh-pefformacce neural networks for class class-imbal imbalanced data. arXiv preprint arXiv:2 1xX.:://06009643 2021.

---

Under review as a conference paper at ICLR 2023
Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathemat- ical Programming, 167(2)223522)) 292, 2018.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-222 216:249-259, 2018.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yin Cui, Menglin Jia, Tsung- Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number samples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9268-99699999 9277-927..20 2019.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace refux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089-20103, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-244220 245. leee, 2009.
Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. In Workshop on learning from imbalanced datasets I1, volume 11, pp. 1-8. Citeseer, 2003.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL WRLhhttpp//archive ics. uci. edu/ml.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2020.
Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research, 58(4-part-1):022-99) 2010.
Anjana Gosain and Saanchi Sardana. Handling class imbalance problem using oversampling tech- niques: A review. In 2017 international conference on advances in computing, communications and informatics (ICACCI), pp. 79-85. IEEE, 2017.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2122 2020.
Hongyu Guo and Herna L. Viktor. Learning from imbalanced data sets with boosting and data
generation: The databoost-im approach. 6(1), 2004. ISSN 1931-0145. doi: 10.1145/1007730. 1007736. URL https: http///o..o//////// org. /00.////.....
1007736.
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderline-smoot new over-sampling method in imbalanced data sets learning. International conference on intelligent computing, pp. 878-887. Springer, 2005.
10

---

Under review as a conference paper at ICLR 2023
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions knowledge and data engineering, 21(9):1263-184 1284, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning to segment the tail. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14045-14054, 2020.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelli- 2019a.
gence,
W Ronny Huang. Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. arXiv preprini arXiv:1906.03291, 2019b.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217. 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URLhttps: http///pppeeeeeeeeeeeeeeeeeeeeea net forum? id=r1gRICVEVE.
Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
pp.
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting contrastive self-supervised representation learning pipelines. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9949-9959, 2021.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009. URL https: //www. Cs toronto......................... wew.:eeduuuu/// wwweed////////// wwwrri//eeamiingfetattteeeee www.ureetteeeeeeee 2009 TR.
Yann LeCun. The mnist database of handwritten digits. htlp://yann. lecun. com/exdb/mmist// 1998.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-2988, 2017.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2544 2019.
Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 864-873, 2016.
Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra. Journal of Machine Learning Research, 21(252): 2020.
Serim Ryou, Seong- Gyun Jeong, and Pietro Perona. Anchor loss: Modulating loss scale based on prediction difficulty. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1992,6001, 2019.
11

---

Under review as a conference paper at ICLR 2023
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-vvranne ver-pprametrized neural networks. arXiv preprint 1rXiv:1706.04454 2017.
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew Gordon Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. arXiv preprint arXiv:2205. 10279 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint urXiv:2106.01344 2021.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving -las-immallaned learning. Advances in neural information processing systems, 13:1929--1930, 2020.
A APPENDIX
A.1 MODEL DETAILS
XGBoost: XGBClassifier'' from XGBoost version 1.6.2
eestimators'' 100
'subsample" 0.5
'eta' 0.3
'max_depth' = 6
SVM: 'LinearVMM' from sklearn version 1.1.2
'dual' false
'max iter' 1000
A.2 SUPPLEMENTAL FIGURES
12

---

Under review as a conference paper at ICLR 2023
Equal Train/ Test Ratio
Balanced Training
Best Train Ratio
1044
10
Majority Test Ervor
10
Majority Test Ervor
10
10
10
10'
10"
16
10
Test Ratio
Test Ratio
Test Ratio
(a) ResNet on CIFAR-IU 10 Dataset
(b) XGBoost on Adult Dataset (c) SVM Forest Cover Dataset
1644
10'
10
10'"
Minority Test Ervor
Minority Test Entor
10
16'
10'
10
10"
10
10a
10
10'
10"
10"
Test
Test Ratio
Ratio
(d) ResNet CIFAR-IO Dataset
XGBoost Adult Dataset (f) SVM Forest Cover Dataset
Figure 11: Test error split by majority and minority classes for balanced test sets. We see similar trends across all models and datasets.
Equal Train/ Ratio
Balanced Training
167
16'
ROCAADIC Agre
Reall
Florre
1d*
10
1073 Tirt Ratoo
10"
Text Ranio
Toe Ratio
Thet Raiio
Figure 12: Additional metrics for XGBoost Adult dataset
Equal Train/ Test Ratio
Balanced Training
Pretten
Bazall W07
ROC ALC
10
10"
Text Ratoo
Test Rocio
Text Rotio
Text Ratio
Figure 13: Additional metrics for SVM on Forest Cover dataset
13

---

Under review as a conference paper at ICLR 2023
10'
Stanire Ovenampling Overiiting
10
Majarity Train Eiror
Train Ervor
10"
10'
Minority
10
10
10
10
10
10
Train Ratio
Train Ratio
Train Ratio
Figure 14: ResNet train enror CIFAR-10
10%
Majority Test Eiror
Minority Test Ercor
10
10'*
10'
10
10'
Overanmpling
Overfiting
16
10
10
10"
10
10"
10"
10"
10"
10"
Train Ratio
Train
Figure 15: ResNet test error on imbalanced test sets from CIFAR-10
10'
Stundan
16'
16
Overaapling
Overfining
Majonity Irain Liror
Minority Train Error
10
10
10
10"
16
10
10
Train Ratio
Train Ratio
Tmin Ratio
Figure 16: XGBoost train error Adult
10
Majority Lost Larorr
Minority Test Ervor
10"
Test 20
t0*
Overamplisg
Overfitiig
Train Ratio
Train
Train Ratio
Figure 17: XGBoost test error on balanced test sets from Adult
14

---

Under review as a conference paper at ICLR 2023
16
Majocity Test Eiror
Minority Test Error
10
10
10'
16
Surdde
Ovenumpling
Overftiing
10
Train Ratio
Train Ratio
Tran Ratio
Figure 18: XGBoost test error on imbalanced test sets (train ratio and test ratio are equal) from Adult
15


Reviews:
[{"review_id": "zeD475b3JEk", "reviewer": "Reviewer_aUfE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies representation learning under data imbalance. In particular, it performs extensive empirical investigation on the learning behaviors for different sets of models, and observe that  re-balancing class-imbalanced data is in general ineffective. Along this line, it studies self-supervised pre-training, Bayesian inference, and  flatness-seeking regularization for data imbalance, and verifies the advantages of these methods in the presence of imbalanced data across a variety of domains.", "summary_of_the_review": "The topic of this paper is interesting. However, as elaborated in the weakness part, none of the observations in the paper is actually new. The contribution is unclear and the novelty is low.\n\nCurrently there are several major drawbacks and weaknesses, in terms of contribution and exepriments. I view this work as a half-baked work and needs significant improvements.\n\nIn summary, I recommend rejection for the current paper. The paper needs to be significantly improved to meet the bar of top conferences like ICLR.", "strength_and_weaknesses": "# Strength \n\n+ The topic is interesting and timely. Data imbalance and few-shot learning is important and practical in real-world datasets, and in-depth analysis is plausible for understanding the behavior of representation learned under such data bias.\n\n+ The writing and overall structure is good, and the paper is easy to follow.\n---\n# Weaknesses\n\nUnfortunately, there are several major weaknesses that exist in the current paper.\n\n### Unclear Contributions\nThe paper focuses on representation learning under class imbalance. However, many topics in this paper have already been extensively studied in the literature. I had a hard time figuring out the real contributions of this paper. Can you explain what's new in the paper, and position your work w.r.t the literature?\n\n**Specifically**, the following \"findings\" in this paper have been well studied / known in the literature.\n- _\"re-balancing training data actually harms accuracy under imbalanced testing\"_\n1. This argument is actually trivial in its form. Without any label shift between training and testing, ERM should be the optimal solution [1]. This phenomenon actually motivates the logit adjustment method, where you only need to adjust the logit to compensate the training / testing label shift for a optimal classifier [1]. Going back to this paper, if the training / testing label distribution is the same, no matter how imbalanced they can be, you simply do not need to tune anything, as it is simply a in-domain generalization problem without any shifts.\n2. Even under the above setup, the implicit assumption we have is $P_{train}(x|y) = P_{test}(x|y)$, where the only bias comes from the label shift between training and testing. Elaborating this assumption could potentially be a more interesting direction, which is however totally missing in the paper.\n3. Perhaps the real interesting question here, is what if the test set is not balanced, but arbitrarily imbalanced (e.g., could be inversely long-tailed compared to the training label distribution). The observation here will easily break when extending to arbitrary testing label shifts. Such scenario is, however, also covered in the literature [2].\n\n- _\"oversampling which fit these samples do not improve generalization\"_\n\n1. Again, this is not even a proper takeway -- tons of papers in this field have already studied this phenomenon, which is why methods like logit adjustment, ensemble learning, self-supervised learning, contrastive learning, semi-supervised learning (with more unlabeled data) have been introduced. This observation is **out-dated** and do not provide anything new to the research field.\n\n- _\"A two-step process of (a) SSL pre-training on the imbalanced training set and ... accuracy across imbalance ratios.\"_\n\n1. This takeaway is what has been exactly done in the paper two years ago [3]. Strictly speaking, nothing new is presented here, despite that less analysis is conducted in this paper. I suggest the authors doing a comprehensive literature review and correcly position the paper to the literature, and what new contribution/observation is made.\n\n- _\"Under class imbalance, models are particularly underspecified by the data\"_\n\n1. Similarly, past works (e.g., [4]) have studied this phenomenon. There are even new methods that involve uncertainty modeling for combating underspecification proposed [4] in imbalanced learning.\n\n### Limited Experiments\n\n- One big drawback in the experiments is that only small datasets are evaluated. CIFAR-10 has been standard but also too small for the data imbalance / long-tailed recognition field. The fields have advanced to larger and more practical datasets with higher resolution, such as ImageNet-LT or iNaturalist. Without validation on these large-scale datasets, the observations may not even be justifiable or convincing.\n\n- No ablation studies across datasets / network architectures / optimization methods performed, which again makes the observation less convincing. If the author wanted to do an in-depth analysis, the first thing they need to make sure is that the observations **really persist** across a range of datasets / network architectures / etc.\n\n- Despite that the title is \"Representation learning ...\", no real analysis on the learned feature space under class imbalance is performed. How exactly is the feature or representation balanced / imbalanced? More analysis and experiments are needed.\n\n### Writing and typos\n- page 4, \"Takeaway\" - the sentence is grammarly incorrect. You might want to change it to something as \"Training with a train set exhibiting class imbalance similar to that of\"\n\n# References\n[1] Long-tail learning via logit adjustment. ICLR 2021.\n\n[2] Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition. 2021.\n\n[3] Rethinking the Value of Labels for Improving Class-Imbalanced Learning. NeurIPS 2020.\n\n[4] Striking the Right Balance With Uncertainty. CVPR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VcKHBI2wNA", "reviewer": "Reviewer_4VFj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explore and evaluate several techniques for class imbalance.   Instead of assuming the test set is balanced, they explore imbalance test ratio.   Based one one image dataset with NN and two tabular datasets with SVM and XGBoost, they made the following observations.\n\n1.   training ratio similar to test ratio is desirable\n2.   oversampling minority samples does not improve generalization\n3.   self-supervised learning for pre-training before supervised learning improves accuracy\n4.   using Bayesian inference to incorporate uncertainty increases accuracy\n\n", "summary_of_the_review": "The claims could be more convincing if more datasets and performance metrics (for imbalanced data) are used.  Currently, the claims seems to be stronger than the evidence can support.   The paper title stated representation learning, but some of the claims are not related to representation learning.", "strength_and_weaknesses": "Strengths:\n\n1.  Instead of assuming the test set is balanced, they explore imbalance test ratio. \n\n2.  the paper is generally well written\n\nWeaknesses:\n\n1.  More datasets would be more convincing--one image dataset and two tabular datasets seem insufficient\n\n2.  accuracy/error is one measurement of performance, however, for imbalanced datasets, other performance measurement such as F1 and AUC of ROC could be used.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mz_Kn87ooi", "reviewer": "Reviewer_D5Wc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the learning of classifiers from class-imbalanced data. In the first part of the paper, the authors empirically investigate the effect of imbalanced classes on the performance of classification models. They find that i) oversampling minority class is ineffective, and that ii) adding too many examples of the majority class can be harmful as well. In the second part, they show that three existing methods can improve the performance of deep models in a class-imbalance scenario, namely i) using self-supervised pretraining followed by a supervised fine-tuning, ii) using Bayesian inference, and iii) using sharpness-aware minimization of the loss.", "summary_of_the_review": "The paper comes with interesting observations regarding the usefulness of existing methods in a class-imbalanced setting. The main deficiency of this purely empirical paper is the limited number of datasets, which makes the conclusions rather speculative.", "strength_and_weaknesses": "Strength:\n\nThe paper addresses a topic of both practically relevant and theoretical interest. \n\nThe authors identify three existing methods that show promising results when applied in class-imbalanced setting. \n\nWeaknesses:\n \nThe authors make a relatively strong conclusion about previously used methods, e.g., that the rebalancing of training data is generally not helpful. The authors' conclusions regarding representation learning are made on the basis of experiments, which are reasonable; however, they use a single NN architecture trained on a single dataset. For me, it is not clear that the observations would apply to different data distributions. A similar problem, i.e. using a single dataset, appears when evaluating the three proposed methods.\n\nQuantities used to characterize data distribution and performance have flows that limit conclusions made and should be at least discussed. First, the class-imbalance ratio used as a way to characterize the entire class distribution completely neglects the intermediate classes, which certainly have a large impact on the results. Second, the classification error is not an appropriate metric in the cases where the test data are severally class-imbalanced, as the performance on the minority class has a negligible influence on the metric. \n\nFor some unknown reason, the three proposed techniques are not compared against each other in terms of prediction performance nor in terms of computational demands.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3G-YWmGa3o_", "reviewer": "Reviewer_Mds7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the class-imbalance problem. In many practical cases, the data set has minority classes that have few samples. To understand the effect of highly imbalanced data sets, the authors perform extensive empirical studies for various ML models, including neural networks, gradient-boosted decision trees, and VMSs. The observations are as follows. Re-balancing and oversampling for the minor samples are not effective. They even degrade performance. Self-supervised pre-training is insensitive to imbalance. Bayesian is effective. Flatness-seeking regularization can provide more margin to the boundary of the minorities. \n", "summary_of_the_review": "This paper provides many experimental results and messages from the experiments for the class-imbalance problem. However, this work does not compare with recent algorithms and the performance metric is not proper.", "strength_and_weaknesses": "Strengths\n\n- From experiments, the authors find interesting observations for class-imbalance cases\n\nWeakness\n\n- Since this paper considers imbalanced data sets, it is not appropriate to use accuracy as the unique performance metric. Other metrics like Precision, recall, bias, f1, CSI, ROC, should be considered.\n\n- The authors should discuss and implement recent works addressing the class-imbalance problem. \n\n- Messages from this paper should be compared with recent works. For instance, one of the claims of this work is \"minority samples are hard to fit, yet algorithms which fit them, such as oversampling, do not improve generalization.\" However, there are some works that improve the performance by fitting the minority. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject