Simplifying Neural Network Training Under Class Imbalance
Authors: Ravid Shwartz-Ziv
Micah Goldblum
Yucen Lily Li
C. Bayan Bruss
Andrew Gordon Wilson
Venue: NeurIPS 2023 poster (2023)
Keywords: Class Imbalance
Hyperparameters
Long-Tailed Distributions
TL;DR: We demonstrate that by optimizing the standard components of deep learning pipelines, we can achieve state-of-the-art performance on imbalanced datasets, eliminating the need for specialized loss functions or samplers.

Abstract: Real-world datasets are often highly class-imbalanced, which can adversely impact the performance of deep learning models. The majority of research on training neural networks under class imbalance has focused on specialized loss functions and sampling techniques. Notably, we demonstrate that simply tuning existing components of standard deep learning pipelines, such as the batch size, data augmentation, architecture size, pre-training, optimizer, and label smoothing, can achieve state-of-the-art performance without any specialized loss functions or samplers. We also provide key prescriptions and considerations for training under class imbalance, and an understanding of why imbalance methods succeed or fail.

Simplifying Neural Network Training Under Class Imbalance
Ravid Shwartz-Ziv' New York University
Micah Goldblum' New York University goldblum@nyu. edu
Yucen Lily Li New York University yucenl any.. edu
ravid. shwartz. v@nyv.
C. Bayan Bruss Capital One
Andrew Gordon Wilson
New York University
bayan. trrusaccapitalone...
andrewgw@cims. nyu edu
Abstract
Real-world datasets are often highly class-imbalanced, which adversely impact the performance of deep learning models. The majority of research on training neural networks under class imbalance has focused on specialized loss functions, sampling techniques, or two-stage training procedures. Notably, we demonstrate that simply tuning existing components of standard deep learning pipelines, such as the batch size, data augmentation. optimizer, and label smoothing, can achieve state-of-the-ee performance without any such specialized class imbalance methods. We also provide key prescriptions and considerations for training under class imbalance, and an understanding of why imbalance methods succeed or fail.
Introduction
Only minuscule proportion of credit card transactions are fraudulent, and most cancer screenings come back negative. In reality, some events are common while others are exceedingly rare. As result, machine learning systems, often developed in class-balanced settings [c.g.. 48, 46, 16], are routinely trained and deployed on class-imbalanced data where relatively few samples are associated with certain minority classes, while majority classes dominate the datasets. Class-imbalanced training data can negatively impact performance. Consequently, a wide body of literature focuses on specially tailored loss functions and sampling methods for counteracting the negative effects of imbalance [10, 14, 25, 27, 50, 61]. In striking contrast to such approaches, we instead show that simply tuning existing components of standard neural network training routines can achieve state-of-the-art performance on class-imbalanced image and tabular benchmarks at little implementation overhead and without requiring any specialized loss functions or samplers designed specifically for imbalance. Like Wightman et al. [69], who found that modern training routines allow ResNets to achieve performance competitive with that later architectures, we show that modern training techniques cause the benefits of specialized dlass-imbllance methods to nearly vanish.
Moreover, our carefully tuned training routine can be combined with existing class imbalance methods for additional performance boosts. Conducting evaluations on real-world datasets, we find that existing methods, which performed well on web-scraped natural image benchmarks on which they were designed, underperform in the real-world setting, whereas our approach is robust.
Our investigation provides key prescriptions and considerations for training under class imbalance:
Authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1. The impact of batch size on performance is much more pronounced in -lass-imallnnced settings, where small batch sizes shine.
2. Data augmentations have an amplified impact on performance under class imbalance, especially on minority-class accuracy. The augmentation strategies in our experiments which achieve the best performance on class-balanced benchmarks yield inferior performance on imbalanced problems.
3. Large architectures, which do not overfit on class-balanced training sets, strongly overfit on imbalanced training sets of the same size. Moreover, newer architectures which work well on class-balanced benchmarks do not always perform well under class imbalance.
4. Adding a self-supervised loss during training can improve feature representations, leading to performance boosts on cass-imbalanced problems.
5. A small modification of Sharpness-Aware Minimization (SAM) [19] pulls decision bound- aries away from minority samples and significantly improves minority- group accuracy.
6. Label smoothing [57], especially on minority class examples, helps prevent overfitting.
To understand why exactly such training routine improvements confer significant benefits, we investigate the role of overfitting in class-imbalanced training. Our analysis shows that naive training routines overfit minority samples, causing neural collapse [60], whereby features extracted from the penultimate layer concentrate around their class-mean. Combining this analysis with decision boundary visualizations, we demonstrate that unsuccessful methods for class-imbalanced training overfit strongly, whereas successful methods regularize.
Related Work
A long line of research has been conducted on class-imbalanced classification. There are several archetypal approaches specially designed to address imbalance:
Resampling the data. In early ensemble learning studies, boosting and bagging algorithms were adjusted to take account of imbalanced data by resampling. Traditionally, resampling involves oversampling minority class samples by simply copying them [25, 10, 27], undersampling majority classes by removing samples [17, 32, 3, 8], so that minority and majority class samples appear equally frequently in the training process.
Loss reweighting. Reweighting methods assign different weights to majority and minority class loss functions, increasing the influence of minority samples which would otherwise play little role in the loss function [14, 34]. For instance, one may scale the loss by inverse class frequency [28] or reweight it using the effective number of samples [14]. As alternative approach, one may focus on hard examples by wwwnwweighing the loss of well-classified examples [50] or dynamically rescaling the cross-entropy loss based on the difficulty of classifying a sample [61]. Bertsimas et al. [6] encourage larger margins for rare classes, while Goh and Sim [21] learn robust features for minority classes using class-uncertainty information which approximates Bayesian methods.
Two-stage fine-tuning and meta-learning. Two-stage methods separate the training process into representation learning and classifier learning [54, 59, 38, 4]. In the first stage, the data is unmodified, and no resampling or reweighting used to train good representations. In the second stage, the classifier is balanced by freezing the backbone and fine-tuning the last layers with resampling or by learning to debias the class confidences. These methods assume that the bias towards majority classes exists only in the classifier layer or that tweaking the classifier layer can correct the underlying biases.
Several works have also inspected representations learned under class imbalance. Kang et al. [38] find that representations learned on class-imball imbalancddd training data via supervised learning perform better when the linear head fine-tuned on balanced samples. Yang and Xu [71] instead examine the effect of self- and cemi-supervised training on imbalanced data and conclude that imbalanced labels are significantly more useful when accompanied by auxiliary data for semi- rumi-suerviseed learning. Kotar et al. [44], Yang and Xu [71] make the observation that self-super pre-training is insensitive to imbalance in the upstream training data. These works study SSL pre-training for the purpose of transfer learning, sometimes using linear probes to evaluate the quality of representations. Inspired by their observations, we find that the addition of an SSL. loss function on the same class-imbalanced dataset, even when no upstream data is available, significantly improve generalization.

---

In summary. existing works propose countless approaches to address class imbalance during training. In contrast, we show that strong performance can be achieved on class-imbalanced datasets simply by tuning the components of standard neural networks training routines, without specialized loss functions or sampling methods designed specifically for imbalance. Our tuned routines require little to no additional implementation compared to standard neural network training pipelines and can be combined with existing specialized approaches for class imbalance. We additionally provide novel prescriptions and considerations for training under class imbalance, well an understanding of how regularization can contribute to the success of training under imbalance.
3 Optimizing Training Routines for Imbalanced Data
While previous research training class im- balance mainly concentrated on developing new
loss functions and sampling methods tailored to imbalanced data, little attention has been paid
Improvement
10-1
to how the components of traditional training routines interact with such data.
Train Ratio
-3
In this section, delve into various methods that can be optimized or modified specifically for imbalanced training scenarios. For clarity and depth of discussion, we divide these meth- ods into two distinct groups, each discussed in its own subsection.
10
-6
-00
10a
10a Batch Size
10
Subsection 3.1 deals with what we identify as the "fundamental building blocks" of conven- tional balanced training. This includes elements like batch size, data augmentation, pre-training, model architecture, and optimizers. We scruti- nize the effects of altering these elements' hy- perparameters, emphasizing their influence the performance of models under imbalanced training. Our discussion is fortified by experi-
Figure 1: Imbalanced data prefers small batch sizes. We plot the percent improvement in accuracy baseline batch size 128 for different train ratios as a function of batch size. Positive values indicate higher accuracy than the baseline. Balanced training sets yield flatter lines, indicating insensitivity batch size. Exper- iments conducted with ResNet-50 on CIFAR-100.
the
ments conducted on both vision datasets (CIFAR-10, CIFAR-100, CINIC-10 and Imny-ImageNet) and tabular datasets across different network architectures.
Subsection 3.2 introduces a set of optimization methods-Joint-SSS SAM, and label smoothing we have reformed to cater more appropriately to imbalanced training. These methods, initially designed for balanced datasets, are subjected to modifications making them more amenable to imbalanced scenarios. The effectiveness of these methods, including comparisons other state-of- the-art techniques across various domains and datasets (vision and tabular), is thoroughly evaluated in Section 1.
This structure allows a separate discourse on the fundamental aspects of deep learning training and the particular adjustments needed for imbalanced training. Our aim is to provide a comprehensive overview of both traditional and innovative techniques, giving readers a broad spectrum of strategies for tackling imbalanced data.
3.1 Tuning the Building Blocks of Neural Network Training
The success of deep learning models hinges on the precise orchestration of their training routine. The success of models is especially sensitive to this orchestration under imbalanced training data. We now explore the building blocks of neural network training routines and their importance in optimizing models for imbalanced data. This study equips researchers and practitioners with practical strategies to improve the performance of their models under class imbalance. In this subsection, we focus on batch size, data augmentation, pre-training, model architecture, and optimizers.
3

---

3.1.1 Experimental Setup
Datasets. To conduct our investigation, we leverage three benchmark image datasets: CIFAR-10 [46], CIFAR-100, and CINIC-10 [15), along with three tabular datasets: Otto Group Product Classification [37], Covertype [7], and Adult datasets [43]. For naturally balanced datasets, we adopt the imbalanced setup proposed by Liu et al. [54], which employs an exponential distribution imbalance classes, closely mirroring real-world long-tailed class distributions. The glass-imbalance ratio (r) represents the ratio of samples in the rarest the most frequent class. A dataset with r 1 fully balanced, while r = 0.1 indicates that the majority class has ten times more samples than the minority class. We investigate varying imbalance ratios in both the training and test sets.
Models. For image datasets, we utilize ResNets [29] and WideResNet [73] with different depths (8, 32, 50, and 152). In addition, to evaluating the role of architecture in imbalanced training, also employ DenseNet [35]. MobileNetV2 [62], Inception v3 [67], EfficientNet [68], and VGG [64]. Tabular datasets are processed using XGBoost [11], SVM, and MLP. We follow the supervised pre-training protocol of Kang et al. [39], while for self-supervised pre-training, we employ SimCLR R
[12] and VICReg [5], where the fine-tuning is described in Kotar et al. [44]. Further details be found in Appendix B.
Metrics. In addition to overall test accuracy, we provide minority and majority class accuracy (representing the 20% of classes with the smallest and largest number of samples) in the appendix, not in the main body of the paper. We conduct five runs with different seeds for each evaluation in our experiments and report the mean along with one standard error.
3.1.2 Results
Batch size. Studies conducted on balanced training data suggest that small batch sizes may exhibit superior convergence behavior, while large batch sizes reach optimal minima unattainable by smaller sizes [41]. In class-imbalanced settings, one intuition is that larger batch sizes may be necessary to obtain enough samples from the minority class and counteract forgetting. On the other hand, large batches can also increase the risk of overfitting.
To examine the influence of batch size in class- imbalanced contexts, train networks with varying batch sizes across multiple training ratios. In Figure 1, we plot the percent improvement in accuracy over the baseline (which is set as best batch size 128 function of batch size for several training ratios. Positive values represent higher accuracy compared to the baseline, whereas negative values denote lower accuracy.
Our analysis reveals that batch size has a much greater impact in highly imbalanced settings and very little impact in balanced settings. Notably, data with a high degree of class imbalance tends to benefit from smaller batch sizes, even though small batches often do not contain minority samples, possibly due to the regularization effects that help mitigate overfitting to the majority classes. See Appendix A.I for additional details and experiments.
Data augmentation. Data augmentation is feature of virtually all modern image classification pipelines. We now investigate the impacts of various augmentation policies across varying levels of class imbalance. Our experiments show that the effects of data augmentation are greatly amplified on imbalanced data, especially for minority classes (Figure 2 left). This finding suggests that augmentation serves as a regularizer, supporting recent studies the role of data augmentation in preventing overfitting during class-balanced training [20]. Moreover, we find that the optimal augmentation policy can depend on the level of imbalance.
We assess our findings using a variety of augmentation methods including horizontal flips, random crops, AugMix [31], IrivialAugmennWide [58], AutoAugment [13], Mixup, CutOut, and Random Erasing.
To identify the most potent data augmentation strategy, we gauge the improvement in accuracy, represented as the percentage increase compared to training without augmentation, in Appendix A.2. While TuimialAugment outperforms other methods on balanced training data, AutoAugment emerges the most effective for imbalanced data
Furthermore, we examine the sensitivity of performance to the specific augmentation method used. By assessing the variance in performance across different augmentations for balanced and imbalanced

---

Minority Majority Classes
Augmentation Varlance
Minority Classes
Majority Classes
10-2
% Improvement
-0-3
10-4 10-3
10-2 10
10
Imbelanced Data
Balanced Data
Train Ratio
Figure Left: Augmentations yield far bigger improvements minority classes. compare percent improvement in test accuracy TrivialAugment compared to training without any augmentation as function of the training ratio. Right: The type of augmentation matters more on imbalanced data. Augmentation variance measures the variance percent improvement augmentation types. Variance across augmentation types imbalanced data is much greater for minority classes, indicating the importance of choosing an appropriate augmentation policy. Experiments conducted with ResNet-50 on CIFAR-100. Error hars represent one standard crror over trials.
training without augmentation across different
Imbalanced Data Acc.
50
51
Reekie--1 Reshve 5o
Test Accuracy
79
Resive
49
50
48
49
EllesrstNetst st
Cecoweles-22 121
Decumltes-EE LED
Imbalanced Data
(noeption)))
Balanced Data
48
47
75
3 Number Parameters
80
1e7
Balanced Data Acc.
Figure 3: Left: Deeper architectures overfit on clas--imbalanced data. While deeper ResNet models perform better on balanced data, they can overfit and underperform on imbalanced data. Right: Performance on balanced and imbalanced datasets is virtually uncorrelated across wide variety of architectures (Pearson correlation coefficient 0.14). Experiments conducted CIFAR-100 with imbalanced train ratio 0.001. Error bars represent one standard error over trials.
situations (see Figure 2), find that minority class performance is particularly sensitive to the chosen augmentation policy See Appendix A.2 for additional details and experiments.
Model architecture. While larger networks often enhance performance on bass-balanced datasets without overfitting, their efficacy on imbalanced data remains unexplored. To probe this behavior, we train ResNets of various sizes on both balanced and imbalanced data with training ratio of 0.01. plot the test accuracy on minority classes as a function the network's size Figure 3 (left). While the network's 's performance on balanced data monotonically improves with increasing size, its performance on imbalanced data peaks at size with 20 million parameters, and declines thereafter. This dip in performance suggests that, unlike their behavior on balanced data, larger networks may be susceptible to overfitting minority classes in the face of severe class imbalance. then plot the the balanced and imbalanced CIFAR- 100 variants
test accuracy of a wide variety of architectures in Figure (right), and find that accuracies these two settings are virtually uncorrelated! Computer vision architectures have largely been developed well-known class- balanced benchmarks like ImageNet [16]. These developments may not generalize to imbalanced settings which are common in the real world. See Appendix A.3 for additional details and experiments.
Pre-training. Fine-tuning models pre-trained on expansive upstream datasets can markedly enhance performance across variety of domains by equipping the model with an informative initial parameter

---

vector learned from pre-training data [12, 65]. Self- Seff-Supervised Learning (SSL) has emerged as highly effective strategy for representation learning fields such as computer vision, natural language processing (NLP), and tabular data. Networks pre-trained using SSL. often produce more transferable representations than those pre-trained through supervised learning. Furthermore, self-supervised pre-training strategies for transfer learning display greater robustness to upstream imbalance com- pared to supervised pre-training [51]. We observe in our experiments that pre-trained backbones provide considerably greater benefits under severe downstream class imbalance than under balanced downstream training sets. The ability for SSL pre-training to mitigate overfitting could therefore be particularly valuable in the class iusssiibblaneed setting.
To gauge the effectiveness of pre-training, we fine-tune various pre-trained models on downstream datasets with range of class imbalance ratios. We make use of pre-trained ResNet-50 weights learned on ImageNet-1KK [16], ImageNet-2IK, and two self-supervised learning methods, SimCLR and VICReg. Figure 4 depicts the relative improvement in test accuracy compared to training from random initialization across different pre-training models. positive value signifies performance improvement. All pre-training methods notably outperform random initialization. However, observe a considerably larger improvement under class imbalanced scenarios, where models pre- trained on larger datasets yield greater boosts in accuracy. Moreover, SSL methods conducted on ImageNet- IK surpass supervised pre- pre-training on the same upstream training set. See Appendix A.4 for additional details and experiments.
3.2 Improved Optimization Methods for Class Imbalance
While the fundamental building blocks above provide a strong foundation for training perfor- mant models, we can also customize modern optimization techniques specifically for imbal- anced training and achieve further gains. This subsection showcases adapted variants of SSL, SAM, and label smoothing.
40
ImagetNet-21k
% Improvement
SimCLR
ImageNet-11
No Pre-Train
20
Self-Supervision. Self-Supervised Learning (SSL) has received substantial attention in rep- resentation learning, particularly in computer vision, NLP, and tabular data [12, 40, 65, 49], especially for training on massive volumes of labeled data. Networks pre-trained via SSL of- ten showcase more transferable representations compared to those pre-trained through super- vised methods [24]. Moreover, self-supervised pre-training strategies for transfer learning hibit more robustness to upstream imbalance compared to supervised pre-training [51]. De- spite these advantages, the availability of mas- sive pre-training datasets can often be limiting factor in many use cases.
10
1e
1e0
Train Ratio
Figure 4: Pre-training more impactful on imbal- anced downstream data. % Improvement refers to the improvement in test accuracy compared training from random initialization. The benefits of SSL over supervised pre- me-training are also amplified under class imbalance. Experiments conducted with ResNet-50 on CIFAR-100.
Traditionally, pre-training involves two-step process: learning on upstream task, followed by fine-tuning on a downstream task. In our approach, we instead integrate supervised learning with an additional self-supervised loss function during from-scratch training, bypassing the need for pre- tecetrrinnng. The simple integration of an SSL loss function, which does not depend on class- imbalanced labels and is insensitive imbalance, results in improved feature representations and better generalization. We refer to this combined training procedure as foint-SSL.
is important to note that our method differs from those used in Kotar et al. [44], Yang and Xu [71], Liu et al. [51], which investigate SSL pre-training on larger datasets for transfer learning, rather than directly training with SSL objective downstream data. See Appendix A.5 for additional details.
Sharpness-Awer Aware Minimization (SAM). SAM [19] is an optimization technique for finding "flat" minima of the loss function, often improving generalization. The technique involves taking inner ascent step followed by a descent step to find parameters that minimize the increase loss from the decision boundaries.
ascent step. Huang al. [36] shows that flat minima correspond to

---

We thus adapt SAM for class-imbalanced cases by increasing the flatness especially for minority class loss terms. To do so, we increase the ascent step size in SAM's inner loop for minority classes, denoting this method SAM-Asymmetric (SAM-A). Plotting the decision boundaries of small multi-layer perceptron on a toy 2D dataset (Appendix Figure 18), we observe that classifiers naturally form small margins surrounding minority class samples. SAM-A expands these margins, preventing the model from overfitting to the minority samples. See Appendix A.6 for additional details.
Label smoothing. Conventionally, classifiers are trained with hard targets, minimizing the cross- entropy between true targets yı: and network outputs pk: as in (y: E - equal for the correct class and O otherwise. Label smoothing uses a smoothing parameter, to instead minimize the cross-entrops between smoothed targets yLS, and network outputs pk, where yLS,, = yu (1 - <) + </K [57].
yv log(ps), with ys
We adapt label smoothing for the class-imbalanced setting by applying more smoothing to minority- class examples than to majority- examples. This procedure prevents overfitting on minority samples. See Appendix A.7 for additional details and experiments.
Dataset curation. Common intuition dictates that training on data that is more balanced than the testing distribution can improve representation learning by preventing overfitting to minority samples [25, 10, 27]. In Appendix A.8, we find that this intuition may be misguided both for neural networks and gradient-booster decision trees, especially on large datasets, and that curating additional samples may in fact be destructive if the proper dataset balance not maintained.
Benchmarking Training Routines under Class Imbalance
In the preceding sections, we examined various building blocks of balanced training routines and presented modifications of optimization methods- label smoothing, Sharpness- Minimization, and supervvperv -ereevisioontaillreeedd tailored specifically for imbalanced scenarios. In this section, compare the performance of models trained using our modified methods to those trained using existing state- of-the-art methods for handling class imbalance. To ensure a fair and unbiased comparison, all methods are trained using the same refined training routines. Our comparison provides evidence of the effectiveness of our proposed methods when coupled with our optimized routines.
4.1 Vision Datasets
4.1.1
Experimental Setup
Datasets. We perform experiments on six image datasets, including natural image, medical, and remote sensing datasets: CIFAR-10 [46], CIFAR-100, CINIC-10 [15], Tiny-ImageNee(16)) SIIM-
ISIC Mclanoma [26], APTOS 2019 Blindness [1] and EuroSAT [30].
Baseline Methods. We compare to the following comprehensive range of baselines: (a) Empirical Risk Minimization (ERM) involves training on the cross-entropy loss without any re-balancing.
(b) Resampling balances the objective by adjusting the sampling probability for each sample. (c)
Synthetic Minority Over-sampling Technique (SMOTE) [10] is a re-baaani variant that involves oversampling minority classes using data augmentation. (d) Reweighting [33] simulates balance by assigning different weights to the majority and minority classes. (e) Deferred Reweighting (DRW) involves deferring the resampling and reweighting until later stage in the training process. (f) Focal Loss (Focal) [50] upweights the objective for difficult examples, thereby focusing more on the minority classes. (g) Saabel-Distibbutinn-e
Margin (LDAM-DRW) [9] trains the classifier to impose a larger margin minority classes. (h) M2m [42] translates samples from majority to minority classes. Lastly, (i) MISLAS [75] is a two-stage training method that combines mixup [74] with label smoothing. We also combine our techniques with previous state-of-the- thee-a-rtt Major-to-ming Translation (M2m) [42] and observe it improves performance over M2m alone.
Evaluation. We follow the evaluation protocol used in [54, 42], training models on class daas-iiballanced training sets and evaluating them on balanced test sets. We evaluate on four benchmark datasets for imbalanced classification: CIFAR-10, CIFAR-100 [46], CINIC-10 [15] and Tiny-ImagNet [47] with training ratios of 0.01, 0.02, and 0.1. Additionally, we use three real-world datasets, namely APTOS 2019 Blindness Detection [1]. SIM-ISIC Melanoma Classification [26], and EuroSAT [30].
7

---

Table 1: Our training routines exceed previous SOTA improve existing methods when combined.
Accuracy on various datasets using different methods. Error bars correspond one standard error over trials.
CITAR 10
CIFAR 100
CINIC-10
10
Tiny ImageNet (0.1)
Mothod
0.01
0.02
0.01
0.02
0.01
0.02
SwinV2 ConyNeXt
FRM
64.9 0.2
84.2 4 0.3
47.7 + 0.5
52.5 0.4
78.6 + 0.2
62.3 +
53.4 + 0.2
53.1 + 0.3
Reweighting
51.8 1: 0.3
79.8 0.4
42.1 0.4
47.8.40..3 0.3
71.4 a 0.4
74.9 0.4.
52.8 a. 0.4
52.3 4 0.1
Resampling
82.1 0.3
79.0
0.3
0.
0.5
7..6
52.1 6.3
Focal Loss
83.2 1: 0.5
84.4 0.4
1..1
0.5
74.9 0.4
78.4 0.3
53.5 0.1
53.1 1 0.4
LDAM-DRW
$4.9 0.3
82.9 0.4
47.2
53.3++0.3 0.3
76.0
70.7 +
64.2
53.4
M2m
84.5 0.2
84.3++ 0.2
47.1
52.1 0.3
78.3 = 0.5
81.3
64.3 + 0.4
53.9 0.2
MISLAS
85.1 0.3
81.3 0.3
47.3 0.3
33.1 0.5
78.1 0.2
81.8 0.3
54.1 0.3
$3.4
SAM-A
85.4 0.3
83.3 + 0.3
47.1 0.4
52.7 0.2
77.3 0.2
80.9 + 0.2
64.7 0.4
53.9 0.2
Joint-SSL
85.2 0.2
84.1 0.2
0.2
0.5
77.5
01.
64.3
53.7
Joint-SSL-
SAM-A Smoothing
65.9 0.2 0.2
84.7 1 0.3
48.0 1.001.0 0.3
52.6 0.3
78.1 L.
82.3 1. 0.4
54.8 L. 0.1
54.1 11 0.4
Joini Joini-SSL+ SSL+
86.0 0.5
85.0 0.4
48.9 53.8 + 0.2
79.2 + 0.4
62.1 +
55.0 0.2
54.3 + 0.3
SAM-A
Training procedure. Our training protocol for all methods incorporates TrivialAugment [58] + CutMix [72] as augmentation policy, along with label smoothing and exponential moving
weight average with small batch size on image classification. We use WideResNet-28x 10 [73] for CIFAR-10, 10, CIFAR-10K and CINIC-10 datasets. For Tiny-ImageNet we use both ConvNeXt [53] and Swin transformer v2 [52] and ResNeXt-50-32x44 [70] for the real-world datasets following the comparisons in Fang et al. [18]. We provide further implementation details in Appendix B.
4.1.2 Image Classification Benchmarks
We see in Table I that our adaptations alone bring the performance of standard ERM training remark- ably close to
exceeding that of ttate-of-the-art imbalanced training methods. Additionally, atop M2m
the integration of our findings from Section with Joint-SSL and Asymmetric-SAl establishes new state-of-the across all benchmark datasets. Even using only self-supervision and Asymmetric- -immt (without M2m) often surpasses the previous state-of-thh-art art. Furthermore, the integration of previous findings from Section 3 with Joint-SSI. and Asymmetric-SAM atop M2m establishes new state-of-the-ar performance across all benchmark datasets.
Our combined method enhances minority class accuracy while preserving accuracy majority classes. On the 20 smallest classes of CIFAR-100, our method reduces error by 5.4% 6.8% compared to baseline methods. On CINIC-10, our approach enhances the generalization the two smallest minority classes by 4.6% - 8.5% compared to baseline methods. Additional results regarding split class accuracy and different backbones can be found in Appendix A.9.
4.1.3 Real-World Datasets
Table 2: Our routines exceed previous SOTA on real- world datasets. Error bars corresponds to one standard error over 5 trials using ResNcXt-50-32244 1 denotes the method the line above.
Since the majority of research on class- imbalanced training focuses web-scraped datasets such as ImageNet or CIFAR-10 and CIFAR-100, now investigate whether such advances are overfit to these datasets or whether they are robust other settings.
SIM--SIC Melanoma
APTOS 2019 Blindness
Method
EuroSAT
ERM
95.1 + 0.3
89.1 + 0.3
99.0 = 0.2
To address this question, we consider the SIIM- ISIC Melanoma, APTOS 2019 Blindness, and EuroSAT datasets. In Table 2, we observe that our highly tuned training routine equipped with Joint-SSL, Asymmetric-SAte bel smoothing delivers equal or superior per- formance compared to previous oftte-of-the-ane imbalanced training methods on all datasets.
Reweighting
94.7 0.3
88.4 +: 0.2
98.4 08.4:10 0.3
Resampling
94.9 + 0.2
88.7 + 0.4
98.7 + 0.3
Focal Loss
95.0 0.2
89.2 + 0.3
98.9 : 0.3
and modified la-
LDAM-DRW
95.2 0.2
89.0 + 0.4
99.0 -:0 0.4
M2m
95.2 0.2
89.6 + 0.5
99.3 0.2
MISLAS
95.4 1..41000 0.2
89.4 1: 0.4
99.2 : 0.3
Joint-SSL
95.7
89.7 + 0.5
99.1 0.2
Correlation between performance on differ- ent datasets. Given the superior performance of
+ SAM-A
96.0 0.2
90.1 0.G
99.1 + 0.2
+ Smoothing
96.2 + 0.2
90.1
0.2
8

---

our approach on these datasets, we now examine the correlation between the performance of various methods on CIFAR-10 (with training ratio of 0.01) and their performance on real-world datasets. Our findings, illustrated in Table 3, reveals a surprisingly low correlation. This low correlation suggests that a method's successful application to web-scraped datasets like CIFAR- does not necessarily translate into equivalently strong performance on real-world datasets. These findings further underscore the need for more diverse range of datasets in the development and testing of machine learning methods for class imbalance.
4.2 Tabular Datasets
Tabular data problems represent a challenging frontier for deep learning research. While recent advances in natural language processing (NLP), vision, and speech recognition have been driven by deep learning models, their efficacy in the tabular domain remains unclear. Notably, there is a debate over the performance of neural networks in compar- ison to decision tree ensembles like XGBoost [11, 63, 56). Despite the fact that most tabular datasets inherently hibit imbalance, there has been limited research addressing the impact of imbalanced data deep learning in tabular domains.
Table 3: There is only weak correlation between CIFAR-10 test accuracy and per- formance on real-world datasets.
Dataset
Correlation
Slope
APTOS 2019 Blindness
0.11
0.04
SIIM-ISIC
0.19
0.03
Melanoma
EuroSAT
0.03
0.04
We thus apply findings imbalanced tabular datasets, using a Multilayer Perceptron (MLP) with the improved numerical feature embeddings of Gorishmiy et al. [23]. Our approach incorporates SAM-A. modified label smoothing, and SGD with cosine annealing performed and small batch size.
We apply our methods to the following three imbalanced tabular datasets: Otto Group Product Classification [37], Covertype [7], and Adult datasets [43]. We compare our methodology with the following baseline methods: (1) XGBoost, (2) MLP, (3) ResNet, and (4) FT-Transformer, where the
last three baselines are employed as in Gorishniy et al. [22]. Our tuning, training, and evaluation
protocols are consistent with those in Gorishniy et al. [23]. For full details about the training procedures, see Appendix 1.9.1.
We see in Table 6 that our tuned training routine outperforms both XGBoost and recent state-of- the-art neural methods on all datasets, demonstrating the applicability of our findings beyond image classification.
5 Regularization and Overfitting in Class-Imbalanced Training
Specialized loss functions and sampling methods for class-imbalanced learning are often designed to mitigate overfitting [42], yet we rarely look under the hood understand what happens to models trained in this setting. In this section, we quantify and visualize overfitting during class-imbalanced training, and we find that successful methods regularize against this overfitiing
One concern for training under class imbalance might be optimization. Perhaps minority samples hard to fit since they occur infrequently in batches during training. In Appendix A.10, we verify that empirical risk minimization, without any special interventions, easily fits all training data. This observation indicates that variations in performance among the different methods we compare stem not from their optimization abilities but from their generalization to unseen test samples.
To understand the differences between classifiers learned imbalanced training data, visualize their decision boundaries on a 2D toy problem with a multi-layer perceptron. Standard training results in small margins around minority class samples, whereas SAM-A, acting as regularizer, expands these margins (Figure 18a). For additional examples, see Appendix A.11, where we also use the method introduced by Somepalli al. [66] to visualize decision boundaries.
To quantify these observations, we examine the Neural Collapse phenomenon [45], which was previously observed in the class-balanced setting. Neural Collapse refers to the tendency of the features in the penultimate layer associated with training samples of the same class to concentrate around their mass-means. Our investigation focuses two metrics:

---

Class-Distance Normalized Variance (CDNV): This metric evaluates the compactness of features from two unlabeled sample sets, S and S2. relative to the distance between their respective feature means. A value trending towards zero indicates optimal clustering.
Nearest Class-Center Classifier (NCC): As training progresses, feature embeddings in the penulti- mate layer undergoing Neural Collapse become distinguishable. Consequently, the classifier tends to align with the 'nearest class-center classifier'.
We see Table 4 that the collapse, measured by CDNV and NCC for minority training examples, is sig- nificantly worse in standard training without class-imbalance interventions.
Table 4: Neural Collapse during class-imbalanced training. Neural collapse (low CDNV and high NCC) corresponds to low test accuracy. Experiments conducted on CIFAR-10.
Method ERM
CDNV
NCC
Accuracy
Furthermore, correlation exists be- tween the degree of collapse and the performance of the different meth- ods. Specifically, methods that suc- cessfully counteract Neural Collapse exhibit superior performance.
0.42 + 0.02
0.92 + 0.02
84.58 1: 0.40
Reweighting
0.38 + 0.02
0.94 + 0.01
82.62 = 0.44
Resampling
0.38 + 0.03
0.97 + 0.01
82.16 + 0.43
LADM-DRW
0.41 0.02
0.94 0.01
83.86 0.23
Joint-SSL
We conclude that well-tuned training routines can regularize and prevent overfitting minority class training samples without specialized loss functions or sampling methods, which is associated with perfor- mance improvements llass-imbllanced data. See Appendix A.10 for more details.
0.43 + 0.02
0.90 + 0.02
84.80 + 0.43
SAM-A
6
Discussion
While neural network training practices have been studied extensively on baass-balanced benchmarks, real-world problems often involve class imbalance. We have shown that class-inbalanced datasets require carefully tuned batch sizes and smaller architectures to avoid overfitting, as well as specially chosen data augmentation policies, supervisvvi,,,, sharpness- awarr optimizers, and label smoothing. Whereas previous state-of- ffate-of-the-arr works for class- imbalanne focused on specialized loss functions or sampling methods, we show that simply tuning standard training routines can significantly improve performance over such ad hoc approaches.
Our findings give rise several important directions for future work:
We saw that existing methods designed for web-scraped natural image classification bench- marks do not always provide improvements on other real-world problems. If we are to reliably compare methods for class imbalance, we need a more diverse benchmark suite.
Since most real-world datasets are clas--iballanced and architectures designed for class- balanced benchmarks like ImageNet highly suboptimal under class imbalance, perhaps future work should build architectures which are specifically optimized for class imbalance.
The generalization theory literature explains the tradcoff between fitting training samples and the complexity of learned solutions [55]. Can PAC-Bayes generalization bounds explain the large role regularization plays in successful training under class imbalance?
Language models perform classification over tokens, but some tokens occur much less frequently than others. How can we apply what we have learned about training under class imbalance to language models?
Acknowledgements. This work is supported by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764401AI NSF IIS- 1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and Amazon Research Award.
10

---

References
[1] Aptos 2019
https://www www.we.aggee..ooo. aptos2019-blinduss--eed detiressoddeeeiioon.. 2019. Accessed: 10 June 2020.
blindness
detection.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. arXiv preprint arXiv: 10902, 2019.
[3] Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 770-785. Springer, 2017.
[4] Arpit Bansal, Micah Goldblum, Valeriia Cherepanova, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Metabalance: High-performance neural networks for stas--iballaneed data. arXiv preprint arXiv:2106.09e0
2021.
[5] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaace-invvriance-ovvai regular- ization for self-supervised learning. arXiv preprint arXiv:2105.0490. 2021.
[6] Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathe- matical Programming, 167(():235-29))) 2018.
[7] Jock A. Blackard and Denis J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24(3):131-151, 1999, ISSN 0168-1699. doi: https://doi. org/10.1016/S0168 11010001666619999999990909 1699(99)000466, 0. URL https: //wp::.///www// /www. sciencedirect. com/science/
articleeppi1/00111119999000
[8] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-259, 2018.
[9] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with bibelddistribution-aware margin loss. Advances neural information processing systems, 32, 2019.
[10] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16: 321-357, 2002.
[11] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[12] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1659-16077. PMLR, 2020.
[13] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: /80.00000, 2018.
[14] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9268 0008-927,,22 2019.
[15] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- is not imagenet cifar-10. arXiv preprint arXiv: 180000:810000300 2018
[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai I.i, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[17] Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In Workshop on learning from imbalanced datasets II, volume 11, pages 1-8. Citeseer, 2003.

---

[18] Alex Fang, Simon Kornblith, and Ludwig Schmidt. Does progress on imagenet transfer to real-world datasets? arXiv preprint arXiv:2301. 04644, 2023.
[19] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
[20] Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, and Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. International Conference on Learning Representations, 2023.
[21] Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research, 3844-part11:90029177 2010.
[22] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 18932-18943, 2021.
[23] Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991-25004, 2022.
[24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 33:21271-21284, 2020.
[25] Hongyu Guo and Herna L. Viktor. Learning from imbalanced data sets with boosting and data generation: The databoost-im approach. SIGKDD Explor Newsl. 6(1):30-39, jun 2004. ISSN 1931-0145. doi: 10. /11144//007773000077775 URL https://doi. 0rg/10... 1145/1007730. 1007736.
[26] Qishen Ha, Bo Liu, and Fuxu Liu. Identifying melanoma images using efficientnet ensem- ble: Winning solution to the siim-isic melanoma classification challenge. arXiv preprint arXiv:2010.05351,,000 2020.
[27] Hui Han, Wen Yuan Wang, and Bing-Huan Mao. Borderline-smote: new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing. pages 878 887. Springer, 2005.
(28) Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9): 2009.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[31] Dan Hendrycks, Norman Mu, Ekin Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: /2.02781 2019.
[32] Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning to segment the tail. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14045-14054, 2020.
[33] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
12

---

[34] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelligence, 42(1 1):2781-2794, 2019.
[35] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[36] W Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. arXiv preprint 2019.
[37] Kaggle. Outo Group Product Classification Challenge, 2015. URL https://www.kkae com/ com/c/ottee httpt///tttteeeeeee
[38] Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 19X0.09910009111 2019.
[39] Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URL https: peatts::/opeeee /openrevine.... net/forum? ? iimmrigBCVVrv
[40] Jacob Devlin Ming-Wo Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[41] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 160.:0660004866, 2016.
[42] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced elassification via major- to-minor translation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18896-13903 2020.
[43] Ron Kohavi and Mehran Sahami. Error-based and entropy-based discretization of continuous features. In Proceedings of the Second International Conference Knowledge Discovery and Data Mining, KD,'96, page 114-119. AAAI Press, 1996.
[44] Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Con- trasting contrastive self-supervised representation learning pipelines. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9949-9959, 2021.
[45] Vignesh Kothapalli. Ebrahim Rasromani, and Vasudev Awatramani. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.0404), 2022.
[46] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[47] Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https:
/ //app...emataeee www.sticcccohllrrrt wtt.attcccccccooooooo/oorpplllla 16664790.
[48] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nnistl/ 1998.
[49] Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. arXiv preprint ar 220. 2022.
[50] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[51] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to datasct imbalance. In International Conference on Learning Representations, 2021.
13

---

[52] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12009-12019, 2022.
[53] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference compuler vision and pattern recognition, pages 11976-1 1986, 2022.
[54] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[55] Sanae Lotti, Mare Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and An- drew G Wilson. Pac-bayes compression bounds tight that they can explain generalization. Advances in Neural Information Processing Systems, 35:31459-31473, 2022.
[56] Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan, Micah Goldblum, Colin White, et al. When do neural nets outperform boosted trees tabular data? arXiv preprint arXiv:2305.02999) 2023.
[57] Rafael Müller, Simon Komblith, and Geoffrey Hinton. When does label smoothing help?
Advances in neural information processing systems, 32, 2019.
[58] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning- miinn--free yet cate-offthhe-r data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774-782, 2021.
[59] Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 864-873, 2016.
[60] Vardan Papyan, XY Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652-24663, 2020.
[61] Serim Ryou, Seong- -uyng-GGunnnn Jeong, and Pietro Perona. Anchor loss: Modulating loss scale based on prediction difficulty. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5992-6001, 2019.
[62] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[63] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84 2022.
[64] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[65] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106. 01342. 2021.
[66] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Yeh-Chiang Yehuda Dar, Richard Bara- niuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13699-13708, 2022.
[67] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
14

---

[68] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[69] Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NeurIPS 2021 Workshop ImageNet: Past, Present, and Future, 2021.
[70] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[71] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 33:19290-19301 2020.
[72] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6033 2019.
[73] Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
[74] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171000111 2017.
[75] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6489-16499) 2021.
15

---

Appendix Outline
This appendix organized as follows:
In Appendix A. provide more detailed results additional datasets for the different components discussed Section 3. Specifically, in subsections Appendices A. to A.7, we present results batch size, data augmentation, model architectures, pre-training, SSL, Sharpness-Awar Minimization, and label smoothing. These subsections delve into the specific effects and outcomes of each component.
In Appendix A.8, examine the relationship between the training and test distributions in imbalanced training. We explore the optimal balance of training data and discuss the potentially destructive impact of collecting additional majority samples.
In Appendix A.9, we present additional and extended experimental results that compare the methods proposed our paper with the baseline methods.
Appendix A. we provide additional experimental results that illustrate how the training process evolves for imbalanced data.
In Appendix A. 1. we include decision boundary visualizations for imbalanced training. Specifically. demonstrate that Sharpness- Aware Minimization (SAM-A) helps decision regions take up similar volumes, whereas standard training routines tend to shrink-wrap the decision boundaries around minority samples.
In Appendix B, we provide detailed information on the hyperparameters, datasets, and architectures used in our experiments.
In Appendix C, we discuss the limitations of our study. This section addresses potential constraints, challenges, and for improvement in our research.
Lastly, in Appendix D, diseuss the broader impact of our work. This section explores the implications, significance, and potential applications our findings beyond the scope the immediate study.
A Additional Experiments
A.1 Batch Size
To investigate the impact of batch size in the context class imbalance, we train networks across various training ratios using different batch sizes. In order compare the accuracy for each training ratio, we calculate the percentage improvement over the baseline (set as the best batch size of 128). Specifically, if we denote Accf, as the accuracy the imbalanced dataset with training ratio p and batch size b, we define the adjusted accuracy new Acof as
Accf, - -Acefs Acco
Acc,,
Positive values represent higher accuracy compared to the baseline, while negative values denote lower accuracy. This normalization allows us to examine the relative effect of batch size. As shown in the main text and Figure 5, data with high degree of class imbalance tends to benefit from smaller batch sizes, despite the fact that small batches often do not contain any minority samples.
A.2 Data Augmentation
In order to evaluate and compare the effectiveness of various popular augmentation tech- piquus-including horizontal flips, random crops, AugMix [31], Trivial/AgmmentWide [58], and AutoAugment [13]-we investigate their impact the accuracy of minority and majority classes across a range of training ratios.
We measure the relative improvement in performance by comparing the accuracy achieved with data augmentation to that achieved without it. We thus plot the percentage improvement as function of the training ratio in Figure 6.
16

---

0.025
100
Percentage Improvement
1001
Train Ratio
-0.025
-10
-0.05 102
10-3
103 Batch Size
104
Figure Batch matter more for imbalanced data where small batch sizes best, whereas the curve corresponding to balanced data flat. Percentage improvement in test accuracy over the default batch size of 128 at different training ratios. Experiments conducted on CIFAR-10.
Our findings reveal that while the newer TrivialAugment method exhibits superior performance on balanced training data, the older AutoAugment method yields better results on highly imbalanced data.
Aug Mix
40
AutoAugment
Crop Horizontal Flip
Horizontal Flip
% Im provememn
TrivialAugment
20
10-3
10-2
10-1
10
Train Ratio
Figure 6: Optimal augmentations depend on the imbalance ratio. We plot the percent improvement in test accuracy for different augmentations compared to training without augmentations across train ratios for different augmentations. We that Augment,,gment, which known outperform AutoAugment on dlass-balanced data, actually performs worse when data is severely imbalanced. Experiments conducted on CIFAR-100.
A.3 Model architecture
In Figure 8, we illustrate the impact of model size the performance of the CIFAR-10 dataset with a training ratio of 0.001. The trend observed similar to the results discussed the main text, where increasing the model size leads overfitting in the case of imbalanced training.
17

---

Minority Classes
Majority Classes
% Improvement
3
1
10-4
10-3
10-2
10-1
100
Train Ratio
Figure Strong augmentations are particularly effective improving minority class accuracy under severe class imbalance. The pereent improvement in test accuracy of TrivialAugment compared to training without any augmentation as a function of the training ratio. Experiments conducted on CIFAR-10. Error bars represent one standard error over 5 trials.
98
79
Test Accuracy
96
78
77
-44
Imbalanced Data
Balanced Data
76
-92
4
5
6
Number of Parameters
1e7
Figure 8: Bigger architectures overfit class-imbalanced data. Experiments conducted CIFAR-10. Error
bars represent standard error over trials.
A.4 Pre-training
To assess the effectiveness of pre-training, we fine-tune several pre-trained models on downstream datasets with varying training ratios. In addition to the main body, Figure 10 illustrates the percentage improvement in test accuracy compared to random initialization for supervised pre-training on ImageNet-Ik - and ImageNet-21k,
well SimCLR on ImageNet-Ik 1k (which is a Supe-Supee Learning (SSL) method), measured by downstream performance on CIFAR-10. This comparison is made across different training ratios (Figure 4). Let Acc denote the accuracy of the model trained from random initialization at a training ratio p. The relative improvement is then defined by:
Acch, Acce
Acch
(2)
Acc
18

---

92
80
Test Accuracy
79
89
78
85
Imbalanced Data
77
Balanced Data
883
3
4
5
6
Number of Parameters
1e7
Figure 9; Bigger architectures overfit on class-imbalanced data. Experiments were conducted on CINIC-10 with imbalanced train ratio of 0.001. Error bars represent standard error over trials.
Positive values indicate an improvement in performance compared to random initialization. It is clear that all pre-training methods improve performance when compared random initialization. Interestingly, these improvements are significantly more pronounced under imbalanced conditions.
ImagetNet-21k
40
SimCLR
% Improvement
ImageNet-1k
No Pre-Train
20
1e 4
1e 2
1e0
Train Ratio
Figure 10: Pre-training yields bigger improvements on more imbalanced data. improvement the test. accuracy compared to training from random initialization. Experiments conducted on CIFAR-10.
A.5 SSL
Self-supervised learning (SSL) has gained substantial traction as a method of representation learning across multiple domains, including computer vision, natural language processing, and tabular data [12, 40, 65]. Networks pre-trained using SSL often demonstrate more transferable representations than those pre-trained with supervision [24]. Pre-training traditionally consists a two-stage process: initial learning on upstream task followed by fine-tuning a downstream task. However, the limitation many use-cases is the lack of large-scale pre-training datasets. In order to solve this problem, our approach diverges from this two-stage process by merging supervised learning with an
19

---

auxiliary self-supervised loss function during ftom-scratch training, effectively eliminating the need for any pre-training.
For this, we employ the Variance- crriance-nvarranc--Covvaaianee Regularization (VICReg) objective [5]:
Given two batches of embeddings. Z = -f(x1),.... fx))]] and Z' = [f(=)),.... f('3 size (B x K), where 2; and oj are two distinct random augmentations of a sample 4ú we derive the covariance matrix C' E K from [z, Z').
each of
Consequently, the VICReg loss can be articulated as:
K
a max (0,7-VCR, Vk. + +0)+ E(Cneot + + Z'Ii//N.
LSSc=
K
In our experiments, the total loss given by
Jjontt Lsst + Superviised
Note that the SSL loss function is independent the class-imbalanced labels.
A.6 SAM
Sharpness- Minimization [19] is an optimization technique that seeks to find "flat" minima of the loss function, often leading to improved generalization. This method consists of taking an initial ascent step followed by a descent step, aiming to find parameters that minimize the increase in loss resulting from the ascent step. Huang et al. [36] demonstrate that flat minima correspond to wide-margin decision boundaries.
Given model parameterized by weights 0 and loss function L(0) that we aim to minimize, SAM performs two steps in each iteration:
First step (gradient ascent): Perform scaled gradient ascent step from the current model weights 0:
(
8' = - - PLL(012 (V)
2. Second step (weight update): Update the weights from 9 in the negative direction of the gradient computed at the post-ascent parameter vector:
0 = 0 L)))
(4)
In the above steps, 7) represents the learning rate, p is hyperparameter determining the size of the neighborhood around the current weights, and | 1 denotes the Euclidean norm.
SAM was initially developed for balanced datasets, where the decision boundaries for each class have comparable areas. However, this assumption does not hold true for imbalanced datasets. To address this, we adapted SAM for use with class-imbalances datasets by increasing the flatness specifically for minority class loss terms. We propose a new method SAM-Asymmetric (SAM-A), Our method adjusts the ascent step size (p) in SAM's inner loop for minority classes by employing step size inversely proportional to the classes' proportions.
Let be the proportion of class i in the training set. We define the class-conditional ascent step size as:
Pi
(5)
1 Pi
where p scaling factor.
By doing this, we widen the margins around under-representee classes, potentially improving generalization in imbalanced datasets.
20

---

A.7 Label Smoothing
Label smoothing is regularization technique often used in training deep learning models. mitigates the model's excessive confidence in class labels, which can improve generalization and reduce overfitting. However, traditional label smoothing assumes a balanced class distribution, which is not always the case in real-world datasets.
To adapt label smoothing for imbalanced training, propose class-conditional label smoothing technique. Instead using a uniform smoothing parameter c, we use a different e for each class i, which is proportional to the inverse of the class's proportion within the dataset.
Let pi be the proportion of class in the training set. We define the class-conditional smoothing parameter as:
4
1 Ps
where is scaling factor.
We then apply label smoothing as follows. Let p be the model's output probability distribution over K classes, and let be the target distribution for class i. The smoothed target distribution is:
9h (1 ,= K
where j 1,2,..., K, y the true class, and is the indicator function.
During training, we minimize the cross-entropy loss between the model's predictions and the class-conditional smoothed labels qú:
K
By log
By using class-conditional label smoothing, we apply more smoothing to the minority classes and less the majority classes, which can help the model generalize better when the class distribution is imbalanced.
A.8
Data Curation
Common intuition dictates that training data that is more balanced than the testing distribution can improve representation learning by preventing overfitting to minority samples [25, 10, 27]. this section, we put that intuition to the test by examining the optimal balance of training data. Moreover, while minority class samples may be scarce, practitioner may be able to collect additional majority class training samples will, so also examine the potentially destructive impact of collecting additional majority samples.
A.8.1 The Relationship Between Train-Timm and Test- Tist-Time Imbalance
The literature on training routines for class imbalance in machine learning is filled with methods designed for scenarios which training data highly imbalanced but testing data balanced. However, data encountered during deployment is typically also imbalanced. Therefore, we disentangle training and testing balances and investigate how sensitive models are to discrepancies between the two. This study may be particularly important if one considers collecting training data for downstream application. Should we gather training data with the same balance we anticipate during testing? How worried should we be if the data we encounter during deployment is more or less balanced than the training data we gathered?
We begin by illustrating three scenarios in Figure 11: (1) identical training and testing ratios, (2) balanced training, and (3) the training ratio with the lowest test error (optimal training ratio). see that training on data with the same imbalance the testing data is superior to training balanced data, and the two strategies only approach equal performance when the testing data becomes balanced. We share additional results over different datasets and models in Figure 20, Figure 21, and Figure 22.
21

---

We then plot for each test ratio the corresponding train ratio that results in the lowest test error in Figure 12. If the two ratios are perfectly aligned, then points will lie on the diagonal. Indeed, the points are close to the diagonal, indicating that it is best to train with very similar imbalance ratio to the test dataset, especially for highly imbalanced testing scenarios.
In these previous experiments, we fixed the size of the training set, but what happens as we gather more and more training data? In Figure 15, we train and evaluate network on different imbalance ratios across training set sizes, and we plot the misalignment between the train and test ratios, referring the average distance between the optimal train ratio and the specified test ratio. As the amount of training data increases, we see that the optimal training ratio becomes more and more close to the ratio of the test data.
0.34
Test Error
Test/Train Alignment
0.28
Balanced Training
Best Train Ratio
0.22
10-2
10-1
Test Ratio
Figure 11: Imbalanced training data is optimal imbalanced testing scenarios. Test accuracy as function of the test ratio for different training setups. Experiments conducted on CIFAR-100.
A.8.2 When More Data Degrades Performance
In practice, a practitioner may not have precise control over the data they collect. Will collecting additional samples always help performance? Instead of fixing the total number of samples and
X=Y
Best Train Ratio
10-1
10-2
10-3
10-3
10-2
10-1
Test Ratio
Figure 12: The optimal train ratio is closely aligned with the test ratio. Experiments conducted on CIFAR- 100.
22

---

X=y
Best Train Ratio
10-1
10
10"3
10-3
10-2
10-1
Test Ratio
Figure 13: The optimal train ratio is closely aligned with the test ratio. Experiments conducted on CINIC-10.
Equal
Balanced
Optimal
Test Error
10-2
10-1
Test Ratio
Figure 14: Test accuracy on minority classes as function of test ratio for different training setups. 'Equal' denotes the same balance between training and testing, and 'Optimal' is the optimal trainset balance amongst the ratios we try. Experiments conducted CIFAR-10.
Train Train/Test Test. Misalignment
CIFAR-10 10
CIFAR-100
0.4
0.2
0.1
102
103
101
Train Samples
Figure 15: Alignment between train and test proportions improves as the number of training samples increases. Train/test misalignment is calculated by taking the mean over test ratios of the difference between the best train ratio (train ratio that gives maximum test accuracy) and the test ratio. If misalignment is 0, then the best train ratio is always the same the ratio.
23

---

2.5
Test Error
1.5
0.9
102
103
Number of Majority Samples
Figure 16: The potentially destructive effects of adding majority class samples. We fix the number of minority samples 200 and vary the number majority samples. Experiments conducted on CIFAR-100.
Table 5: Our training routines exceed previous state-of-the-art improve existing methods when combined. Split class accuracy classes Few, Med and Many examples of WideResNNet284 on long-tailed CIFAR 100 and CINIC-10. Error bars correspond to one standard error over trials.
CINIC-10
CIFAR-100
Method
Few
Med
Many
Few
Med
Many
ERM
40.5 0.4
64.1 0.3
90.1 i 0.5
20.1 0.3
42.3 +
70.5 + 0.6
Reweighting
36.6 1: 0.5
63.1 + 0.3
87.8 1 0.3
17.1 0.4
39.3 1 0.3
67.1 1: 0.4
Resampling
37.4 1. 0.5
63.6 1 0.6
87.9 1 0.4
18.4 1 0.2
38.1 1 0.2
68.9 1 0.3
Focal Loss
39.1 0.2
63.9 1 0.2
88.2 1 0.5
19.8 0.4
39.0 1 0.5
69.3 1 0.6
LDAM-DRW
40.1 0.4
64.3
89.8 4 0.3
20.8 0.5
42.1 + 0.3
70.6 0.4
M2m
42.8 0.7
64.1 + 0.6
90.3 + 0.4
20.1 0.6
41.8 + 0.4
69.4 + 0.5
SAM-A
43.2 0.3
62.3 + 0.6
89.7 +.3++... 0.3
22.5 + 0.4
40.3 + 0.3
70.1 0.4
Joint-SSL + SAM-A
43.9 + 0.4
63.3 + 0.5
90.4 0.5
22.9 + 0.3
41.3 + 0.6
69.9 + 0.6
Joint-SSL-
44.1 0.3
64.2 + 0.4
90.9 + 0.3
23.9 0.4
42.3 + 0.2
70.4 + 0.3
SAM-A + M2m
varying their imbalance ratio, fix the number of samples from the minority class and vary the number of total majority class samples.
In Figure 16, we see that increasing the number of samples from the majority class initially boosts performance on a balanced test set. Nevertheless, in both cases, the performance reaches an optimum before the growing training data imbalance eventually degrades test aceuracy. Thus, adding training data can help, but if we add enough majority samples, we must be careful not to cause too sharp a mismatch between training and testing distributions. Notably, the optimal training set ratio is nearly balanced, matching the test set, even when we are allowed to gather extra samples from one class without having forego samples from another.
A.9 Benchmarking Results
In Table 5, we present additional experimental results that compare the methods proposed in our paper with the baseline methods. In accordance with Kang al. [38], we also report the accuracy across three distinct subsets: (1) Many-shot classes, which contain more than 100 training samples.
(2) Medium-shot classes, comprising 20 to 100 samples, and (3) Few-shot classes, including classes with fewer than 20 samples.
24

---

Table 6: SAM-A, modified label smoothing, and small batch sizes improve performance class- imbalanced tabular datasets.
Method
Otto
Adult
Coverlyype
XGBoost
82.7
87.5
96.9
ML.P
83.0
87.4
97.5
ResNet
82.5
87.4
97.5
FT-Transformer
82.3
87.3
97.5
MLP w/ SAM-A
83.2
87.6
Smoothing
A.9.1 Tabular Datasets
Training procedure Our tuning, training, and evaluation protocols are consistent with those in Gorishniy et al. [23]
Accordingly, we each model's hyperparameters for each dataset a validation split. We the Optuna library [2] to execute Bayesian optimization via the Tree-Structured Parzen Estimator algorithm. In our evaluation process, we run 15 experiments for each tuned configuration using different random seeds, and report the average performance on the test set.
A.10 Regularization and Overfitting
In order to determine whether the performance differences among various methods from their optimization abilities or their generalization to unseen test samples, we evaluate the training error without any regularization or specialized optimization method. Specifically. network on CIFAR-10 10 and CIFAR- 100 datasets using SGD with an initial learning rate of 0.5 and cosine annealing, across different levels of training data imbalance. As seen in Figure 17, although fitting all training examples takes longer as we increase the imbalance ratio of our datasets. the empirical risk minimization successfully fits all training data eventually, including minority samples.
train a ResNet-50
1.0
Train Accuracy
0.8
P 1.0
0.7
- 0=0.1
0=0.01
0.6
0.0.001
0=0.0001
0.5
200
400
600
800
1000
1200
Epoch
Figure Imbalanced data harder fit. Training accuracy every epoch for imbalanced training with various imbalance ratios. Experiments conducted on CIFAR-I0.
A.11 Decision Boundary Visualizations
To explore the differences between classifiers trained on imbalanced data, we visualize their decision boundaries. A variety of methods have been established for visualizing the decision boundaries of deep learning models, offering valuable insights into their intricate internal operations. Apart from the methods discussed in the main text, we utilize the approach introduced by Somepalli et al. [66] to visualize the decision boundaries of ResNet-50 network trained on the CIFAR-10 dataset. In Figure 19, display the decision boundaries resulting from standard training (right), which yiclds
25

---

(a) After naive training
(b) After SAM training
Figure SAM-A pulls decision boundaries away from minority samples, whereas standard training routines overfit. Experiments conducted on a toy 2D classification problem with a lulti perceptron.
(a) After naive training
(b) After SAM training
Figure 19: SAM-A makes decision regions take up similar volumes, whereas standard training routines shrink wrap the decision boundaries around minority samples. Experiments conducted CIFAR-IO with ResNet-18.
narrow margins around minority classes (green, grey, and orange), and SAM-A (left), which notably broadens these margins and all the classes occupy similar area in input space.
B Experimental Details
In this section, we provide additional implementation details that included in the main text. For the CIFAR-10 10, CIFAR-100, and CINIC-10 datasets, we follow the imbalanced setup proposed by Liu et al. [54], using exponential distribution to create imbalances between classes. Across all methods, we use TrivialAugment [58] combined with CutMix as our augmentation policy, supple- mented by label smoothing and exponential moving weight average. Our model of choice is the WideRessNt-288: 10 [73].
We employ the SGD optimizer with momentum 0.9 and weight decay coefficient 2 x 10 10-4, Our models are trained for 300 epochs with cosine annealing and a linear warm-up of the learning rate. The learning rate is initialized at 0.1.
For the APTOS 2019 Blindness Detection, SIMM-ISIC Melanoma Classification, and EuroSAT datasets, we largely follow the approach detailed in Fang et al. [18], utilizing the ResNeXt-50-322 model, which was identified as the best model for these datasets in the comparison by Fang al. [18).
Our implementation done in PyTorch, utilizing the PyTorch Lightning library for training. All of our models were trained on V100 GPUs.
C Limitations
In our paper, we found that existing methods for class imbalance are unreliable on real-world datasets. While our tuned routine was effective on the real-world datasets considered, these general trends
26

---

Equal Train/Test Ratio
Balanced Training
Best Train Ratio
10
Majority Test Eiror
Majority Test Error
10
16*2
10'
10
10"
10"
10
10
10'
10
10
10"
10
10"
10'
Test Ratio
Test Ratio
Ratio
(a) ResNet on CIFAR-IO Dataset (b) XGBoost on Adult Dataset (c) SVM Forest Cover Dataset
10
Minority L'stt Error
30'
Minority Tast Eroor
10'
10
10
10
10
10
10
Test Ratio
Test Ratio
Test Ratio
(d) ResNet on CIFAR-IO Dataset
(e) XGBoost on Adult Dataset (I) SVM on Forest Cover Dataset
Figure 20: Test split by majority and minority classes for balanced test sets. We see similar trends across all models and datasets.
Equal Train/Te Test Ratio
Balanced Training
ROC ADC
Mecal
Ratio
Ratio
Rutio
Ratio
Figure 21: Additional metrics for XGBoost the Adult dataset.
Equal Train Test Ratio
Balanced Training
107
167
ROCADC tn11
Pooton
10
Keah
16'
10*
Tol Ratio
Text Rusio
Teat Rutio
Tox Raiio
Figure 22: Additional metrics for SVM on the Forest Cover dataset.
27

---

raise the concern that solutions which are effective on some class-imbaland datasets may fail on others. second limitation of work is that some tools we utilize are only applicable in certain domains. For example, data augmentations and self-supervised learning for tabular data are not widely accepted.
D Broader Impacts
Across a wide variety of high-impen domains, ranging from credit card fraud detection to disease diagnosis, data is severely class-imbalanced. Therefore, performance increases for class-imbalanced data is highly valuable. With this potential for value also comes the potential that proposed methods make false promises which won't 't benefit real-world practitioners and may in fact cause harm when deployed in sensitive applications. For this reason, release our numerical results across diverse datasets, and we also include implementation details for the sake of transparency and reproducibility. As with all new state-of-the-art methods, improvements may also improve models used for malicious intentions.
28


Reviews:
[{"review_id": "VtRL1WfDoe", "reviewer": "Reviewer_kUXT", "summary": "The authors suggest tackle the class imbalance issue from a hyperparameter optimization perspective. Throughout an extensive empirical study, they raise questions about the behavior of well-established techniques for balanced data under long-tail data distribution.  The synergy of the resulting prescriptions is also checked on several datasets", "questions": "From one side, the batch size should not grow to much but on the other side, applying a SSL helps.  It is known that SSL models tend to perform better for larger batch sizes and balanced datasets. Does this rule of thunb apply as well for unbalanced data?", "limitations": "none", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper introduces a new approach to tackle the class imbalance issue by optimizing several hyperparameters. The paper is clearly written,  The claims are  well-stated and sufficiently  supported empirically in most of the cases. \n* The authors tried to sketch intuitive explanations for some 'unexpected' outcomes \n* The authors reported the performance of several combinations of their micro-receipes and show strong results on several datasets.", "weaknesses": "* The content of this paper is clearly going beyond the maximum allowed number of pages.  To not violate the rules, the authors decided to move very relevent parts of the paper to the appendix which makes the paper less readable. Nevertheless, I insist that this action does not compromise the clarity. I would have been nicer to cut the less important components and limit the scope of the paper to the most important hyperparameters.    \n* The novelty of this paper is limited some 'findings' are valid for the balanced setup as well (pretraining, SSL) or just trivial (AutoAugment) \n* For a purily empirical paper, it is important to diversify the architectures in order to draw more valuable conclusions. I would have loved to see the performance of a transformer based model under different contraints\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer  kUXT,\n\nThank you for your detailed and thorough evaluation of our submission.  We appreciate your comments and address your concerns and questions below.\n\n## Page Limit and Readability: \nWe acknowledge your concern about the content exceeding the page limit and agree that moving some parts to the appendix might have affected the readability. Our intention was to provide as much detail as possible to support our claims, while adhering to the page limit. We understand that some parts of the paper might be less critical to the central argument and can be omitted or simplified. We have now split Section 4 into two sections based on the types of methods: (1) the first ones (batch size, data augmentation, pre-training, model architecture, optimizers), which serve as the fundamental 'building blocks' of regular balanced training. For this group, we investigated how altering their hyperparameters impacts the performance of imbalanced training. (2) The second ones are optimization methods that we slightly modified to address imbalanced training (Joint-SSL, SAM, and label smoothing). We examined the effects of these methods on performance only in the next section. Furthermore, we are revising the paper to improve the balance between detailed explanation and readability, focusing on the most crucial aspects of our approach. Additionally, we are open to revising the paper even further to improve the balance between detailed explanation and readability.\n\n## Novelty: \nWe believe the value of our paper lies in the new perspectives we provide on class-imbalanced training. We demonstrate the benefits of our simple and easy-to-use tuned or imbalance-adapted training techniques, specifically in the context of class imbalance, which was previously dominated by specialized loss functions or samplers designed specifically for class imbalance. Our work represents a perspective-shift on training under class imbalance, but moreover, we achieve significant performance gains across multiple benchmark and real-world datasets.\n\n## Diversity of Architectures:\nOur submission focused on traditional architectures due to their prevalent use in the class imbalance literature. However, we agree that the analysis would be enriched by considering more architectures like transformers. Prompted by your feedback, we have now added additional experiments. We ran our analysis on more models - Swin Transformer v2 [1] and ConvNeXt [2]. These new models achieve similar behavior to the current models. (see Table 1 on the attached page).  Moreover, we expanded our analysis to tabular datasets. These datasets represent a challenging frontier for deep learning research. Despite tabular datasets often inherently exhibiting imbalance, limited research has addressed the impact of imbalanced data on deep learning in this domain. We incorporate our suggested methods (SAM-A, modified label smoothing, and SGD with cosine annealing performed with a small batch size)  to three tabular datasets -  the Otto Group Product Classification [3], Covertype [4], and Adult [5] datasets using a Multilayer Perceptron (MLP) with improved numerical feature embeddings. Our methodology outperforms both XGBoost[REF] and recent state-of-the-art neural methods such as XGBoost, ResNet, and  FT-Transformer [6]  on all three datasets, demonstrating the applicability of our findings beyond image classification.  See the full results in Table 2 on the attached page). These additional results showcase the robustness of our findings and further substantiate our conclusions. \n\n\n## Question on Batch Size and SSL: \nYour observation about the relationship between batch size and SSL is indeed intriguing. SSL models tend to perform better with larger batch sizes under balanced datasets. However, the dynamics change when it comes to imbalanced data. Our study on batch size provides both novel empirical results and intuitive reasoning. Specifically, we found that smaller batch sizes prevent overfitting on minority classes, acting as a form of regularization in SGD (see [7] for an analysis of the role of batch size as a regularizer for the loss landscape which encourages generalization). This effect introduces noise into gradient estimation, smoothing the loss landscape, and counteracting the tendency to overfit to minority examples. This suggests that the rule of thumb for SSL models might not directly apply to imbalanced data. To test this, we trained our joint-SSL algorithm using different batch sizes on CIFAR-10. As can be seen in Figure 1 on the provided page, the effect of the batch size is less significant, with joint-SSL behaving as an alternative regularizer, preventing the overfitting of the minority classes.\n\nThank you again for your thoughtful review. We made an effort to address your feedback including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References: \n[1] Liu et al., 2021, \"Swin transformer v2: Scaling up capacity and resolution\".\n\n[2] Liu et al., 2022, \"A convnet for the 2020s\".\n\n[3] Kaggle, 2015, \"Otto Group Product Classification Challenge\".\n\n[4] Blackard and Dean, 1990, \"Predicting forest cover types\".\n\n[5] Kohavi and Sahami, 1996, \"Error-based and entropy-based discretization\".\n\n[6] Gorishniy et al., 2021, \"Revisiting deep learning models for tabular data\".\n\n[7] Sekhari et al., 2021. \"SGD: The role of implicit regularization, batch-size, and multiple-epochs\".\n"}, {"review_id": "3Ck0WLmzMY", "reviewer": "Reviewer_8NJC", "summary": "In this paper, the authors study the problem of class imbalance. To be specific, they first investigate the effects of different hyper-parameters & design choices in an imbalanced setting. Moreover, the authors use such optimized settings with existing methods to show that significant improvements can be obtained.\n\nAfter the rebuttal:\nThe authors addressed my concerns with detailed explanations and additional experiments. With those, I believe the paper provides useful insights for people working on class imbalance, therefore, I vouch for the paper's acceptance.", "questions": "See above.", "limitations": "None.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Easy to follow text.\n2. Class imbalance is an important problem and therefore, analyses that offer more insights are valuable.", "weaknesses": "1. The most important issue I see with the paper is that its contribution is limited. \n\n1.1. Out of the 6 Lessons (take-away points) in Introduction, Lessons 2-6 are already known in the literature. Only the analysis on batchsize is novel (as far as I know) and the results are intriguing.\n\n1.2. The application of the tuned setting with the other methods does not offer any contributions. \n\n2. More challenging datasets such as ImageNet-LT, iNaturalist are missing.\n\n3. Regarding Lesson 1:\n\n3.1. Lesson 1 (batchsize): The paper does not provide any intuition as to why batchsize has such an effect in an imbalanced setting.\n\n3.2. Lesson 1: It would be worthwhile to see the same analysis with different architectures because, as we see in Figure 3, different architectures exhibit different behaviors under imbalance.\n\n4. If I may, I suggest the authors to focus only on batchsize and provide solid & theoretical insights about why/how it affects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We address your points below.\n\n## Contribution:\nWhile the methods we examine have been explored in the context of balanced training, our unique contribution lies in the in-depth analysis of these methods within the challenging environment of class imbalance. Our investigation resulted in an imbalance-specific pipeline which greatly improves the performance of existing methods. Our approach offers actionable insights for practitioners and enriches the understanding of how to approach class imbalance problems.\n\n- For Lesson 2, while several methods have suggested tailored data augmentations for training with imbalanced data [1,2], to our knowledge, there has been no comprehensive analysis of how high-performance augmentations designed for balanced training affect minority classes. Importantly, previous works suggest using new, complex data augmentation techniques to deal with class imbalance. In contrast, our analysis suggests that existing data augmentation techniques can achieve state-of-the-art results.\n\n- For Lesson 3, some works have demonstrated that increasing model size [3, 4] improves performance across many datasets in the balanced learning scenario. However, no work has analyzed the effect of model size on imbalanced data and its significant differences from balanced training.\n\n- For Lesson 4, several works have proposed SSL pre-training for class imbalance [5, 6, 7]. Yet, none have suggested training with the same data, without the need for additional data, by integrating supervised training with SSL training. We show that this technique has major performance benefits.\n\n- For Lesson 5, [6] suggests using SAM directly to combat imbalance without an optimal perturbation per class. In contrast, our modifying SAM treats majority and minority classes differently, adjusting the decision boundaries only for minority classes. Notably, parallel to our work, [8] suggested a more complex version of SAM that requires a two-stage process.\n\n- For Lesson 6, [9] found that applying the same smoothing to all classes is ineffective for imbalanced datasets. However, our suggested method applies different degrees of smoothing to different classes based on imbalance. Namely, we apply more smoothing to the minority classes.\n\nAs detailed above, we are unaware of works that encapsulate these lessons. Can you please suggest references for lessons 2-6 already known in the literature?\n\n## More Challenging Datasets and Models:\nPrompted by your feedback, we have expanded our experiments and analyses to Tiny-ImageNet and iNaturalist, and to Swin Transformer v2 and ConvNeXt.  Our suggested methods surpass SOTA imbalanced methods on these datasets.  Moreover, these new architectures exhibit similar behavior to our previous experiments. We have also extended our analysis to tabular datasets, with our methodology outperforming SOTA methods. Please refer to the general rebuttal for full details.\n\n## Batch Size\n- **Insights on Batch Size:** Our study on batch size provides novel empirical results and intuitive reasoning. Specifically, we found that smaller batch sizes prevent overfitting, acting as a form of regularization in SGD (see [10] for the role of batch size). This effect introduces noise into gradient estimation and counteracting the tendency to overfit. Our findings pave the way for a new understanding of imbalanced data and offer practical guidance for selecting appropriate batch sizes.\n\n- **Different Architectures for Lesson 1:** We agree with your point and have expanded our analysis to include more models.\n\n- **Focus on Batch Size:** We appreciate your suggestion to concentrate solely on batch size. While we recognize the importance of batch size exploration, our paper aims to provide a holistic view of the class imbalance problem and associated practical recommendations. Our wide-ranging investigation, encompassing various hyperparameters and design choices, offers unique insights and caters to various real-world scenarios, offering large performance benefits for practitioners. To delve deeper into this issue, we examine its interaction with the joint-SSL method. While SSL models typically perform better with larger batch sizes for balanced datasets, our study, suggests conventional wisdom for SSL models may not apply in this setting. We trained our joint-SSL algorithm with different batch sizes on CIFAR-10 to verify this. As seen in Figure 1 of the provided page, the impact of batch size is less pronounced, with joint-SSL acting as an alternative regularizer. While we are dedicated to further exploring the role of batch size in future work, we continue to value the multifaceted contributions of our current paper.\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple new experiments and paper edits, and we would appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References:\n[1] Dablain et al., 2023. Efficient augmentation for imbalanced deep learning.\n\n[2] Temraz et al., 2022. Solving the class imbalance problem using a counterfactual method for data augmentation.\n\n[3] Goldberg et al., 2020. Rethinking fun: Frequency-domain utilization networks.\n\n[4] Zhai et al., 2022. Scaling vision transformers.\n\n[5] Kotar et al., 2021. Contrasting contrastive self-supervised representation learning pipelines.\n\n[6] Liu et al., 2021. Self-supervised learning is more robust to dataset imbalance.\n\n[7] Yang et al., 2020. Rethinking the value of labels for improving class-imbalanced learning.\n\n[8] Zhou et al., 2023. Class-Conditional Sharpness-Aware Minimization for Deep Long-Tailed Recognition.\n\n[9] Shen et al., 2021. Is label smoothing truly incompatible with knowledge distillation: An empirical study.\n\n[10] Sekhari et al., 2021. SGD: The role of implicit regularization, batch-size, and multiple-epochs.\n\n\n\n\n"}, {"review_id": "ooNeg3OFCF", "reviewer": "Reviewer_D6vk", "summary": "This paper presents different approaches to enhance the performance of neural network classifiers over imbalanced datasets. Unlike most research focusing on specialized loss functions or resampling techniques, this study promises that state-of-the-art performance is achievable over the neural classifiers by simply tuning existing components of standard deep learning pipelines. Section three suggests that the study is evaluated over six image datasets and two datasets from UCI Machine Learning Repository, validated over ResNets and WideResNet classifiers, and compared with nine standard baselines. The reported evaluation measures include overall test accuracy and minority and majority class accuracy over the 20% of classes with the smallest and highest number of samples. The key observations from the study are -- (a) small batches perform better for the imbalanced data, (b) the role of augmentation as a regularizer is undeniable, but the performance over the minority class is sensitive to the chosen augmentation policy, (c) the larger networks are more susceptible to overfitting minority class in the case of imbalanced data, (d) models pre-trained on larger datasets tends to perform well in this case, (e) the integration of self-supervised loss with the supervised learning referred to as Joint-SSL performs well, (f) Sharpness-Aware Minimization (SAM-A) pulls decision boundaries away from minority samples, (g) whereas standard training routines overfits, and (h) applying more smoothing to minority class examples than majority class examples prevents overfitting on minority samples. Furthermore, integrating these findings with Joint-SSL and SAM-A atop M2m (SOTA) establishes new state-of-the-art performance across all (class-imbalanced) benchmark datasets, CIFAR-10, CIFAR-100, and CINIC-10. Some other important observations made in the study are (i) SGD optimizer performed uniformly better on imbalanced data, (ii) training on data that is more balanced than the testing distribution did not improve representation learning by preventing overfitting to minority samples, (iii) a low correlation was observed between the performance over the web-scrapped (benchmark) datasets and real-world datasets, and (iv) a correlation was observed between neural collapse and low test accuracy. Overall, this study emphasizes that simply tuning standard training routines can significantly improve performance in the case of imbalanced datasets.", "questions": "1. How does investigating train and test sets with varying imbalance ratios help?\n2. Why are ResNets and WideResNet considered for experimentation out of so many neural network architectures? \n3. Can the same behavior be expected in the case of vanilla neural networks and in the case of CNN, LSTM, and BiLSTM?", "limitations": "1. The paper does not propose the class imbalance issue in general.\n2. The paper references other research work for experimental setup in many instances, which could have been avoided.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This study approaches the significant and challenging problem of imbalanced learning.\n2. The paper does extensive study and finds that state-of-the-art performance is achievable by finetuning the different components of existing deep learning pipelines. \n3. It is interesting to investigate the imbalance problems using the proposed solutions, which are easy to understand.", "weaknesses": "1. The paper has bold statements at the beginning (resolving class imbalance in general) but has yet to explore the class imbalance issue in textual datasets. \n2. The evaluation metrics used to assess the proposed solution revolve around accuracy.\n3. Comparison with SMOTE is not reported but is mentioned as a baseline.\n4. The dataset statistics, reflecting the nature of the imbalance in the dataset, need to be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer D6vk,\n\nThank you for your thorough and insightful review of our submission. We appreciate the time you've dedicated to evaluating our work and providing feedback. We address your points below:\n\n## Exploration of Class Imbalance in Other Data Modalities:\nWhile our study primarily focuses on image datasets, which present significant class imbalance problems, we agree that the issue is applicable to other modalities as well. Prompted by your feedback, we have now expanded our analysis to tabular datasets, which represent a challenging frontier for deep learning research. Despite tabular datasets often inherently exhibiting imbalance, limited research has addressed the impact of imbalanced data on deep learning in this domain. We incorporate our suggested methods (SAM-A, modified label smoothing, and SGD with cosine annealing performed with small batch size) to three tabular datasets -  the Otto Group Product Classification [1], Covertype [2], and Adult [3] datasets using a Multilayer Perceptron (MLP) with improved numerical feature embeddings. Our methodology outperforms both XGBoost and recent state-of-the-art neural methods such as ResNet and  FT-Transformer [4] on all three datasets, demonstrating the applicability of our findings beyond image classification.  See the full results in Table 2 on the attached page. These additional results showcase the robustness of our findings, and we will include these updates in the camera-ready version.\n\n## Evaluation Metrics:\nThe two most commonly used evaluation metrics under class imbalance are general accuracy, due to its simplicity and direct interpretability, and class-wise accuracy, where classes are grouped by the number of examples within each class [5,6]. These metrics provide comprehensive information about model behavior across different classes. In our work, we have reported these metrics both in the main text (for instance, in Figure 2) and in the appendix. \nPrompted by your feedback, we have also incorporated the test likelihood error, which exhibits behavior similar to the aforementioned metrics. The inclusion of this metric provides an additional dimension to our evaluation, helping to assess the model's probabilistic performance on test data. \n\n\n## Comparison with SMOTE:\nThanks for pointing out our oversight in not including SMOTE in the manuscript. Our study aims to compare our findings with relevant baselines to provide a comprehensive evaluation, and we will include SMOTE in our camera-ready version.\n\n## Dataset Statistics:\nWe agree that detailed dataset statistics would be useful to include, so we have updated our draft accordingly.\n\n\n## Investigating Train and Test Sets with Varying Imbalance Ratios:\nIn many real-world applications of machine learning, practitioners can curate data in a controlled fashion.  Practitioners may choose, for example, to curate extra minority class samples or to curate a training set which reflects the balance they anticipate in the inference-time data.  In this regard, the level of training imbalance can be a controllable hyperparameter, making it important to understand its effect on performance.\n\n## Choice of ResNets and WideResNet:\nWe selected ResNets and WideResNet because they frequently appear in existing works involving class-imbalanced problems [7,8]. Prompted by your suggestion to explore other types of architectures, we have now conducted additional experiments using Swin Transformer v2 [9] and ConvNeXt [10]. These new models exhibit behaviors similar to the existing models, as displayed in the attached page. Additionally, we have now run various models on three tabular datasets[1-3], including MLP, FT-Transformer[4], ResNet, and XGBoost.\n\n## Addressing Class Imbalance in General:\nWhile our study provides numerous insights and methods to handle class imbalance, we agree that it is a broad and complex topic that cannot be fully addressed in a single paper. \nNevertheless, we believe our work contributes valuable pieces to the puzzle of handling class imbalance and provides a robust foundation for future investigations.\n\n## References to Other Works for Experimental Setup:\nWe referenced other works for the experimental setup to provide readers with more context and to ensure reproducibility. We understand your comment about excessive referencing and will strive to strike a better balance in the camera-ready version.\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback, including multiple experiments and paper edits, and we would appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References:\n[1] Kaggle, 2015, Otto Group Product Classification Challenge.\n\n[2] Blackard and Dean, 1990, Predicting forest cover types.\n\n[3] Kohavi and Sahami, 1996, Error-based and entropy-based discretization.\n\n[4] Gorishniy et al., 2021, Revisiting deep learning models for tabular data.\n\n[5] Cao et al., 2019. Learning imbalanced datasets with label-distribution-aware margin loss. \n\n[6] Cui et al., 2019. Class-balanced loss based on effective number of samples.\n\n[7] Dablain et al.,  2023, Efficient augmentation for imbalanced deep learning.\n\n[8] Johnson et al., 2020, Survey on deep learning with class imbalance.\n\n[9] Liu et al., 2021, Swin transformer v2: Scaling up capacity and resolution.\n\n[10] Liu et al., 2022, A convnet for the 2020s.\n\n\n\n\n"}, {"review_id": "9VP1g5h1Ge", "reviewer": "Reviewer_nEkH", "summary": "The paper studies the long-tail recognition problem and the impact of existing components of standard deep learning pipelines on the generalization performance, such as the batch size, data augmentation, architecture size, pre-training, optimizer, and label smoothing. They find that simply tuning those components can achieve state-of-the-art performance without any specialized loss functions or samplers.", "questions": "Please see questions in Weaknesses.", "limitations": "- Since ImageNet has also been used as a standard dataset for long-tailed recognition for quite long time, I wondered whether the authors had any results on that to verify if their findings scale beyond CIFAR-10/100.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Rethinking the effect of existing components in long-tailed recognition is a very important research topic.\n- It is interesting to show that the minority class performance decreases at some point with the increase of the model size for imbalanced data.\n- Extensive experiments are conducted on small-scale datasets.", "weaknesses": "- The batch size experiment in Fig. 1 is not very convincing. What is the training method used? Is it ERM? Does this result hold across different methods? If it is ERM, I wonder what if you do cRT afterwards, will the same conclusion (data with a high degree of class imbalance tends to benefit from smaller batch sizes) hold?\n- For data augmentation, it is also not clear what method is evaluated. Besides, data augmentation improves more on the minority classes is not new knowledge IMHO. There are even papers designing augmentations that strengthen the minor classes specifically, e.g. MFW [1] and TFE [2]. Also, to what extent does the claim \"AutoAugment emerges as the most effective for imbalanced data\" hold? Does it hold across different datasets, architectures, and training methods? From Fig.7, it is very convincing if AutoAugment outperforms TrivialAugment significantly since the error bar is not reported. \n- Model architecture: I'm not sure that balanced and imbalanced accuracy are \"virtually uncorrelated\" is well supported as the performance difference between methods in Fig.3 right is very small. How stable are the results in Fig.3 right? \n- What was the performance of ERM without the \"tuned routines\"? What is the performance improvement of each component in the refined ERM (batch size and data augmentation)? I think showing this ablation can be very helpful in understanding the paper.\n\n[1] Procrustean training for imbalanced deep learning, ICCV 2021\n\n[2] Co-Learning of Representation and Classifier for Imbalanced Semi-Supervised Learning, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nEkH,\n\nThank you for your thoughtful feedback, and we appreciate the time you took to provide your insight. We address your points below:\n\n## Batch Size Experiment:\nThe training method used in Figure 1 is ERM. Following your comment, we found that cRT benefits from smaller batch sizes, whereas MiSLAS, a method designed for imbalance, benefits less (Figure 1 on the attached page). MiSLAS prevents overfitting to the minority class, acting as a regularizer.\nSmaller batch sizes improve performance by acting as regularization in SGD, preventing overfitting to minority classes (see [1] for a theoretical analysis of batch size as a regularizer). This effect introduces noise, smooths the loss landscape, and is not prevented by cRT but by MiSLAS, reducing batch size's impact. These findings emphasize the complex relationship between batch size and training methods, offering insights for imbalanced data training. To further explore, we examined joint-SSL with CIFAR-10, showing that the dynamics change with imbalanced data. The conventional wisdom for SSL models may not directly apply, and batch size's impact is less pronounced with joint-SSL as an alternative regularizer, mitigating the \noverfitting of minority classes.\n\n## Data Augmentation:\nWe used ERM when we evaluated the data augmentation methods. We found that AutoAugment was most effective for imbalanced data for CIFAR10, CIFAR100, and CINIC-10 but not on balanced data. We appreciate your point about the error bars in Figure 7 and have updated our working draft for clarity.\n\nRegarding the specific augmentation methods you mentioned (MFW and TFE), these are indeed valuable techniques. However, our focus was on analyzing the effect of standard, commonly-used high-performance augmentations in the context of imbalanced data rather than specialized augmentations designed for imbalanced data. Our findings provide useful insights for practitioners using standard augmentation techniques in their pipelines.\n\n## Model Architecture:\nBased on your comment, to check how stable the results are, we conducted additional experiments with different random seeds to assess the stability of our results. Specifically, we ran each model configuration with five seeds and added error bars to our figures representing one standard error. The resulting figure, which can be seen on the additional page, shows that the error bars are relatively small compared to the differences between the models, indicating that our results are indeed stable across seeds. Interestingly, although the differences in performance between models are small, there is no clear correlation between balanced and imbalanced performance across the models. This further emphasizes the complexity of the class imbalance problem and the importance of considering both balanced and imbalanced performance when designing models.\nWe believe these additional experiments strengthen our findings.  We have added them to our working draft and will include them in the camera-ready version. \n\n## Performance of ERM and Improvement of Each Component:\nWe concur with your suggestion that an ablation study would be beneficial in demonstrating the contribution of each component in our approach, so we have now added it to the table on the attached page. In general, without our proposed \"tuned routines\", the performance of ERM is somewhat lower than that of reweighting methods. Breaking down the impact of individual components, we find that data augmentation is the primary contributor to performance improvement, while the influence of batch size, though still beneficial, is less pronounced.\n\n## More datasets:\nWe agree with your comment about adding more datasets. Prompted by your feedback, we have now expanded our experiments. We now include more challenging visual datasets, namely Tiny-ImageNet[2] and iNaturalist[3].. As can be seen in Table 1 on the attached page, our finely-tuned training routine, equipped with Joint-SSL, Asymmetric-SAM, and modified label smoothing, delivers performance that matches or surpasses that of previous state-of-the-art imbalanced training methods on these new datasets.  We have also extended our analysis to tabular datasets. These datasets represent a challenging frontier for deep learning research. While tabular datasets often inherently exhibit imbalance, there has been limited research addressing the impact of imbalanced data on deep learning in this domain. We incorporated our suggested methods (SAM-A, modified label smoothing, and SGD with cosine annealing performed with a small batch size) on three tabular datasets - the Otto Group Product Classification [4], Covertype [5], and Adult [6] datasets using a Multilayer Perceptron (MLP) with improved numerical feature embeddings. Our methodology outperforms both XGBoost and recent state-of-the-art neural methods such as XGBoost, ResNet, and FT-Transformer [7] on all three datasets, illustrating the applicability of our findings beyond image classification. Please refer to Table 2 on the attached page for the full results.\n\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback, including multiple experiments and paper edits, and we would appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References:\n\n[1] Sekhari et al., 2021. Sgd: The role of implicit regularization, batch-size and multiple-epochs.\n\n[2] Le et al, 2015, Tiny imagenet visual recognition challenge.\n\n[3] Map of Camille Isben's contributions to Biodiversity of Magnuson Park, Seattle, WA, project, 2017, iNaturalist\n\n[4] Kaggle, 2015, Otto Group Product Classification Challenge.\n\n[5] Blackard and Dean, 1990, Predicting forest cover types.\n\n[6] Kohavi and Sahami, 1996, Error-based and entropy-based discretization.\n\n[7] Gorishniy et al., 2021, Revisiting deep learning models for tabular data.\n\n"}, {"review_id": "JSszjKnPs8", "reviewer": "Reviewer_E6os", "summary": "The authors show that simply tuning standard hyperparameters provides state-of-the-art performance on a wide variety of class-imbalanced datasets, which may be surprising and give an impact to the community: We have to re-think the experimental settings for performance evaluation on imbalanced datasets. \n\nThe authors show that simply tuning existing components of DNNs, such as the batch size, data augmentation, architecture size, pre-training, optimizer, and label smoothing, can achieve state-of-the-art performance without any specialized loss functions or samplers. \nSpecifically, (1) imbalanced data prefers small batch sizes, (2) the data augmentation strategies that achieve the best performance on balanced datasets yield inferior performance on imbalanced datasets, (3) large models that do not overfit to balanced datasets strongly overfit to imbalanced datasets of the same size, (4) adding an self-supervised loss during training can improve the performance on imbalanced datasets, (5) the SAM optimizer improves minority accuracy, and (6) label smoothing prevents the overfitting to minority classes.\n\nThorough experiments are performed on six image datasets, including natural image, medical, and remote sensing datasets as well as two tabular datasets. The models are a variety of CNNs, XGboost, and SVM. The authors run five seeds for each evaluation and report the mean and one standard error. The performance improvement compared with state-of-the-art (e.g., [Zhou+, CVPR2023, http://home.ustc.edu.cn/~zzp1994/2023068462.pdf]) is significant (especially on CIFAR-10 and -100).\n\nThe authors conclude that, even though they attain the state-of-the-art, existing methods designed for web-scraped natural image classification benchmarks do not always yield improvements on other real-world problems. If we are to reliably compare methods for class imbalance, we need a more diverse benchmark suite, i.e., there may not a universally good model for imbalanced datasets.\n\nThe paper is well-written and easy to follow. I enjoyed reading the paper. Contribution is clear. Experimental settings are well-explained. Distinctions from previous works are clearly stated. Overall, the contribution and impact of the paper is significant compared with other papers that have been published in top conferences.", "questions": "- [Question] Could you discuss the differences of underlying mechanism of Joint-SSL and supervised contrastive learning, e.g., [Wang+, CVPR2021, https://arxiv.org/abs/2103.14267]? (I know the difference of the loss functions between the two.) This is a bit abstract question, but I would appreciate it if you could answer this.\n\n- [Question] It is shown that SAM and its variant are effective on imbalanced datasets in [Azzuni+, ISBIC2022, https://ieeexplore.ieee.org/document/9854725] and [Zhou+, CVPR'23, http://home.ustc.edu.cn/~zzp1994/2023068462.pdf].\nCould you briefly discuss the relationships between the present paper and these two?\n\n- [Question] In [Shen+, ICLR2021, https://openreview.net/pdf?id=PObuuGVrGaZ], they discuss in Section 7 that label smoothing is ineffective when the dataset is imbalanced in the context of knowledge distillation. Could you discuss the relationships between this result and 6. in Introduction of the current paper?\n\n- [Comment] On the other hand, label distribution smoothing in imbalanced regression problems is shown to be effective in [Yang+, ICML2021, http://dir.csail.mit.edu/]. There may be something more in label smoothing and imbalance. Interesting.\n\n- [Question] Could you explain why are smaller batch sizes beneficial for more imbalanced datasets? The authors discuss this point: there is a tradeoff between (1) large batch sizes prevent from forgetting minor classes by including minority samples in almost every batch and (2) large batch sizes lead to overfitting to majority classes because they dominate almost all batches. If so, what is the main factor that controls this tradeoff? This would be a difficult question, and just a guess is welcome.\n\n- [Comment (minor)] (Related to Figure 1) I would like to see more results on different datasets and models other than CIFAR-10 and -100 with ResNet, if possible.\n\n- [Question (minor)] I know the focus of the present paper is CNNs, but are there any experimental results of Transformer-based models? I wonder if there is any differences between CNNs and Transformers.\n\n- [Comment (major)] In this paper, hyperparameter tuning is shown to be important for performance on imbalanced datasets. THEREFORE, I would like to see ALL the hyperparameter settings and tuning method of ALL the experiment. I would like to see the code to reproduce the experimental results. This is also because it seems that several specialized methods are needed to achieve SOTA such as TrivialAugment + CutMix + label smoothing + exponential moving weight average + Joint-SSL + SAM-A (+ M2m) (please correct me if it is wrong). In addition, batch size, choice of data augmentation methods, learning rate, and other hyperparameters are also tuned, I guess.\n\n- [Comment (minor)] Optimizer dependence of the classification performance on imbalanced datasets is also interesting.", "limitations": "- As stated by the authors, the optimal learning configuration depends on the task, and thus it may be unclear how universal the know-how obtained in this paper is.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors show that simply tuning standard hyperparameters provides state-of-the-art performance on a wide variety of class-imbalanced datasets, which may be surprising and give an impact to the community: We have to re-think the experimental settings for performance evaluation on imbalanced datasets.\n\n- Experimental results are convincing. Thorough experiments are performed on six image datasets, including natural image, medical, and remote sensing datasets as well as two tabular datasets, which are sufficient compared with other contemporary papers published top conferences. The models are a variety of CNNs, XGboost, and SVM. The authors run five seeds for each evaluation and report the mean and one standard error. The performance improvement compared with state-of-the-art (e.g., [Zhou+, CVPR2023, http://home.ustc.edu.cn/~zzp1994/2023068462.pdf]) is significant (especially on CIFAR-10 and -100).\n\n- The paper is well-written and easy to follow. Contribution is clear. Experimental settings are well-explained. Distinctions from previous works are clearly stated. Overall, the contribution and impact of the paper is significant compared with other papers that have been published in top conferences.\n\n- There are some papers that claims that supervised contrastive learning is effective for imbalanced learning, e.g., [Wang+, CVPR2021, https://arxiv.org/abs/2103.14267]. There is also a seminal paper that analyze self-supervised learning and class-imbalance [44], in which Liu+ claim that pre-trained self-supervised representations are more robust to imbalance than supervised representations. To my understanding, the authors' approach is somewhat similar to (but not exactly the same as) these two, i.e., adding self-supervised loss to supervised training and mitigating the pre-training phase (Joint-SSL). The authors show that this approach is effective on imbalanced datasets, which is a consistent result with the two references above, although Joint-SSL only is not always helpful or the improvement is marginal (Tables 1 & 2).", "weaknesses": "- In this paper, hyperparameter tuning is shown to be important for performance on imbalanced datasets. THEREFORE, I would like to see ALL the hyperparameter settings and tuning method of ALL the experiment. I would like to see the code to reproduce the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nEkH,\n\nThank you for your thorough and insightful feedback. We are grateful for your recognition of our work's significance and potential impact. We address each of your points below:\n\n## Hyperparameter Settings and Code Availability:\nRegarding hyperparameter tuning, we adopt a standard process as reported in previous works [1]. In our revised working draft, we now provide a more comprehensive appendix detailing all hyperparameter settings. Furthermore, we will make our code publicly available upon publication to ensure full reproducibility of our results.\n\n## Compare to Supervised Contrastive Learning:\nIn our work, Joint-SSL combines self-supervised and supervised losses, aiming to leverage the benefits of both approaches. Unlike supervised contrastive learning, even if every sample was in the same class, the SSL objective allows us to learn useful features.  Another difference lies in the choice of self-supervised loss: In our implementation, we use a non-contrastive objective - VICReg - along with the supervised loss, while supervised contrastive learning uses a contrastive loss. We appreciate your comment and clarify this in our revised version.\n\n## Relationships with SAM:\nThank you for pointing out these papers. We can't find the Azzuni+ paper and would appreciate a pointer. [1] suggest using SAM directly to combat data imbalance without specifying optimizing the perturbation sizes per class. However, [2], whose work was parallel to ours, suggests a more complicated version of SAM requiring a two-stage process. We agree that the first step of their method is similar to ours, and we will update our revised paper accordingly.\n\n## Label Smoothing and Imbalance:\nShen et al. (2021) found that smoothing the same for all classes is ineffective for imbalanced datasets. However, our suggested method applies different smoothing for each class based on their distributions. We apply more smoothing to the minority classes, which need regularization. This is consistent with our other findings that methods increasing regularization and robustness against overfitting of the minority classes are beneficial for imbalanced training.\n\n## Why Smaller Batch Sizes are Beneficial for More Imbalanced Datasets:\n\nOur current understanding is that modern DNNs tend to perform well with large batch sizes during balanced training. This is partly because larger batches inherently provide a form of implicit regularization. They include diverse samples from different classes in each batch, creating heterogeneity in the data that promotes generalization. On the other hand, this heterogeneity is often lacking for imbalanced training. Therefore, with large batches, DNNs may overfit to minority classes. Conversely, smaller batch sizes introduce a form of regularization. Specifically, we found that smaller batch sizes help prevent overfitting minority classes. This acts as a form of regularization in SGD as described by Sekhari et al. [3], who analyzed the role of batch size as a regularizer for the loss landscape, encouraging generalization. This effect introduces noise into gradient estimation and counteracting the tendency to overfit to minority examples. \n\nTo delve deeper into this issue, we examine its interaction with our joint-SSL method. While SSL models typically perform better with larger batch sizes for balanced datasets, our study suggests that the conventional wisdom for SSL models may not apply to imbalanced data. To verify this, we trained our joint-SSL algorithm with different batch sizes on CIFAR-10. As seen in the provided page, the impact of batch size is less pronounced, with joint-SSL acting as an alternative regularizer, mitigating the overfitting of minority classes.\n\n\nWe suspect the heterogeneity of samples might control the trade-off between including more samples in each batch and avoiding overfitting. However, we acknowledge that this is a complex topic, requiring further investigation to understand the underlying mechanism fully.\n\n## Additional Results on Different Datasets and Models:\nPrompted by your feedback, we have included additional results with the Tiny-Imagnet and iNaturalist, using Swin Transformer v2 and ConvNeXt. The results of these additional experiments consistently align with our previous findings, enhancing our work's robustness and general applicability. Furthermore, these enhancements increase the diversity of our architectural analysis, contributing to a broader understanding of various architectures' behavior under imbalance.\n\n\n## Experimental Results of Transformer-Based Models:\nWhile our focus in this paper is primarily on CNNs, in response to your comment, we have analyzed the transformer-based model, Swin Transformer v2. As you can see in the attached figure, the resulting behavior is similar to that exhibited in CNN experiments. \n\n## Optimizer Dependence on Classification Performance:\nWe have now added a more rigorous analysis to our revised working draft, and we’ll include it in our camera ready version. In brief, the differences between SGD and AdamW are small. Generally, while AdamW was better for larger batch sizes and more robust to hyperparameter choices, SGD achieved better results for imbalanced training.\n\nThank you again for your thoughtful review. We made a significant effort to address your feedback, including multiple experiments and paper edits, and we would appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References:\n[1]  Liu et al., 2021, “Self-supervised learning is more robust to dataset imbalance”.\n\n[2] Zhou et al., 2023, “Class-Conditional Sharpness-Aware Minimization for Deep Long-Tailed Recognition”.\n\n[3]  Sekhari et al., 2021, \"Sgd: The role of implicit regularization, batch-size and multiple-epochs”.\n\n"}]

Author rebuttal:
Dear Reviewers,

Thank you for your thoughtful and detailed reviews of our work. We appreciate your time and the constructive feedback you have provided. We have carefully considered your comments and concerns and would like to address them in a unified response:

**1. Diversity of Architectures and Additional Experiments and Results:**   
In response to your feedback, we expanded our experiments to include different models, such as Swin Transformer v2[1] and ConvNeXt[2], and more challenging visual datasets, like Tiny-ImageNet[3] and iNaturalist[4]. The results of these additional experiments consistently align with our previous findings, which enhances the robustness and general applicability of our work. Furthermore, these enhancements increase the diversity of our architectural analysis, contributing to a broader understanding of various architectures' behavior under imbalance.

**2. Inclusion of Tabular Datasets:**  
Recognizing the significance of class imbalance in various domains, we have extended our analysis to incorporate tabular datasets. This addition enriches our research, as tabular datasets often inherently exhibit imbalance and have been less explored in the context of deep learning. Our methodology was applied to three tabular datasets[5-7], demonstrating its effectiveness beyond image classification. Our methodology outperforms both XGBoost and recent state-of-the-art neural methods [8]  on all three datasets. These results underscore the robustness of our findings and their relevance to real-world applications.

**3. Novelty and Contribution:**  
Our work demonstrates that techniques designed for class-balanced data can easily be tuned or adapted to class-imbalanced data for state-of-the-art results, even outperforming methods specifically designed for imbalance. We show that existing data augmentation techniques, SSL, SAM, and label smoothing, which are commonly used in balanced setups, can be effectively applied to imbalanced training. Moreover, we provide critical insights into the performance of imbalance-specific methods in real-world scenarios, illustrating a lack of direct correlation between their performance on conventional benchmark datasets and their effectiveness on naturally imbalanced datasets. This finding encourages a broader evaluation of methods and contributes to our understanding of class imbalance training. Our work represents a perspective-shift on training under class imbalance but moreover, we achieve significant performance gains across multiple benchmark and real-world datasets.

**4. Batch Size and Interaction with Joint-SSL:**
Prompted by your comments on our study of batch size, we now delve deeper in this direction. We find that smaller batch sizes tend to yield better performance by preventing overfitting on minority classes, acting as a form of regularization in SGD[9]. This effect introduces noise into gradient estimation, smoothing the loss landscape and counteracting the tendency to overfit to minority examples. Furthermore, we investigated the interplay between batch size and joint-SSL, an aspect of regularization. While SSL models usually perform better with larger batch sizes under balanced datasets, our study shows that the dynamics change when handling imbalanced data. We discovered that the effect of batch size is less significant when using joint-SSL, which acts as an alternative regularizer, mitigating the overfitting of minority classes. These findings highlight the complex interplay between batch size and regularization methods, providing valuable insights for practitioners dealing with imbalanced data.

We have put forth a significant effort to address all your feedback and feel that your suggestions have improved our work.  We welcome further discussion, and we appreciate your valuable input.

### References:
[1] Liu et al., 2021, Swin transformer v2: Scaling up capacity and resolution.

[2] Liu et al., 2022, A convnet for the 2020s.

[3] Le et al, 2015, Tiny imagenet visual recognition challenge.

[4] Map of Camille Isben's contributions to Biodiversity of Magnuson Park, Seattle, WA, project, 2017, iNaturalist.
[5] Kaggle, 2015, Otto Group Product Classification Challenge.

[6] Blackard and Dean, 1990, Predicting forest cover types.

[7] Kohavi and Sahami, 1996, Error-based and entropy-based discretization.

[8] Gorishniy et al., 2021, Revisiting deep learning models for tabular data.

[9] Sekhari et al., 2021. SGD: The role of implicit regularization, batch-size, and multiple-epochs.





Meta-review: This paper initially received diverse scores. After the rebuttal period, most of the reviewers acknowledged their concerns addressed and raised their scores to suggest acceptance. One reviewer concerning about the limited novelty and the architecture variants tested did not respond to the authors’ rebuttal. The AC inspected the paper and all the comments and believed that these issues have been addressed by the authors. The AC agreed with the other reviewers on the quality and significant contributions of this paper, and also recommended accept.
Decision: Accept (poster)