SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset curation. Common intuition dictates that training on data that is more balanced than the testing distribution can improve representation learning by preventing overfitting to minority samples [25, 10, 27]. In Appendix A.8, we find that this intuition may be misguided both for neural networks and gradient-boosted decision trees, especially on large datasets, and that curating additional samples may in fact be destructive if the proper dataset balance is not maintained.
4 Benchmarking Training Routines under Class Imbalance
In the preceding sections, we examined various building blocks of balanced training routines and presented modifications of optimization methods—label smoothing, Sharpness-Aware Minimization, and self-supervision—tailored specifically for imbalanced scenarios. In this section, we compare the performance of models trained using our modified methods to those trained using existing state-of-the-art methods for handling class imbalance. To ensure a fair and unbiased comparison, all methods are trained using the same refined training routines. Our comparison provides evidence of the effectiveness of our proposed methods when coupled with our optimized routines.
4.1 Vision Datasets
4.1.1 Experimental Setup
Datasets. We perform experiments on six image datasets, including natural image, medical, and remote sensing datasets: CIFAR-10 [46], CIFAR-100, CINIC-10 [15], Tiny-ImageNet[16], SIIM-ISIC Melanoma [26], APTOS 2019 Blindness [1] and EuroSAT [30].
Baseline Methods. We compare to the following comprehensive range of baselines: (a) Empirical Risk Minimization (ERM) involves training on the cross-entropy loss without any re-balancing. (b) Resampling balances the objective by adjusting the sampling probability for each sample. (c) Synthetic Minority Over-sampling Technique (SMOTE) [10] is a re-balancing variant that involves oversampling minority classes using data augmentation. (d) Reweighting [33] simulates balance by assigning different weights to the majority and minority classes. (e) Deferred Reweighting (DRW) involves deferring the resampling and reweighting until a later stage in the training process. (f) Focal Loss (Focal) [50] upweights the objective for difficult examples, thereby focusing more on the minority classes. (g) Label-Distribution-Aware Margin (LDAM-DRW) [9] trains the classifier to impose a larger margin on minority classes. (h) M2m [42] translates samples from majority to minority classes. Lastly, (i) MiSLAS [75] is a two-stage training method that combines mixup [74] with label smoothing. We also combine our techniques with previous state-of-the-art Major-to-minor Translation (M2m) [42] and observe it improves performance over M2m alone.
Evaluation. We follow the evaluation protocol used in [54, 42], training models on class-imbalanced training sets and evaluating them on balanced test sets. We evaluate on four benchmark datasets for imbalanced classification: CIFAR-10, CIFAR-100 [46], CINIC-10 [15] and Tiny-ImagNet [47] with training ratios of 0.01, 0.02, and 0.1. Additionally, we use three real-world datasets, namely APTOS 2019 Blindness Detection [1], SIIM-ISIC Melanoma Classification [26], and EuroSAT [30].
Table 1: Our training routines exceed previous SOTA or improve existing methods when combined. Accuracy on various datasets using different methods. Error bars correspond to one standard error over 5 trials.
Method	CIFAR-10	CIFAR-100	CINIC-10	Tiny-ImageNet (0.1)
0.01	0.02	0.01	0.02	0.01	0.02	SwinV2	ConvNeXt
ERM	84.9 ± 0.2	84.2 ± 0.3	47.7 ± 0.5	52.5 ± 0.4	78.6 ± 0.2	82.3 ± 0.3	53.4 ± 0.2	53.1 ± 0.3
Reweighting	81.8 ± 0.3	79.6 ± 0.4	42.1 ± 0.4	47.8 ± 0.3	71.4 ± 0.4	74.9 ± 0.4	52.8 ± 0.4	52.3 ± 0.1
Resampling	82.1 ± 0.3	79.0 ± 0.2	42.8 ± 0.3	48.2 ± 0.4	72.1 ± 0.5	75.6 ± 0.4	52.5 ± 0.3	52.1 ± 0.2
Focal Loss	83.2 ± 0.5	84.4 ± 0.4	47.1 ± 0.5	52.9 ± 0.5	74.9 ± 0.4	78.4 ± 0.3	53.5 ± 0.1	53.1 ± 0.4
LDAM-DRW	84.9 ± 0.3	82.9 ± 0.4	47.2 ± 0.4	53.3 ± 0.3	76.0 ± 0.3	79.7 ± 0.4	54.2 ± 0.2	53.4 ± 0.3
M2m	84.5 ± 0.2	84.3 ± 0.2	47.1 ± 0.3	52.1 ± 0.3	78.3 ± 0.5	81.3 ± 0.5	54.3 ± 0.4	53.9 ± 0.2
MiSLAS	85.1 ± 0.3	84.3 ± 0.3	47.3 ± 0.3	53.1 ± 0.5	78.1 ± 0.2	81.8 ± 0.3	54.1 ± 0.3	53.4 ± 0.1
SAM-A	85.4 ± 0.3	83.3 ± 0.3	47.1 ± 0.4	52.7 ± 0.2	77.3 ± 0.2	80.9 ± 0.2	54.7 ± 0.4	53.9 ± 0.2
Joint-SSL	85.2 ± 0.2	84.1 ± 0.2	47.0 ± 0.2	51.7 ± 0.5	77.5 ± 0.3	81.3 ± 0.3	54.3 ± 0.2	53.7 ± 0.3
Joint-SSL + SAM-A + Smoothing	85.9 ± 0.2	84.7 ± 0.3	48.0 ± 0.3	52.6 ± 0.3	78.1 ± 0.6	82.3 ± 0.4	54.8 ± 0.1	54.1 ± 0.4
Joint-SSL + SAM-A + M2m	86.0 ± 0.5	85.0 ± 0.4	48.9 ± 0.2	53.8 ± 0.2	79.2 ± 0.4	82.1 ± 0.3	55.0 ± 0.2	54.3 ± 0.3
Training procedure. Our training protocol for all methods incorporates TrivialAugment [58] + CutMix [72] as an augmentation policy, along with label smoothing and an exponential moving weight average with small batch size on image classification. We use WideResNet-28×10 [73] for CIFAR-10, CIFAR-100, and CINIC-10 datasets. For Tiny-ImageNet we use both ConvNeXt [53] and Swin transformer v2 [52] and ResNeXt-50-32×4d [70] for the real-world datasets following the comparisons in Fang et al. [18]. We provide further implementation details in Appendix B.
4.1.2 Image Classification Benchmarks
We see in Table 1 that our adaptations alone bring the performance of standard ERM training remarkably close to or even exceeding that of state-of-the-art imbalanced training methods. Additionally, the integration of our findings from Section 3 with Joint-SSL and Asymmetric-SAM atop M2m establishes a new state-of-the-art across all benchmark datasets. Even using only our self-supervision and Asymmetric-SAM (without M2m) often surpasses the previous state-of-the-art. Furthermore, the integration of our previous findings from Section 3 with Joint-SSL and Asymmetric-SAM atop M2m establishes new state-of-the-art performance across all benchmark datasets.
Our combined method enhances minority class accuracy while preserving accuracy on majority classes. On the 20 smallest classes of CIFAR-100, our method reduces error by  $5.4\% - 6.8\%$  compared to baseline methods. On CINIC-10, our approach enhances the generalization on the two smallest minority classes by  $4.6\% - 8.5\%$  compared to baseline methods. Additional results regarding split class accuracy and different backbones can be found in Appendix A.9.
4.1.3 Real-World Datasets
Since the majority of research on class-imbalanced training focuses on web-scraped datasets such as ImageNet or CIFAR-10 and CIFAR-100, we now investigate whether such advances are overfit to these datasets or whether they are robust in other settings.
To address this question, we consider the SIIMISIC Melanoma, APTOS 2019 Blindness, and EuroSAT datasets. In Table 2, we observe that our highly tuned training routine equipped with Joint-SSL, Asymmetric-SAM, and modified label smoothing delivers equal or superior performance compared to previous state-of-the-art imbalanced training methods on all datasets.
Correlation between performance on different datasets. Given the superior performance of
Table 2: Our routines exceed previous SOTA on real-world datasets. Error bars correspond to one standard error over 5 trials using ResNeXt-50-32×4.  $\uparrow$  denotes the method on the line above.
Method	SIIM-ISIC
Melanoma	APTOS 2019
Blindness	EuroSAT
ERM	95.1 ± 0.3	89.1 ± 0.3	99.0 ± 0.2
Reweighting	94.7 ± 0.3	88.4 ± 0.2	98.4 ± 0.3
Resampling	94.9 ± 0.2	88.7 ± 0.4	98.7 ± 0.3
Focal Loss	95.0 ± 0.2	89.2 ± 0.3	98.9 ± 0.3
LDAM-DRW	95.2 ± 0.2	89.0 ± 0.4	99.0 ± 0.4
M2m	95.2 ± 0.2	89.6 ± 0.5	99.3 ± 0.2
MiSLAS	95.4 ± 0.2	89.4 ± 0.4	99.2 ± 0.3
Joint-SSL	95.7 ± 0.3	89.7 ± 0.5	99.1 ± 0.2
↑ + SAM-A	96.0 ± 0.2	90.1 ± 0.6	99.1 ± 0.2
↑ + Smoothing	96.2 ± 0.2	90.1 ± 0.6	99.3 ± 0.2
our approach on these datasets, we now examine the correlation between the performance of various methods on CIFAR-10 (with a training ratio of 0.01) and their performance on real-world datasets. Our findings, illustrated in Table 3, reveals a surprisingly low correlation. This low correlation suggests that a method's successful application to web-scraped datasets like CIFAR-10 does not necessarily translate into equivalently strong performance on real-world datasets. These findings further underscore the need for a more diverse range of datasets in the development and testing of machine learning methods for class imbalance.
4.2 Tabular Datasets
Tabular data problems represent a challenging frontier for deep learning research. While recent advances in natural language processing (NLP), vision, and speech recognition have been driven by deep learning models, their efficacy in the tabular domain remains unclear. Notably, there is a debate over the performance of neural networks in comparison to decision tree ensembles like XGBoost [11, 63, 56]. Despite the fact that most tabular datasets inherently exhibit imbalance, there has been limited research addressing the impact of imbalanced data on deep learning in tabular domains.
Table 3: There is only a weak correlation between CIFAR-10 test accuracy and performance on real-world datasets.
Dataset	Correlation	Slope
APTOS 2019 Blindness	0.11	0.04
SIIM-ISIC Melanoma	0.19	0.03
EuroSAT	0.03	0.04
We thus apply our findings to imbalanced tabular datasets, using a Multilayer Perceptron (MLP) with the improved
numerical feature embeddings of Gorishniy et al. [23]. Our approach incorporates SAM-A, modified label smoothing, and SGD with cosine annealing performed and small batch size.
We apply our methods to the following three imbalanced tabular datasets: Otto Group Product Classification [37], Covertype [7], and Adult datasets [43]. We compare our methodology with the following baseline methods: (1) XGBoost, (2) MLP, (3) ResNet, and (4) FT-Transformer, where the last three baselines are employed as in Gorishniy et al. [22]. Our tuning, training, and evaluation protocols are consistent with those in Gorishniy et al. [23]. For full details about the training procedures, see Appendix A.9.1.
We see in Table 6 that our tuned training routine outperforms both XGBoost and recent state-of-the-art neural methods on all datasets, demonstrating the applicability of our findings beyond image classification.
5 Regularization and Overfitting in Class-Imbalanced Training
Specialized loss functions and sampling methods for class-imbalanced learning are often designed to mitigate overfitting [42], yet we rarely look under the hood to understand what happens to models trained in this setting. In this section, we quantify and visualize overfitting during class-imbalanced training, and we find that successful methods regularize against this overfitting.
One concern for training under class imbalance might be optimization. Perhaps minority samples are hard to fit since they occur infrequently in batches during training. In Appendix A.10, we verify that empirical risk minimization, without any special interventions, easily fits all training data. This observation indicates that variations in performance among the different methods we compare stem not from their optimization abilities but from their generalization to unseen test samples.
To understand the differences between classifiers learned on imbalanced training data, we visualize their decision boundaries on a 2D toy problem with a multi-layer perceptron. Standard training results in small margins around minority class samples, whereas SAM-A, acting as a regularizer, expands these margins (Figure 18a). For additional examples, see Appendix A.11, where we also use the method introduced by Somepalli et al. [66] to visualize decision boundaries.
To quantify these observations, we examine the Neural Collapse phenomenon [45], which was previously observed in the class-balanced setting. Neural Collapse refers to the tendency of the features in the penultimate layer associated with training samples of the same class to concentrate around their class-means. Our investigation focuses on two metrics:
Class-Distance Normalized Variance (CDNV): This metric evaluates the compactness of features from two unlabeled sample sets,  $S_{1}$  and  $S_{2}$ , relative to the distance between their respective feature means. A value trending towards zero indicates optimal clustering.
Nearest Class-Center Classifier (NCC): As training progresses, feature embeddings in the penultimate layer undergoing Neural Collapse become distinguishable. Consequently, the classifier tends to align with the 'nearest class-center classifier'.
We see in Table 4 that the collapse, measured by CDNV and NCC for minority training examples, is significantly worse in standard training without class-imbalance interventions. Furthermore, a correlation exists between the degree of collapse and the performance of the different methods. Specifically, methods that successfully counteract Neural Collapse exhibit superior performance.
We conclude that well-tuned training routines can regularize and prevent overfitting to minority class training
Table 4: Neural Collapse during class-imbalanced training. Neural collapse (low CDNV and high NCC) corresponds to low test accuracy. Experiments conducted on CIFAR-10.
Method	CDNV	NCC	Accuracy
ERM	0.42 ± 0.02	0.92 ± 0.02	84.58 ± 0.40
Reweighting	0.38 ± 0.02	0.94 ± 0.01	82.62 ± 0.44
Resampling	0.38 ± 0.03	0.97 ± 0.01	82.16 ± 0.43
LADM-DRW	0.41 ± 0.02	0.94 ± 0.01	83.86 ± 0.23
Joint-SSL + SAM-A	0.43 ± 0.02	0.90 ± 0.02	84.80 ± 0.43
samples without specialized loss functions or sampling methods, which is associated with performance improvements on class-imbalanced data. See Appendix A.10 for more details.
While neural network training practices have been studied extensively on class-balanced benchmarks, real-world problems often involve class imbalance. We have shown that class-imbalanced datasets require carefully tuned batch sizes and smaller architectures to avoid overfitting, as well as specially chosen data augmentation policies, self-supervision, sharpness-aware optimizers, and label smoothing. Whereas previous state-of-the-art works for class-imbalance focused on specialized loss functions or sampling methods, we show that simply tuning standard training routines can significantly improve performance over such ad hoc approaches.
Our findings give rise to several important directions for future work:
- We saw that existing methods designed for web-scraped natural image classification benchmarks do not always provide improvements on other real-world problems. If we are to reliably compare methods for class imbalance, we need a more diverse benchmark suite.
- Since most real-world datasets are class-imbalanced and architectures designed for class-balanced benchmarks like ImageNet are highly suboptimal under class imbalance, perhaps future work should build architectures which are specifically optimized for class imbalance.
- The generalization theory literature explains the tradeoff between fitting training samples and the complexity of learned solutions [55]. Can PAC-Bayes generalization bounds explain the large role regularization plays in successful training under class imbalance?
- Language models perform classification over tokens, but some tokens occur much less frequently than others. How can we apply what we have learned about training under class imbalance to language models?
Acknowledgements. This work is supported by NSF CAREER IIS-2145492, NSF I-DISRE 193471, NIH R01DA048764-01A1, NSF IIS-1910266, NSF 1922658 NRT-HDR, Meta Core Data Science, Google AI Research, BigHat Biosciences, Capital One, and an Amazon Research Award.
[68] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.
[69] Ross Wightman, Hugo Touvron, and Herve Jegou. Resnet strikes back: An improved training procedure in timm. In NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
[70] Saining Xie, Ross Girshick, Piotr Dólár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[71] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 33:19290-19301, 2020.
[72] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[73] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[74] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[75] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489-16498, 2021.
. This appendix is organized as follows:
- In Appendix A, we provide more detailed results on additional datasets for the different components discussed in Section 3. Specifically, in subsections Appendices A.1 to A.7, we present results on batch size, data augmentation, model architectures, pre-training, SSL, Sharpness-Aware Minimization, and label smoothing. These subsections delve into the specific effects and outcomes of each component.
- In Appendix A.8, we examine the relationship between the training and test distributions in imbalanced training. We explore the optimal balance of training data and discuss the potentially destructive impact of collecting additional majority samples.
- In Appendix A.9, we present additional and extended experimental results that compare the methods proposed in our paper with the baseline methods.
- In Appendix A.10, we provide additional experimental results that illustrate how the training process evolves for imbalanced data.
- In Appendix A.11, we include decision boundary visualizations for imbalanced training. Specifically, we demonstrate that Sharpness-Aware Minimization (SAM-A) helps decision regions take up similar volumes, whereas standard training routines tend to shrink-wrap the decision boundaries around minority samples.
- In Appendix B, we provide detailed information on the hyperparameters, datasets, and architectures used in our experiments.
- In Appendix C, we discuss the limitations of our study. This section addresses potential constraints, challenges, and areas for improvement in our research.
- Lastly, in Appendix D, we discuss the broader impact of our work. This section explores the implications, significance, and potential applications of our findings beyond the scope of the immediate study.
A.1 Batch Size
To investigate the impact of batch size in the context of class imbalance, we train networks across various training ratios using different batch sizes. In order to compare the accuracy for each training ratio, we calculate the percentage improvement over the baseline (set as the best batch size of 128). Specifically, if we denote  $Acc_{b}^{\rho}$  as the accuracy on the imbalanced dataset with training ratio  $\rho$  and batch size  $b$ , we define the adjusted accuracy  $new_{A}cc_{b}^{\rho}$  as
$$
\bar {A c c} _ {b} ^ {\rho} = \frac {A c c _ {b} ^ {\rho} - A c c _ {1 2 8} ^ {\rho}}{A c c _ {b} ^ {\rho}}. \tag {1}
$$
Positive values represent higher accuracy compared to the baseline, while negative values denote lower accuracy. This normalization allows us to examine the relative effect of batch size. As shown in the main text and Figure 5, data with a high degree of class imbalance tends to benefit from smaller batch sizes, despite the fact that small batches often do not contain any minority samples.
A.2 Data Augmentation
In order to evaluate and compare the effectiveness of various popular augmentation techniques—including horizontal flips, random crops, AugMix [31], TrivialAugmentWide [58], and AutoAugment [13]—we investigate their impact on the accuracy of minority and majority classes across a range of training ratios.
We measure the relative improvement in performance by comparing the accuracy achieved with data augmentation to that achieved without it. We thus plot the percentage improvement as a function of the training ratio in Figure 6.
Figure 5: Batch size matter more for imbalanced data where small batch sizes are best, whereas the curve corresponding to balanced data is flat. Percentage improvement in test accuracy over the default batch size of 128 at different training ratios. Experiments conducted on CIFAR-10.
Our findings reveal that while the newer TrivialAugment method exhibits superior performance on balanced training data, the older AutoAugment method yields better results on highly imbalanced data.
Figure 6: Optimal augmentations depend on the imbalance ratio. We plot the percent improvement in test accuracy for different augmentations compared to training without augmentations across train ratios for different augmentations. We see that TrivialAugment, which is known to outperform AutoAugment on class-balanced data, actually performs worse when data is severely imbalanced. Experiments conducted on CIFAR-100.
In Figure 8, we illustrate the impact of model size on the performance of the CIFAR-10 dataset with a training ratio of 0.001. The trend observed is similar to the results discussed in the main text, where increasing the model size leads to overfitting in the case of imbalanced training.
Figure 7: Strong augmentations are particularly effective at improving minority class accuracy under severe class imbalance. The percent improvement in test accuracy of TrivialAugment compared to training without any augmentation as a function of the training ratio. Experiments conducted on CIFAR-10. Error bars represent one standard error over 5 trials.
Figure 8: Bigger architectures overfit on class-imbalanced data. Experiments conducted on CIFAR-10. Error bars represent one standard error over 5 trials.
A.4 Pre-training
To assess the effectiveness of pre-training, we fine-tune several pre-trained models on downstream datasets with varying training ratios. In addition to the main body, Figure 10 illustrates the percentage improvement in test accuracy compared to random initialization for supervised pre-training on ImageNet-1k and ImageNet-21k, as well as SimCLR on ImageNet-1k (which is a Self-Supervised Learning (SSL) method), measured by downstream performance on CIFAR-10. This comparison is made across different training ratios (Figure 4). Let  $\mathrm{Acc}_{\mathrm{Rand}}^{\rho}$  denote the accuracy of the model trained from random initialization at a training ratio  $\rho$ . The relative improvement is then defined by:
$$
\bar {A c c} _ {b} ^ {\rho} = \frac {A c c _ {b} ^ {\rho} - A c c _ {\text {R a n d}} ^ {\rho}}{A c c _ {\text {R a n d}} ^ {\rho}} \tag {2}
$$
Figure 9: Bigger architectures overfit on class-imbalanced data. Experiments were conducted on CINIC-10 with an imbalanced train ratio of 0.001. Error bars represent one standard error over 5 trials.
Positive values indicate an improvement in performance compared to random initialization. It is clear that all pre-training methods improve performance when compared to random initialization. Interestingly, these improvements are significantly more pronounced under imbalanced conditions.
Figure 10: Pre-training yields bigger improvements on more imbalanced data. The improvement in the test accuracy compared to training from random initialization. Experiments conducted on CIFAR-10.
A.5 SSL
Self-supervised learning (SSL) has gained substantial traction as a method of representation learning across multiple domains, including computer vision, natural language processing, and tabular data [12, 40, 65]. Networks pre-trained using SSL often demonstrate more transferable representations than those pre-trained with supervision [24]. Pre-training traditionally consists of a two-stage process: initial learning on an upstream task followed by fine-tuning on a downstream task. However, the limitation in many use-cases is the lack of large-scale pre-training datasets. In order to solve this problem, our approach diverges from this two-stage process by merging supervised learning with an