SlowGuess/NeurIPS_2023_OCR huggingface dataset:
auxiliary self-supervised loss function during from-scratch training, effectively eliminating the need for any pre-training.
For this, we employ the Variance-Invariance-Covariance Regularization (VICReg) objective [5]:
Given two batches of embeddings,  $\mathbf{Z} = [f(\mathbf{x}_1),\dots ,f(\mathbf{x}_B)]$  and  $\mathbf{Z}' = [f(\mathbf{x}_1'),\dots ,f(\mathbf{x}_B')]$ , each of size  $(B\times K)$ , where  $\mathbf{x}_i$  and  $\mathbf{x}_i'$  are two distinct random augmentations of a sample  $I_{i}$ , we derive the covariance matrix  $\mathbf{C}\in \mathbb{R}^{K\times K}$  from  $[\mathbf{Z},\mathbf{Z}^{\prime}]$ .
Consequently, the VICReg loss can be articulated as:
$$
\mathcal {L} _ {\mathcal {S S L}} = \frac {1}{K} \sum_ {k = 1} ^ {K} \left(\alpha \max  \left(0, \gamma - \sqrt {\boldsymbol {C} _ {k , k} + \epsilon}\right) + \beta \sum_ {k ^ {\prime} \neq k} \left(\boldsymbol {C} _ {k, k ^ {\prime}}\right) ^ {2}\right) + \gamma \| \boldsymbol {Z} - \boldsymbol {Z} ^ {\prime} \| _ {F} ^ {2} / N.
$$
In our experiments, the total loss is given by
$$
L _ {J o i n t - S S L} = L _ {S S L} + \lambda L _ {\text {S u p e r v i s e d}}.
$$
Note that the SSL loss function is independent of the class-imbalanced labels.
A.6 SAM
Sharpness-Aware Minimization [19] is an optimization technique that seeks to find "flat" minima of the loss function, often leading to improved generalization. This method consists of taking an initial ascent step followed by a descent step, aiming to find parameters that minimize the increase in loss resulting from the ascent step. Huang et al. [36] demonstrate that flat minima correspond to wide-margin decision boundaries.
Given a model parameterized by weights  $\theta$  and a loss function  $L(\theta)$  that we aim to minimize, SAM performs two steps in each iteration:
1. First step (gradient ascent): Perform a scaled gradient ascent step from the current model weights  $\theta$ :
$$
\theta^ {\prime} = \theta + \rho | \nabla L (\theta) | _ {2} \frac {\nabla L (\theta)}{| \nabla L (\theta) | _ {2}} \tag {3}
$$
2. Second step (weight update): Update the weights from  $\theta$  in the negative direction of the gradient computed at the post-ascent parameter vector:
$$
\theta = \theta - \eta \nabla L \left(\theta^ {\prime}\right) \tag {4}
$$
In the above steps,  $\eta$  represents the learning rate,  $\rho$  is a hyperparameter determining the size of the neighborhood around the current weights, and  $|\cdot|_2$  denotes the Euclidean norm.
SAM was initially developed for balanced datasets, where the decision boundaries for each class have comparable areas. However, this assumption does not hold true for imbalanced datasets. To address this, we adapted SAM for use with class-imbalanced datasets by increasing the flatness specifically for minority class loss terms. We propose a new method - SAM-Asymmetric (SAM-A). Our method adjusts the ascent step size  $(\rho)$  in SAM's inner loop for minority classes by employing a step size inversely proportional to the classes' proportions.
Let  $p_i$  be the proportion of class  $i$  in the training set. We define the class-conditional ascent step size as:
$$
\rho_ {i} = \frac {\rho}{1 - p _ {i}}, \tag {5}
$$
where  $\rho$  is a scaling factor.
By doing this, we widen the margins around under-represented classes, potentially improving generalization in imbalanced datasets.
A.7 Label Smoothing
Label smoothing is a regularization technique often used in training deep learning models. It mitigates the model's excessive confidence in class labels, which can improve generalization and reduce overfitting. However, traditional label smoothing assumes a balanced class distribution, which is not always the case in real-world datasets.
To adapt label smoothing for imbalanced training, we propose a class-conditional label smoothing technique. Instead of using a uniform smoothing parameter  $\epsilon$ , we use a different  $\epsilon_{i}$  for each class  $i$  which is proportional to the inverse of the class's proportion within the dataset.
Let  $p_i$  be the proportion of class  $i$  in the training set. We define the class-conditional smoothing parameter as:
$$
\epsilon_ {i} = \frac {\epsilon}{1 - p _ {i}}, \tag {6}
$$
where  $\epsilon$  is a scaling factor.
We then apply label smoothing as follows. Let  $p$  be the model's output probability distribution over  $K$  classes, and let  $q_{i}$  be the target distribution for class  $i$ . The smoothed target distribution is:
$$
q _ {i, j} = \left(1 - \epsilon_ {i}\right) I _ {y = j} + \frac {\epsilon_ {i}}{K}, \tag {7}
$$
where  $j \in 1,2,\dots,K$ ,  $y$  is the true class, and  $I_{j}$  is the indicator function.
During training, we minimize the cross-entropy loss between the model's predictions  $p$  and the class-conditional smoothed labels  $q_{i}$ :
$$
L = - \sum_ {i = 1} ^ {K} q _ {i, y} \log p _ {y} \tag {8}
$$
By using class-conditional label smoothing, we apply more smoothing to the minority classes and less to the majority classes, which can help the model generalize better when the class distribution is imbalanced.
A.8 Data Curation
Common intuition dictates that training on data that is more balanced than the testing distribution can improve representation learning by preventing overfitting to minority samples [25, 10, 27]. In this section, we put that intuition to the test by examining the optimal balance of training data. Moreover, while minority class samples may be scarce, a practitioner may be able to collect additional majority class training samples at will, so we also examine the potentially destructive impact of collecting additional majority samples.
A.8.1 The Relationship Between Train-Time and Test-Time Imbalance
The literature on training routines for class imbalance in machine learning is filled with methods designed for scenarios in which training data is highly imbalanced but testing data is balanced. However, data encountered during deployment is typically also imbalanced. Therefore, we disentangle training and testing balances and investigate how sensitive models are to discrepancies between the two. This study may be particularly important if one considers collecting training data for a downstream application. Should we gather training data with the same balance we anticipate during testing? How worried should we be if the data we encounter during deployment is more or less balanced than the training data we gathered?
We begin by illustrating three scenarios in Figure 11: (1) identical training and testing ratios, (2) balanced training, and (3) the training ratio with the lowest test error (optimal training ratio). We see that training on data with the same imbalance as the testing data is superior to training on balanced data, and the two strategies only approach equal performance when the testing data becomes balanced. We share additional results over different datasets and models in Figure 20, Figure 21, and Figure 22.
We then plot for each test ratio the corresponding train ratio that results in the lowest test error in Figure 12. If the two ratios are perfectly aligned, then points will lie on the diagonal. Indeed, the points are close to the diagonal, indicating that it is best to train with a very similar imbalance ratio to the test dataset, especially for highly imbalanced testing scenarios.
In these previous experiments, we fixed the size of the training set, but what happens as we gather more and more training data? In Figure 15, we train and evaluate a network on different imbalance ratios across training set sizes, and we plot the misalignment between the train and test ratios, referring to the average distance between the optimal train ratio and the specified test ratio. As the amount of training data increases, we see that the optimal training ratio becomes more and more close to the ratio of the test data.
Figure 11: Imbalanced training data is optimal for imbalanced testing scenarios. Test accuracy as a function of the test ratio for different training setups. Experiments conducted on CIFAR-100.
A.8.2 When More Data Degrades Performance
In practice, a practitioner may not have precise control over the data they collect. Will collecting additional samples always help performance? Instead of fixing the total number of samples and
Figure 12: The optimal train ratio is closely aligned with the test ratio. Experiments conducted on CIFAR-100.
Figure 13: The optimal train ratio is closely aligned with the test ratio. Experiments conducted on CINIC-10.
Figure 14: Test accuracy on the minority classes as a function of the test ratio for different training setups. 'Equal' denotes the same balance between training and testing, and 'Optimal' is the optimal trainset balance amongst the ratios we try. Experiments conducted on CIFAR-10.
Figure 15: Alignment between train and test proportions improves as the number of training samples increases. Train/test misalignment is calculated by taking the mean over test ratios of the difference between the best train ratio (train ratio that gives maximum test accuracy) and the test ratio. If misalignment is 0, then the best train ratio is always the same as the test ratio.
Figure 16: The potentially destructive effects of adding majority class samples. We fix the number of minority samples to be 200 and vary the number of majority samples. Experiments conducted on CIFAR-100.
Table 5: Our training routines exceed previous state-of-the-art or improve existing methods when combined. Split class accuracy for classes with Few, Med and Many examples of WideResNet-28×10 on long-tailed CIFAR-100 and CINIC-10. Error bars correspond to one standard error over 5 trials.
CINIC-10	CIFAR-100
Method	Few	Med	Many	Few	Med	Many
ERM	40.5 ± 0.4	64.1 ± 0.3	90.1 ± 0.5	20.1 ± 0.3	42.3 ± 0.3	70.5 ± 0.6
Reweighting	36.6 ± 0.5	63.1 ± 0.3	87.8 ± 0.3	17.1 ± 0.4	39.3 ± 0.3	67.1 ± 0.4
Resampling	37.4 ± 0.5	63.6 ± 0.6	87.9 ± 0.4	18.4 ± 0.2	38.1 ± 0.2	68.9 ± 0.3
Focal Loss	39.1 ± 0.2	63.9 ± 0.2	88.2 ± 0.5	19.8 ± 0.4	39.0 ± 0.5	69.3 ± 0.6
LDAM-DRW	40.1 ± 0.4	64.3 ± 0.4	89.8 ± 0.3	20.8 ± 0.5	42.1 ± 0.3	70.6 ± 0.4
M2m	42.8 ± 0.7	64.1 ± 0.6	90.3 ± 0.4	20.1 ± 0.6	41.8 ± 0.4	69.4 ± 0.5
SAM-A	43.2 ± 0.3	62.3 ± 0.6	89.7 ± 0.3	22.5 ± 0.4	40.3 ± 0.3	70.1 ± 0.4
Joint-SSL + SAM-A	43.9 ± 0.4	63.3 ± 0.5	90.4 ± 0.5	22.9 ± 0.3	41.3 ± 0.6	69.9 ± 0.6
Joint-SSL + SAM-A + M2m	44.1 ± 0.3	64.2 ± 0.4	90.9 ± 0.3	23.9 ± 0.4	42.3 ± 0.2	70.4 ± 0.3
varying their imbalance ratio, we now fix the number of samples from the minority class and vary the number of total majority class samples.
In Figure 16, we see that increasing the number of samples from the majority class initially boosts performance on a balanced test set. Nevertheless, in both cases, the performance reaches an optimum before the growing training data imbalance eventually degrades test accuracy. Thus, adding training data can help, but if we add enough majority samples, we must be careful not to cause too sharp a mismatch between training and testing distributions. Notably, the optimal training set ratio is nearly balanced, matching the test set, even when we are allowed to gather extra samples from one class without having to forego samples from another.
A.9 Benchmarking Results
In Table 5, we present additional experimental results that compare the methods proposed in our paper with the baseline methods. In accordance with Kang et al. [38], we also report the accuracy across three distinct subsets: (1) Many-shot classes, which contain more than 100 training samples. (2) Medium-shot classes, comprising 20 to 100 samples, and (3) Few-shot classes, including classes with fewer than 20 samples.
Table 6: SAM-A, our modified label smoothing, and small batch sizes improve performance on class-imbalanced tabular datasets.
Method	Otto	Adult	CoverType
XGBoost	82.7	87.5	96.9
MLP	83.0	87.4	97.5
ResNet	82.5	87.4	97.5
FT-Transformer	82.3	87.3	97.5
MLP w/ SAM-A + Smoothing	83.2	87.6	97.6
A.9.1 Tabular Datasets
Training procedure - Our tuning, training, and evaluation protocols are consistent with those in Gorishniy et al. [23]
Accordingly, we tune each model's hyperparameters for each dataset on a validation split. We use the Optuna library [2] to execute Bayesian optimization via the Tree-Structured Parzen Estimator algorithm. In our evaluation process, we run 15 experiments for each tuned configuration using different random seeds, and report the average performance on the test set.
A.10 Regularization and Overfitting
In order to determine whether the performance differences among various methods stem from their optimization abilities or their generalization to unseen test samples, we evaluate the training error without any regularization or specialized optimization method. Specifically, we train a ResNet-50 network on CIFAR-10 and CIFAR-100 datasets using SGD with an initial learning rate of 0.5 and cosine annealing, across different levels of training data imbalance. As seen in Figure 17, although fitting all training examples takes longer as we increase the imbalance ratio of our datasets, the empirical risk minimization successfully fits all training data eventually, including minority samples.
Figure 17: Imbalanced data is harder to fit. Training accuracy every epoch for imbalanced training with various imbalance ratios. Experiments conducted on CIFAR-10.
A.11 Decision Boundary Visualizations
To explore the differences between classifiers trained on imbalanced data, we visualize their decision boundaries. A variety of methods have been established for visualizing the decision boundaries of deep learning models, offering valuable insights into their intricate internal operations. Apart from the methods discussed in the main text, we utilize the approach introduced by Somepalli et al. [66] to visualize the decision boundaries of a ResNet-50 network trained on the CIFAR-10 dataset. In Figure 19, we display the decision boundaries resulting from standard training (right), which yields
(a) After naive training
(b) After SAM training
Figure 18: SAM-A pulls decision boundaries away from minority samples, whereas standard training routines overfit. Experiments conducted on a toy 2D classification problem with a multi-layer perceptron.
(a) After naive training
Figure 19: SAM-A makes decision regions take up similar volumes, whereas standard training routines shrink wrap the decision boundaries around minority samples. Experiments conducted on a CIFAR-10 with ResNet-18.
narrow margins around minority classes (green, grey, and orange), and SAM-A (left), which notably broadens these margins and all the classes occupy similar area in input space.
In this section, we provide additional implementation details that were not included in the main text.
For the CIFAR-10, CIFAR-100, and CINIC-10 datasets, we follow the imbalanced setup proposed by Liu et al. [54], using an exponential distribution to create imbalances between classes. Across all methods, we use TrivialAugment [58] combined with CutMix as our augmentation policy, supplemented by label smoothing and an exponential moving weight average. Our model of choice is the WideResNet-28×10 [73].
We employ the SGD optimizer with momentum 0.9 and weight decay coefficient  $2 \times 10^{-4}$ . Our models are trained for 300 epochs with cosine annealing and a linear warm-up of the learning rate. The learning rate is initialized at 0.1.
For the APTOS 2019 Blindness Detection, SIIM-ISIC Melanoma Classification, and EuroSAT datasets, we largely follow the approach detailed in Fang et al. [18], utilizing the ResNeXt-50-32×4d model, which was identified as the best model for these datasets in the comparison by Fang et al. [18].
Our implementation was done in PyTorch, utilizing the PyTorch Lightning library for training. All of our models were trained on V100 GPUs.
In our paper, we found that existing methods for class imbalance are unreliable on real-world datasets. While our tuned routine was effective on the real-world datasets we considered, these general trends
(d) ResNet on CIFAR-10 Dataset
(a) ResNet on CIFAR-10 Dataset
(b) XGBoost on Adult Dataset
(e) XGBoost on Adult Dataset
(c) SVM on Forest Cover Dataset
(f) SVM on Forest Cover Dataset
Figure 20: Test error split by majority and minority classes for balanced test sets. We see similar trends across all models and datasets.
Figure 21: Additional metrics for XGBoost on the Adult dataset.
Figure 22: Additional metrics for SVM on the Forest Cover dataset.
raise the concern that solutions which are effective on some class-imbalanced datasets may fail on others. A second limitation of our work is that some tools we utilize are only applicable in certain domains. For example, data augmentations and self-supervised learning for tabular data are not widely accepted.
Across a wide variety of high-impact domains, ranging from credit card fraud detection to disease diagnosis, data is severely class-imbalanced. Therefore, performance increases for class-imbalanced data is highly valuable. With this potential for value also comes the potential that proposed methods make false promises which won't benefit real-world practitioners and may in fact cause harm when deployed in sensitive applications. For this reason, we release our numerical results across diverse datasets, and we also include implementation details for the sake of transparency and reproducibility. As with all new state-of-the-art methods, our improvements may also improve models used for malicious intentions.