SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiang-Xin Shi $^{1*}$  Tong Wei $^{2*}$  Yuke Xiang $^{3}$  Yu-Feng Li $^{1\dagger}$
$^{1}$ National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
$^{2}$ School of Computer Science and Engineering, Southeast University, Nanjing, China
<sup>3</sup>Consumer BG, Huawei Technologies, Shenzhen, China
{shijx,liyf}@lamda.nju.edu.cn，weit@seu.edu.cn，yuke.xiang@huawei.com
Long-tail learning has received significant attention in recent years due to the challenge it poses with extremely imbalanced datasets. In these datasets, only a few classes (known as the head classes) have an adequate number of training samples, while the rest of the classes (known as the tail classes) are infrequent in the training data. Re-sampling is a classical and widely used approach for addressing class imbalance issues. Unfortunately, recent studies claim that re-sampling brings negligible performance improvements in modern long-tail learning tasks. This paper aims to investigate this phenomenon systematically. Our research shows that re-sampling can considerably improve generalization when the training images do not contain semantically irrelevant contexts. In other scenarios, however, it can learn unexpected spurious correlations between irrelevant contexts and target labels. We design experiments on two homogeneous datasets, one containing irrelevant context and the other not, to confirm our findings. To prevent the learning of spurious correlations, we propose a new context shift augmentation module that generates diverse training images for the tail class by maintaining a context bank extracted from the head-class images. Experiments demonstrate that our proposed module can boost the generalization and outperform other approaches, including class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods. The source code is available at https://www.lamda.nju.edu.cn/code_CSA.ashx.
Deep neural networks have achieved great success by applying well-designed models on large-scale elaborated datasets [1, 2, 3]. However, real-world data often exhibits a long-tail class distribution [4, 5, 6]. Learning from long-tail datasets has two main challenges, one is the class-imbalanced problem which causes the model biased towards the dominated head classes, and another is the data scarcity problem leading to the poor generalization on those rare tail classes [7, 8, 9, 10].
One simple and intuitive approach to deal with the class-imbalanced problem is re-sampling [11, 12], i.e., create the replicate of the dataset to estimate the model parameters. Unfortunately, it has been reported that re-sampling methods achieve limited effects when applied to most long-tail datasets [6, 13]. Currently, there are still few concrete and comprehensive explanations for this observation. Existing works mainly conclude that re-sampling will lead to the overfitting problem, and thus will be harmful to long-tail representation learning [6, 14].
Recently, many two-stage approaches have been proposed to improve the tail-class performance by adopting re-sampling in the second training stage. For instance, DRS [15] adopts a re-sampling
schedule at the last several episodes of the training process. cRT [6] first trains a preliminary model using the uniform sampler, then fixes the representations and re-trains the linear classifier using a class-balanced sampler. Last but not least, BBN [14] adjusts the whole model to first learn from the conventional learning branch and dynamically move to the re-balancing branch. Overall, the two-stage method has attracted widespread attention due to its basic hypothesis that uniform sampling is beneficial to representation learning, and the class-balanced sampling can be used to fine-tune the linear classifier. In light of the success of the two-stage method, a natural question is:
Can re-sampling benefit long-tail learning in the single-stage framework?
To answer this question, this paper empirically studies the re-sampling strategies and finds that re-sampling leads to opposite effects on long-tail datasets. Figure 1 gives a brief view of this phenomenon. Moreover, we deduce that if the training samples are highly semantically related to their target labels, class-balanced re-sampling can learn discriminative feature representations; otherwise, uniform sampling is even better than class-balanced re-sampling, as the latter suffers from oversampling redundant unrelated contexts. To verify this, we design a pair of synthetic benchmarks with the same content but different contexts, one containing irrele
Figure 1: Performance of re-sampling on two long-tail datasets. The sampling weight for a data point  $(x,y)$  is defined as  $n_y^{-\gamma}$  where  $n_y$  denotes the class frequency of class  $y$ .
vant context in training samples and the other not. Experiments confirm that re-sampling achieves conspicuous different performances on these two benchmarks. In particular, when irrelevant context exists, class-balanced re-sampling learns poorer representations compared to uniform sampling, thus the irrelevant context negatively affects re-sampling methods.
We believe that re-sampling can benefit long-tail learning in the single-stage framework. It fails on some long-tail datasets mainly because it overfits the oversampled irrelevant contexts and learns unexpected spurious correlations. If such spurious correlations are avoided, re-sampling can be helpful for long-tail learning. Motivated by this, we propose a new context-shift augmentation module, which transfers well-separated context from head-class data to tail-class data. Specifically, it extracts the unrelated contexts (e.g. the backgrounds or the unrelated foreground objects) from head-class images and pastes them onto tail-class images to generate diverse novel samples. In this way, it encourages the model to learn more discriminative results for the tail classes. We conduct experiments on three long-tail datasets, CIFAR10-LT, CIFAR100-LT, and ImageNet-LT. The results show that the proposed module achieves competitive performance compared to the baseline methods. In summary, our main contributions are:
- We conduct empirical analyses on different datasets and discover that re-sampling does not necessarily work or fail in long-tail learning.
- We deduce that the failure of re-sampling may be attributed to overfitting on irrelevant contexts, and our empirical studies confirm our hypothesis.
- We propose a new context-shift augmentation module to prevent re-sampling from overfitting to irrelevant contexts in a single-stage framework.
- Extensive experiments verify the effectiveness of the proposed module against class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods.
The rest of the paper is organized as follows. Section 2 studies the effects of re-sampling approaches. Section 3 presents the proposed context-shift augmentation module. Section 4 briefly reviews related works. Section 5 concludes the paper.
2 A Closer Look at Re-sampling
Given a training dataset  $\mathcal{D} = \{\pmb{x}_i, y_i\}_{i=1}^N$ , where  $\pmb{x}_i$  is a training sample and  $y_i \in \mathcal{C} = [K] = \{1, \dots, K\}$  is the class label assigned to it. We assume that the training data follows a long-tail class
Table 1: Test accuracy (\%) of CE with uniform sampling, classifier re-training (cRT), and class-balanced re-sampling (CB-RS) on four long-tail benchmarks. We report the accuracy in terms of all, many-shot, medium-shot, and few-shot classes.
MNIST-LT	Fashion-LT	CIFAR100-LT	ImageNet-LT
All	Many	Med.	Few	All	Many	Med.	Few	All	Many	Med.	Few	All	Many	Med.	Few
CE	65.8	99.1	89.9	0.0	45.6	94.7	43.1	0.0	39.1	65.8	36.8	8.8	35.0	57.7	26.5	4.7
cRT	82.5	96.6	89.4	58.8	60.3	77.1	61.4	42.1	41.6	63.0	40.4	16.5	41.9	52.9	39.2	23.6
CB-RS	90.8	98.7	94.4	77.7	80.5	86.6	74.3	82.8	34.1	59.5	31.1	6.2	37.6	47.5	36.5	16.7
(a) Uniform sampling.
Figure 2: Visualization of learned representation of training and test set on MNIST-LT. Using class-balanced re-sampling yields more discriminative and balanced representations.
(b) Class-balanced re-sampling.
distribution where the class prior distribution  $\mathbb{P}(y)$  is highly skewed so that many underrepresented classes have a very low probability of occurrence. Specifically, we define the imbalance ratio as  $\rho = \max_y\mathbb{P}(y) / \min_y\mathbb{P}(y)$  to indicate the skewness of data. Classes with high  $\mathbb{P}(y)$  are referred to as head classes, while others are referred to as tail classes.
In practice, since the data distribution is unknown, Empirical Risk Minimization (ERM) uses the training data to achieve an empirical estimate of the underlying data distribution. Typically, one minimizes the softmax cross-entropy as following
$$
\ell (y, f (\boldsymbol {x})) = - \log \frac {\exp \left(f _ {y} (\boldsymbol {x})\right)}{\sum_ {y ^ {\prime} \in [ K ]} \exp \left(f _ {y ^ {\prime}} (\boldsymbol {x})\right)} \tag {1}
$$
where  $f_{y}(\pmb{x})$  denotes the predictive logit of model  $f$  on class  $y$ . However, this ubiquitous approach neglects the issue of class imbalance and makes the model biased toward head classes [8, 16].
To deal with the class-imbalance problem, the re-sampling strategy assigns a probability of being selected for each training sample according to its class frequency [6]. The probability of sampling a data point from class  $k$  can be written as:
$$
p _ {k} = \frac {n _ {k} ^ {q}}{\sum_ {k ^ {\prime} \in [ K ]} n _ {k ^ {\prime}} ^ {q}} \tag {2}
$$
where  $n_k$  denotes the frequency of class  $k$  and  $q \in [0,1]$ . When  $q = 1$ , Equation (2) denotes uniform sampling, where each training sample has an equal probability of being selected. When  $q = 0$ , Equation (2) denotes the class-balanced re-sampling, which selects samples from every class  $k$  with the identical probability of  $1 / K$ .
2.2 Exploring the Effect of Re-sampling
2.2.1 Re-sampling can learn discriminative representations
To better explore the effect of the re-sampling strategy, we conduct experiments on multiple long-tail datasets, including MNIST-LT, Fashion-LT, CIFAR100-LT [15], and ImageNet-LT [5]. We compare three different learning methods: 1) Cross-Entropy (CE) with uniform sampling; 2) Classifier Re-Training (cRT) which uses uniform sampling to learn the representation and class-balanced resampling to fine-tune the classifier; 3) Class-Balanced Re-Sampling (CB-RS) for the whole training process. We report the experimental results in Table 1.
Figure 3: Visualization of features with Grad-CAM [17] on CIFAR100-LT. Uniform sampling mainly learns label-relevant features, while re-sampling overfits the label-irrelevant features.
(a) Comparison of class accuracy.
Figure 4: Comparison of Uniform sampling, cRT, and CB-RS on MNIST-LT and CMNIST-LT.
(b) Overall accuracy.
According to the results, cRT performs best on CIFAR100-LT and ImageNet-LT, which is consistent with previous works [14, 6]. CE and cRT use the same representation but cRT achieves higher performances, which indicates that re-sampling can help for classifier learning. However, on MNIST-LT and Fashion-LT, CB-RS surprisingly achieves the highest performance and outperforms CE and cRT by a large margin. Since cRT and CB-RS both use class-balanced re-sampling for classifier learning, the results indicate that CB-RS learns better representations than uniform sampling on MNIST-LT and Fashion-LT.
To further understand the effect of re-sampling, we visualize the learned representation on MNIST-LT in Figure 2. The figures show that with uniform sampling, the representation space is dominated by head classes, and the representations of tail classes are hard to distinguish. By applying class-balanced re-sampling, the representations of both head and tail classes are discriminative.
2.2.2 Re-sampling is sensitive to irrelevant contexts
We have demonstrated the generalization ability of the re-sampling strategy on MNIST-LT and Fashion-LT. Nevertheless, re-sampling performs unsatisfactorily on the other two datasets. Since the training samples and target labels on MNIST and Fashion are highly semantically correlated [18, 19], while samples on CIFAR and ImageNet contain complex contexts [20, 3], we hypothesize that re-sampling is sensitive to the contexts in training samples.
To support our hypothesis, we visualize the Grad-CAM of the representation learned by different sampling strategies in Figure 3. When training with a uniform sampler, models can distinguish the contexts from samples of head classes. However, when adopting a class-balanced re-sampler, the model tends to overfit the irrelevant context from the over-sampled tail data, which unexpectedly affects the representation of head classes. For example, when classifying different kinds of animals, re-sampling might focus on the posture rather than the appearance. Also, when classifying different vehicles, re-sampling is easily influenced by the human in tail-class images, and thus mistakenly focuses on the human in head-class images.
To further validate our point, we design a homogeneous benchmark of MNIST-LT termed Colored-MNIST-LT (CMNIST-LT). We inject colors into MNIST-LT to artificially construct irrelevant contexts.
Specifically, we design CMNIST-LT based on two considerations. First, head classes are prone to have rich contexts, so we inject different colors into the samples of each head class. Second, tail classes have limited contexts, so we inject an identical color into the samples of each tail class. We conduct uniform sampling, classifier re-training, and class-balanced re-sampling on MNIST-LT and CMNIST-LT. The experimental results are illustrated in Figure 4. The results show that when applied to MNIST-LT, CB-RS can boost the tail-class performance without degradation on head classes. However, for CMNIST-LT, CB-RS performs worse than uniform sampling and cRT on both head and tail classes, thus validating the negative impact of irrelevant contexts on re-sampling methods. Since re-sampling succeeds on MNIST-LT and fails on CMNIST-LT, we propose that re-sampling does not always fail, it can help for long-tail learning if avoiding the irrelevant contexts.
2.2.3 Proposed benchmark datasets
We follow the previous works [15, 14] to construct MNIST-LT, Fashion-LT and CIFAR100-LT and set the imbalance ratio to 100. ImageNet-LT is proposed by [5]. For MNIST-LT and Fashion-LT, we use LeNet [21] as the backbone network and add a linear embedding layer before the fully connected layer to project the representation into 2-dimensional space for better presentation. We use standard SGD with a mini-batch size of 128, an initial learning rate of 0.1 and a cosine annealing schedule to train the model for 8 epochs. When applying cRT, we retrain the last fully connected layer for 4 epochs by fixing the other layers. For CIFAR100-LT and ImageNet-LT, more details are in Section 3.3.
To construct CMNIST-LT, we follow the idea of CMNIST [22] to first randomly flip the label and then inject colors into the training samples. However, different from CMNIST which converts the MNIST to a binary classification dataset, we keep the ten classes to better simulate a long-tail class distribution. To generate flipped labels on the long-tail dataset MNIST-LT, we follow the method in [23] and set the flipping probability to  $1/4$ . We generate ten different colors using the seaborn<sup>2</sup> package. For the five head classes, we randomly inject one of these ten colors into each sample with equal probability. For the other five tail classes, we inject samples of each class with a single color.
3 A Simple Approach to Make Re-sampling Robust to Context-shift
3.1 Extracting Rich Contexts from Head-class Data
By studying the effects of re-sampling methods in different scenarios, we can draw a conclusion: when the training samples contain irrelevant contexts, simply over-sampling the tail-class samples might cause the model to unexpectedly focus on these redundant contexts, thereby resulting in the overfitting problem. However, the head classes have rich data to learn a model with good generalization ability. We naturally raise a question: can we utilize the rich contexts from head data to augment the over-sampled tail data to alleviate the negative impact of irrelevant contexts? Inspired by this motivation, we design a context-shift augmentation module by extracting the rich contexts from head-class data to enrich the over-sampled tail-class data.
To leverage the rich contexts in head-class data, we utilize a model  $f^u$  trained with uniform sampling for context extraction. First, we select well-learned samples with fitting probability larger than a threshold  $\delta$  to improve the extraction quality. The fitting probability for sample  $x_i$  can be calculated by the Softmax function as follows
$$
p (y \mid \boldsymbol {x} _ {i}, f ^ {u}) = \frac {\exp \left(\boldsymbol {z} _ {i , y} ^ {u}\right)}{\sum_ {y ^ {\prime} \in [ K ]} \exp \left(\boldsymbol {z} _ {i , y ^ {\prime}} ^ {u}\right)} \tag {3}
$$
where  $\boldsymbol{z}_i^u$  denotes the logits predicted by  $f^u$ , i.e.,  $\boldsymbol{z}_i^u = [z_{i,1}^u,\dots,z_{i,K}^u] = f^u(\boldsymbol{x}_i)$ . Then, we use off-the-shelf methods such as Grad-CAM [24, 17] to extract the image contexts, which is also used in previous works regarding open-set learning and adversarial learning [25, 26]. Specifically, given an image  $x_i$ , we calculate its class activation map  $\mathrm{CAM}(\boldsymbol{x}_i\mid f^u)$ . Then, we inverse the map to get the background mask  $M_i$ , i.e.,  $M_i = 1 - \mathrm{CAM}(\boldsymbol{x}_i\mid f^u)$ . Here  $M_i$  is a matrix of the same size as  $x_i$ , with values between 0 and 1. A higher value indicates that the corresponding pixel is more likely to be the background. Different from previous works that discretize the mask matrix to binary values [27, 28], we keep the floating values in the matrix to conserve more information.
<sup>2</sup>http://seaborn.pydata.org/generated/seaborn.color Palette.html
After calculating the background mask  $M_{i}$  of image  $x_{i}$ , we paste the mask onto the original image by  $M_{i} \odot x_{i}$  to obtain a background image. In this way, we separate the semantically related contents from the images and keep the rest contexts for further augmentation. Finally, the extracted contexts are pushed into a memory bank  $Q$  for augmentation of re-sampled data.
For the training of the uniform module, we apply the conventional ERM algorithm. For each training sample  $\boldsymbol{x}_i$ , we calculate its loss by
$$
\boldsymbol {z} _ {i} ^ {u} = f ^ {u} \left(\boldsymbol {x} _ {i}\right) \tag {4}
$$
$$
\mathcal {L} _ {i} ^ {u} = \ell^ {u} \left(\boldsymbol {z} _ {i} ^ {u}, y _ {i}\right) \tag {5}
$$
where  $\ell^u$  can be any loss function. Generally, we use the standard cross-entropy loss.
3.2 Balanced Re-sampling with Context-shift Augmentation
Simply adopting balanced sampling might generate many repeated samples from the tail classes and lead to the overfitting problem. Therefore, we ask for background images from the context memory bank  $Q$  and paste them onto the re-sampled images. In this way, we generate more diverse novel samples by simulating each tail-class image within various contexts. Specifically, for a re-sampled training image  $\tilde{\boldsymbol{x}}_i$ , we ask for another image  $\tilde{\boldsymbol{x}}_i$  together with its mask  $M_i$  from  $Q$ , and fuse it with  $\tilde{\boldsymbol{x}}_i$  to generate a novel sample, and calculate its training loss as follow:
$$
\lambda \sim \operatorname {U n i f o r m} (0, 1) \tag {6}
$$
$$
\tilde {\boldsymbol {x}} _ {i} = \lambda \boldsymbol {M} _ {i} \odot \check {\tilde {\boldsymbol {x}}} _ {i} + (1 - \lambda \boldsymbol {M} _ {i}) \odot \tilde {\boldsymbol {x}} _ {i} \tag {7}
$$
$$
\boldsymbol {z} _ {i} ^ {b} = f ^ {b} \left(\tilde {\boldsymbol {x}} _ {i}\right) \tag {8}
$$
$$
\mathcal {L} _ {i} ^ {b} = \ell^ {b} \left(\boldsymbol {z} _ {i} ^ {b}, \tilde {\mathcal {Y}} _ {i}\right) \tag {9}
$$
Here  $\lambda$  is randomly generated between [0, 1] to increase the diversity. Different from previous mixup-based methods[29, 30], our method does not change the target label, because the pasted background is not related to the semantics of any class labels.
To reduce the computational complexity, the uniform module and the balanced re-sampling module can be trained simultaneously by sharing the same feature extractor  $\phi$ , and training their own linear classifiers  $\psi^u$  and  $\psi^b$ , i.e.,  $f^u(\cdot) = \psi^u(\phi(\cdot))$  and  $f^b(\cdot) = \psi^b(\phi(\cdot))$ . Since the classifier is lightweight, it does not add much additional computational overhead. Moreover, the memory bank  $Q$  is designed as a first-in-first-out queue with a maximum volume of  $V$  for a convenient query. After extracting the context from the uniform module, we append the  $\pmb{x}_i$  and  $M_i$  pair into the context bank  $Q$ . When the size of  $Q$  reaches its maximized volume, the oldest contexts are pushed out. In practice, the volume size is set equal to the mini-batch size in order to minimize the overhead as well as ensure the querying requirements. Finally, the summarized loss function is
$$
\mathcal {L} = \mathcal {L} ^ {u} + \mathcal {L} ^ {b} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {i} ^ {u} + \frac {1}{N} \sum_ {i = 1} ^ {N} \mathcal {L} _ {i} ^ {b} \tag {10}
$$
Figure 5 gives a brief overview of the proposed module. The detailed training procedure is given in the supplementary material due to the page limit.
In the inference phase, only the balanced re-sampling module is used. In other words, the uniform module only serves as an assistant to provide more rich contexts for the re-sampling module during the training phase. Formally, for a test data point  $\pmb{x}$ , we obtain the prediction by  $z = f^{b}(\pmb{x})$ , and then employ the Softmax function to obtain the predictive probabilities.
3.3 Empirical Results
We demonstrate the efficacy of the proposed module context shift augmentation by comparing it with different kinds of long-tail learning methods, including:
- Re-sampling or re-weighting methods, such as Focal Loss [31], CB-Focal [7], CE-DRS [15], CE-DRW [15], LDAM-DRW [15], cRT [6], LWS [6], and BBN [14],
- Head-to-tail knowledge transfer methods, such as M2m [32], OLTR [5], and FSA [28],
Figure 5: An overview of the proposed method.
- Data augmentation methods, such as Mixup [29], Remix [33], CAM-BS [13], and CMO [27].
We conduct experiments on three long-tail datasets, including CIFAR10-LT [15], CIFAR100-LT [15], and ImageNet-LT [5]. CIFAR10-LT and CIFAR100-LT are the long-tail versions of CIFAR datasets by sampling from the raw dataset with an imbalance ratio  $\rho$ . Following previous works [15, 14], we conduct experiments with  $\rho \in \{100, 50, 10\}$ . ImageNet-LT is a long-tail version of the ImageNet [3], which contains 1000 classes, each with a number of samples ranging from 5 to 1280. For each long-tail training dataset, we evaluate the model on another corresponding class-balanced test set by calculating the overall prediction accuracy.
The results for CIFAR10-LT and CIFAR100-LT are summarized in Table 2. We report the results under imbalance ratio  $\rho = 100,50,10$ . As shown in the table, our method achieves superior performance compared with the baseline methods in all settings.
Table 2: Test accuracy (%) on CIFAR datasets with various imbalanced ratios.
Dataset	CIFAR100-LT	CIFAR10-LT
Imbalance Ratio	100	50	10	100	50	10
CE	38.3	43.9	55.7	70.4	74.8	86.4
Focal Loss [31]	38.4	44.3	55.8	70.4	76.7	86.7
CB-Focal [7]	39.6	45.2	58.0	74.6	79.3	87.1
CE-DRS [15]	41.6	45.5	58.1	75.6	79.8	87.4
CE-DRW [15]	41.5	45.3	58.1	76.3	80.0	87.6
LDAM-DRW [15]	42.0	46.6	58.7	77.0	81.0	88.2
cRT [6]	42.3	46.8	58.1	75.7	80.4	88.3
LWS [6]	42.3	46.4	58.1	73.0	78.5	87.7
BBN [14]	42.6	47.0	59.1	79.8	82.2	88.3
mixup [29]	39.5	45.0	58.0	73.1	77.8	87.1
Remix [33]	41.9	-	59.4	75.4	-	88.2
M2m [32]	43.5	-	57.6	79.1	-	87.5
CAM-BS [13]	41.7	46.0	-	75.4	81.4	-
CMO [27]	43.9	48.3	59.5	-	-	-
cRT+mixup [34]	45.1	50.9	62.1	79.1	84.2	89.8
LWS+mixup [34]	44.2	50.7	62.3	76.3	82.6	89.6
CSA (ours)	45.8	49.6	61.3	80.6	84.3	89.8
CSA + mixup (ours)	46.6	51.9	62.6	82.5	86.0	90.8