SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, the methods based on re-sampling or re-weighting such as DRS, DRW, cRT, and LWS can ease the class-imbalanced problem to some degree but the performance gain is limited due to the neglect of the representation learning. The methods based on knowledge transfer and data augmentations, such as M2m and CMO, achieve higher performance.
Moreover, by combining cRT and LWS with mixup, the performance achieves an obvious improvement. However, such two-stage training methods are not end-to-end approaches. In contrast, the proposed context shift augmentation module enhances representation learning by enriching the contexts of samples and adopting the class-balanced re-sampling to ensure a balanced classifier. In this manner, it achieves more improvement with an end-to-end framework. Since cRT and the proposed module both use class-balanced sampling for classifier learning, the results indicate that the proposed context shift augmentation can achieve better representations.
We further conduct the experiments on a larger scale dataset ImageNet-LT. We calculate the accuracy of the overall test set and the average accuracy of the many-shot classes (more than 100 images in the training set), the medium-shot  $(20\sim 100$  images), and the few-shot classes (less than 20 images). We report the results in Table 3. It shows that the proposed module is superior to most other baseline methods. The performance is similar to another data augmentation method CMO, but our method achieves higher accuracy on few-shot classes, which demonstrates the generalization ability of context-shift augmentation for tail-class data.
Table 3: Test accuracy (%) on ImageNet-LT dataset.
ResNet-10 (All)	All	ResNet-50	Few
Many	Med.
CE	34.8	41.6	64.0	33.8	5.8
Focal Loss [31]	30.5	-	-	-	-
OLTR [5]	35.6	-	-	-	-
FSA [28]	35.2	-	-	-	-
cRT [6]	41.8	47.3	58.8	44.0	26.1
LWS [6]	41.4	47.7	57.1	45.2	29.3
BBN [14]	-	48.3	-	-	-
CMO [27]†	-	49.1	67.0	42.3	20.5
CSA (ours)	42.7	49.1	62.5	46.6	24.1
CSA† (ours)	43.2	49.7	63.6	47.0	23.8
$^\dagger$  denotes a longer training of 100 epochs.
We provide more detailed studies to analyze the effect of each component in the proposed module. Due to the page limit, we report the results in the supplementary material.
4.1 Re-sampling and Re-weighting
Re-sampling is a widely used strategy in class-imbalanced learning [11, 12]. There are two main ideas of re-sampling: 1) Over-sampling by repeating data from the rare classes. 2) Under-sampling by abandoning a proportion of data from the frequent classes. However, when the class distribution is highly skewed, re-sampling methods often fail. Previous works point out that under-sampling may discard precious information which inevitably degrades the model performance, and over-sampling tends to cause the overfitting problem on the tail classes [14, 15].
Recent work [6] develops class-balanced re-sampling where samples from each class have an identical probability of being sampled. Class-balance re-sampling can bring performance gain for classifier learning but hurts representation learning. Therefore, two-stage approaches [15, 6, 14] adopt it at the late stage of the whole training process in order not to impact the representation.
Re-weighting aims to generate more balanced predictions by adjusting the losses for different classes [35, 36]. The most intuitive way is to weight each training sample by the inverse of its class frequency [4]. Similar to class-balanced re-sampling, re-weighting can achieve better results for tail classes but usually deteriorates its performance for head classes [7].
In contrast, our empirical study reveals that class-balanced re-sampling can be an effective method as long as there exist no irrelevant contexts. It fails in some cases mainly due to the unexpected overfitting
towards the over-sampled redundant contexts. When applied with context-shift augmentation, class-balanced re-sampling can achieve competitive performance on long-tail datasets.
4.2 Head-to-tail Knowledge Transfer
As the head classes have adequate training data while the tail classes have limited data, recent works aim to leverage the knowledge gained from head classes to enhance the generalization of tail classes. Feature transfer learning [37] utilizes the intra-class variance from head classes to guide the feature augmentation for tail classes. MetaModelNet [4] uses the head data to train a meta-network to predict many-shot model parameters from few-shot model parameters, then transfers the meta-knowledge to the tail classes. Major-to-minor translation (M2m) [32] uses the over-sampling method and translates the head-class samples to replace the duplicated tail-class samples via adversarial perturbations. OLTR [5] maintains a dynamic meta-embedding between head and tail classes to transfer the semantic deep features from head to tail classes.
These methods assume that the head classes and the tail classes share some common knowledge such as the same intra-class variances, the same model parameters, or the same semantic features. In this work, we regard the contexts as such knowledge and transfer the contexts from head-class data to enrich the tail-class data.
4.3 Data Augmentation
Several data augmentation approaches have been proposed to improve the model generalization ability. In contrastive learning [38, 39], curriculum learning [40], meta-learning methods [41] and instance segmentation tasks [42], data augmentation strategies have been shown to effectively improve the generalization of tail classes. MiSLAS [34] studies the mixup [29] technology in long-tail learning and finds that mixup can have a positive effect on representation learning but a negative or negligible effect on classifier learning. Remix [33] adapts the mixup method to a re-balanced version. It assigns the mixed label in favor of the tail class by designing a disproportionately higher weight for the tail class. CMO [27] applies CutMix [30] by cutting out random regions of a sample from head classes and filling the removed regions with another sample from tail classes. By this means, it enriches the contexts of the tail data; but the random cutout operation does not necessarily separate the contents and contexts.
The attention or CAM-based methods have been proposed to improve long-tail learning via feature decomposition and augmentation. CAM-BS [13] separates the foreground and background of each sample, then augments the foreground part by flipping, translating, rotating, or scaling. Feature Space Augmentation (FSA) [28] uses CAM to decompose the features of each class into a class-generic component and a class-specific component, and generates novel samples in the feature space by combining the class-specific components from the tail classes and the class-generic components from the head classes. Attentive Feature Augmentation (AFA) [43] adopts feature decomposition and augmentation via the attention mechanism. Note that FSA and AFA can also be seen as head-to-tail knowledge transfer approaches. Nevertheless, these methods neglect that the learned model has limited generalization ability on tail classes, and most foregrounds (or class-specific components) of samples from tail classes are incredible. In comparison, our method applies CAM to separate related contents and unrelated contexts for samples mainly from head classes. It then pastes the contexts extracted from the head-class data onto the over-sampled tail-class data to enrich the contexts.
In this work, we study the re-sampling strategy for long-tail learning. Our empirical investigations reveal that the impact of re-sampling is highly dependent on the existence of irrelevant contexts and is not always harmful to long-tail learning. To reduce the influence of irrelevant contexts, we propose a new context-shift augmentation module that leverages the well-separated contexts from the head-class images to augment the over-sampled tail-class images. We demonstrate the superiority of the proposed module by conducting experiments on several long-tail datasets and comparing it against class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods.
Broader Impact and Limitations
This paper investigates the reasons behind the success/failure of re-sampling approaches in long-tail learning. In critical and high-stakes applications, such as medical image diagnosis and autonomous driving, the presence of imbalanced data poses the risk of producing biased predictions. By shedding light on this problem, we aim to inspire more research on safe and robust re-sampling approaches.
One may be concerned about combining the proposed module with other methods such as self-supervised learning [44], logit adjustment [16]. We conduct additional experiments and report the results in the supplementary material due to the page limit. Nevertheless, the proposed module can not achieve comparable performance with the well-designed models [45, 46, 47], since our intention is not to achieve performance that is on par with state-of-the-art methods. Instead, we hope that our findings will inspire future research regarding re-sampling methods.
Data Availability Statement
The source code of our method is available at https://www.lamda.nju.edu.cn/code_CSA.ashx or https://github.com/shijxcs/CSA.
This research was supported by the National Key R&D Program of China (2022ZD0114803), the National Science Foundation of China (62176118, 61921006).
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444, 2015.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[4] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, volume 30, pages 7029-7039, 2017.
[5] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[6] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
[7] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268-9277, 2019.
[8] Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems, volume 33, pages 4175–4186, 2020.
[9] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9495-9504, 2021.
[10] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems, volume 33, pages 19290-19301, 2020.
[48] Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. arXiv preprint arXiv:1909.09148, 2019.
A Training Procedure
The training procedure of context-shift augmentation is summarized in Algorithm 1.
Algorithm 1 Training procedure of context-shift augmentation
Input: training data  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^N$  ; context memory bank  $Q$  , maximum volume size  $V$  ; model parameters  $\phi ,f^{u},f^{b}$  ; loss functions  $\ell^u,\ell^b$    
Procedure:   
1: Initialize model parameters  $\phi ,f^{u},f^{b}$    
2: Re-sampling a class-balanced dataset  $\widetilde{\mathcal{D}} = \{(\tilde{x}_i,\tilde{y}_i)\}_{i = 1}^N$    
3: Empty memory bank  $Q$    
4: for epoch  $= 1,\dots ,T$  do   
5: repeat   
6: Draw a mini-batch  $(\pmb {x}_i,\pmb {y}_i)^B_{i = 1}$  from  $\mathcal{D}$    
7: Draw a mini-batch  $(\tilde{x}_i,\tilde{y}_i)^B_{i = 1}$  from  $\widetilde{\mathcal{D}}$    
8: // uniform module   
9: for  $i = 1,\ldots ,B$  do   
10: Calculate  $z_i^u = f^u (\phi (x_i))$  and  $\mathcal{L}_i^u = \ell^u (z_i^u,y_i)$  .   
11: if  $p(y = y_{i}\mid x_{i},\phi ,f^{u})\geq \delta$  then   
12: Calculate background mask  $M_{i}$  of  $x_{i}$  .   
13: Push  $(x_{i},M_{i})$  into  $Q$  .   
14: end if   
15: end for   
16: Calculate  $\mathcal{L}^u = \frac{1}{B}\sum_{i = 1}^B\mathcal{L}_i^u$  .   
17: // balanced re-sampling module   
18: if Size of  $Q$  reaches  $V$  then   
19: Obtain contexts  $(\tilde{\pmb{x}}_i,M_i)_i = 1$  from  $Q$  .   
20:  $\lambda \sim$  Uniform(0,1);   
21: for  $i = 1,\dots ,B$  do   
22:  $\tilde{x}_i = \lambda M_i\odot \tilde{x}_i + (1 - \lambda M_i)\odot \tilde{x}_i;$    
23: Calculate  $z_i^b = f^b (\phi (\tilde{x}_i))$  and  $\mathcal{L}_i^b = \ell^b (z_i^b,\tilde{y}_i)$  .   
24: end for   
25: Calculate  $\mathcal{L}^b = \frac{1}{B}\sum_{i = 1}^B\mathcal{L}_i^b$  .   
26: else   
27: Assign  $\mathcal{L}^b = 0$  .   
28: end if   
29: // total objective function   
30: Calculate  $\mathcal{L} = \mathcal{L}^u +\mathcal{L}^b$  .   
31: Update model parameters  $\phi ,f^{u},f^{b}$  with  $\mathcal{L}$  .   
32: until all training data are traversed.   
33: end for
B Implementation Details for context-shift augmentation
For experiments on CIFAR10-LT and CIFAR100-LT, we use ResNet-32 as the backbone network and train it using standard SGD with a momentum of 0.9, a weight decay of  $2 \times 10^{-4}$ , a batch size of 128. The model is trained for 200 epochs. The initial learning rate is set to 0.2 and is annealed by a factor of 10 at 160 and 180 epochs. We train each model with 1 NVIDIA GeForce RTX 3090.
For experiments on ImageNet-LT, we implement the proposed method on ResNet-10 and ResNet-50. We use standard SGD with a momentum of 0.9, a weight decay of  $5 \times 10^{-4}$ , and a batch size of 256 to train the whole model for a total of 90 epochs. We use the cosine learning rate decay with an initial learning rate of 0.2. We train each model with 2 NVIDIA Tesla V100 GPUs.
In all experiments, we first warm up the uniform module for 10 epochs and then train the uniform module and the balanced re-sampling module simultaneously for the rest epochs. For the uniform module, we follow the simple data augmentation used in [2] with only random crop and horizontal
flips. For the re-sampling module, we use the proposed context-shift augmentation method. We apply the trick proposed by [48] to disable the augmentation in the balanced re-sampling module at the last 3 epochs to obtain further improvements, which is also applied in other baseline methods [13, 27]. We set the threshold  $\delta$  to 0.8.
C Additional Illustrations
For convenience in understanding context and content, we give an example in Figure 6. Context refers to the semantically unrelated parts in the images, and content refers to the semantically related parts. Moreover, We give a brief illustration of the generated dataset CMNIST-LT in Figure 7.
Figure 6: Illustration of context and content. Taking a photo of the bicycle as an example.
Figure 7: Illustration of the CMNIST-LT benchmark.
D.1 Effects of the context bank
The context bank  $Q$  is a novel component of context-shift augmentation which receives diverse contexts from the uniform module and provides them to augment the data in the re-sampling module. To verify the effectiveness of the context bank, we remove it from the framework and train the model on CIFAR100-LT with an imbalance ratio of 100. The results are reported in Table 4.
Table 4: Ablation study on the context bank  $Q$ .
All	Many	Med.	Few
Ours	45.8	64.3	49.7	18.2
Ours w/o Q	41.2	65.1	41.9	10.7
(-4.6)	(+0.8)	(-7.8)	(-7.5)
The results show that without context bank  $Q$ , the performance decreases by a large margin. The performance degradation mostly comes from the medium-shot classes and the few-shot classes, which indicates that the context bank can significantly improve the generalization of tail classes.
Moreover, we study the effect of different variants in the context bank. First, we study the threshold  $\delta$  for sample selection and report the results in Table 5. On the one hand, if  $\delta$  is too high, the selected samples will be very few. On the other hand, if  $\delta$  is too small, the selected samples might be not well learned. Nevertheless, as the training process progresses, most samples will fit well, so our method is not sensitive to  $\delta$ . We set  $\delta = 0.8$  considering its best performance.
Second, we study the influence of the volume size  $V$  of the context bank  $Q$  and report the results in Table 6. Since the bank  $Q$  is a first-in-first-out queue, the latest incoming contexts are more convincing. When the volume size is too large, the bank might contain more past samples. Besides, a larger size of  $V$  would bring more memory overhead. So we set the volume size  $V$  equal to the mini-batch size  $B$  in our method.
Table 5: Influence of the threshold  $\delta$  .
δ	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
Accuracy	45.47	45.37	45.55	44.83	45.52	45.59	45.42	45.08	45.83	44.93
Table 6: Influence of the bank volume size (compared with the mini-batch size  $B$ ).
Volume	×1	×2	×4	×8	×16	×32	×64
Accuracy	45.83	45.60	45.57	45.32	45.55	45.37	45.26
D.2 Influence of augmentation variants
We use a variant  $\lambda \sim \mathrm{Uniform}(0,1)$  for generating novel samples. The value of  $\lambda$  can result in different proportions of foreground and background in the novel sample. Also, the size of the sampling space affects the diversity of the novel images. To explore the effect of  $\lambda$ , we try  $\lambda \sim \mathrm{Uniform}(a,b)$  and  $\lambda \sim \mathrm{Beta}(a,b)$  to train context-shift augmentation on CIFAR100-LT with imbalance ratio 100 and report the results in Figure 8.
First, when  $\lambda$  is close to 0, the background merely takes effect, and the performance decreases a lot. Second, when  $\lambda = 1$ , the background image might cover the important content in the foreground image. Also, the diversity of new samples is limited. Although the performance is better than that of  $\lambda = 0$ , it is still unsatisfactory. Overall, choosing  $\lambda \sim \mathrm{Uniform}(0,1)$  or  $\lambda \sim \mathrm{Beta}(1,1)$  lead to the best performance.
Figure 8: Influence of variants  $\lambda$ .
D.3 Comparison between different modules
In our framework, the uniform sampling module is only enabled in the training phase. While in the inference phase, we use the balanced re-sampling module to predict unseen instances. To verify the superiority of the re-sampling module, we compare the performance of these two modules as well as their ensemble. We report the results in Table 7. The results show that the re-sampling module is superior to the uniform module, and even achieves higher accuracy than the ensembled results. This also indicates the superiority of the proposed context-shift re-sampling method.
Moreover, we study the influence of different balance ratios on our re-sampling module and compare it with the vanilla re-sampling method. We report the results in Figure 9. For vanilla re-sampling, adopting a more balanced re-sampling would yield more severe performance degradation. In contrast, our method achieves higher performance through class-balanced resampling.
Table 7: Comparison between the uniform module and the re-sampling module in context-shift augmentation.
All	Many	Med.	Few
Uniform module	39.4	68.3	37.3	6.1
Re-sampling module	45.8	64.3	49.7	18.2
Ensemble results	43.0	67.5	44.1	11.4
Figure 9: Comparison of re-sampling and our method under different balance ratios  $\gamma$ .
D.4 Comparison of the learned representation
We visualize the representation of CIFAR100-LT in Figure 10. Each color represents a class, with darker colors representing head classes, and lighter colors representing tail classes. Although the fine-grained colors make it difficult to distinguish some classes, it can still be seen that CB-RS learns worse representation compared with vanilla CE. Moreover, our proposed method can learn a more distinguishable representation. Moreover, we visualize the Grad-CAM for examples with our proposed context-shift augmentation. We choose the same examples in Figure 3 in the main paper and report the results in Figure 11. The results show that our method can alleviate the negative impact on head-class samples caused by the overfitting problem.
Figure 10: Visualization of learned representation on CIFAR100-LT.
Figure 11: Visualization of features with Grad-CAM on CIFAR100-LT. Our method can alleviate the negative impact on head-class samples caused by the overfitting problem.
D.5 The influence of Grad-CAM
The Grad-CAM [24] is utilized to extract unrelated contexts in previous works such as open-set learning and adversarial learning [25, 26]. Also, we use Grad-CAM in the context-shift augmentation to generate diverse contexts for tail-class data. We compare Grad-CAM with CAM [17]. The results shown in Tab. 8 demonstrate that Grad-CAM is superior to CAM when applied to our method. One may be concerned with the generated activation map of tail-class images. We visualize some tail-class samples with predicted probabilities higher than  $\delta$  in Figure 12, which shows that the model can still grasp accurate activation maps for tail-class samples.
Table 8: Comparison between CAM and Grad-CAM in context-shift augmentation
All	Many	Med.	Few
Ours w/ CAM	45.1	63.8	48.1	18.0
Ours w/ Grad-CAM	45.8	64.3	49.7	18.2
class:train p=94.94%
class:trout p=96.01%
class:tulip  $p = 89.41\%$
class: wardrobe p=99.82%
class:whale p=98.22%
class:wolf p=92.88%
Figure 12: Visualization of features with Grad-CAM. For tail-class samples with predicted probabilities higher than the threshold (0.8), the model can also grap accurate activation maps.
D.6 Combination with self-supervised learning
It is interesting to combine self-supervised methods with context-shift augmentation. Inspired by this, we follow the self-supervised + fine-tune method, i.e., SimSiam+rwSAM in [44] and conduct extensive experiments on the CIFAR10-LT dataset. The results are shown in Tab. 9.
Table 9: Combing self-supervised method SimSiam+rwSAM with different re-balancing methods
Pre-train model	Fine-tune method	Accuracy (%)
SimSiam+rwSAM	CE	69.5
SimSiam+rwSAM	CB-RS	72.8
SimSiam+rwSAM	Ours	75.5
Note that in [44], the models are pre-trained with the long-tailed dataset, while fine-tuned with the balanced in-domain dataset. However, it is hard to achieve a balanced in-domain version of a long-tailed dataset in real-world scenarios. So we use the long-tailed dataset with a balancing method including class-balanced re-sampling (CB-RS), and re-sampling with context-shift augmentation. The results show that our method is superior to CB-RS.
D.7 Combination with the logit adjustment
Since our work aims to study the effectiveness of re-sampling in long-tail learning, we use the Class-Balanced Re-Sampling (CB-RS) in our method. We consider combining our method with other re-balancing methods, such as Logit Adjustment (LA) [16]. Specifically, we change the classbalanced re-sampling to the uniform sampling while adopting context-shift augmentation. Moreover, we consider combining class-balanced re-sampling and LA simultaneously. The comparison results are shown in Tab. 10. The results show that our method can be combined with logit adjustment to yield a higher performance. However, by applying the balanced loss and the balanced sampling, the model puts much focus on tail classes and results in a deterioration of overall accuracy.
Table 10: Combination with Logit Adjustment (LA). Accuracy results  $(\%)$  on CIFAR100-LT with an imbalance ratio of 100 are reported.
All	Many	Med.	Few
w/ CB-RS	45.8	64.3	49.7	18.2
w/ LA	47.2	62.4	48.8	26.1
w/ both	45.0	48.7	51.0	31.4