Title: How Re-sampling Helps for Long-Tail Learning?
Venue: NeurIPS 2023
Accepted: yes
Keywords: long-tail learning

class-imbalanced learning

re-sampling
Abstract: Long-tail learning has received significant attention in recent years due to the challenge it poses with extremely imbalanced datasets. In these datasets, only a few classes (known as the head classes) have an adequate number of training samples, while the rest of the classes (known as the tail classes) are infrequent in the training data. Re-sampling is a classical and widely used approach for addressing class imbalance issues. Unfortunately, recent studies claim that re-sampling brings negligible performance improvements in modern long-tail learning tasks. This paper aims to investigate this phenomenon systematically. Our research shows that re-sampling can considerably improve generalization when the training images do not contain semantically irrelevant contexts. In other scenarios, however, it can learn unexpected spurious correlations between irrelevant contexts and target labels. We design experiments on two homogeneous datasets, one containing irrelevant context and the other not, to confirm our findings. To prevent the learning of spurious correlations, we propose a new context shift augmentation module that generates diverse training images for the tail class by maintaining a context bank extracted from the head-class images. Experiments demonstrate that our proposed module can boost the generalization and outperform other approaches, including class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods. The source code is available at https://www.lamda.nju.edu.cn/code_CSA.ashx.

How Re-sampling Helps for Long-Tail Learning?

Abstract

Long-tail learning has received significant attention in recent years due to the challenge it poses with extremely imbalanced datasets. In these datasets, only a few classes (known as the head classes ) have an adequate number of training samples, while the rest of the classes (known as the tail classes ) are infrequent in the training data. Re-sampling is a classical and widely used approach for addressing class imbalance issues. Unfortunately, recent studies claim that re-sampling brings negligible performance improvements in modern long-tail learning tasks. This paper aims to investigate this phenomenon systematically. Our research shows that re-sampling can considerably improve generalization when the training images do not contain semantically irrelevant contexts. In other scenarios, however, it can learn unexpected spurious correlations between irrelevant contexts and target labels. We design experiments on two homogeneous datasets, one containing irrelevant context and the other not, to confirm our findings. To prevent the learning of spurious correlations, we propose a new context shift augmentation module that generates diverse training images for the tail class by maintaining a context bank extracted from the head-class images. Experiments demonstrate that our proposed module can boost the generalization and outperform other approaches, including class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods. The source code is available at https://www.lamda.nju.edu.cn/ code_CSA.ashx .

1 Introduction

Deep neural networks have achieved great success by applying well-designed models on large-scale elaborated datasets [1, 2, 3]. However, real-world data often exhibits a long-tail class distribution [4, 5, 6]. Learning from long-tail datasets has two main challenges, one is the class-imbalanced problem which causes the model biased towards the dominated head classes, and another is the data scarcity problem leading to the poor generalization on those rare tail classes [7, 8, 9, 10].

One simple and intuitive approach to deal with the class-imbalanced problem is re-sampling [11, 12], i.e., create the replicate of the dataset to estimate the model parameters. Unfortunately, it has been reported that re-sampling methods achieve limited effects when applied to most long-tail datasets [6, 13]. Currently, there are still few concrete and comprehensive explanations for this observation. Existing works mainly conclude that re-sampling will lead to the overfitting problem, and thus will be harmful to long-tail representation learning [6, 14].

Recently, many two-stage approaches have been proposed to improve the tail-class performance by adopting re-sampling in the second training stage. For instance, DRS [15] adopts a re-sampling

schedule at the last several episodes of the training process. cRT [6] first trains a preliminary model using the uniform sampler, then fixes the representations and re-trains the linear classifier using a class-balanced sampler. Last but not least, BBN [14] adjusts the whole model to first learn from the conventional learning branch and dynamically move to the re-balancing branch. Overall, the two-stage method has attracted widespread attention due to its basic hypothesis that uniform sampling is beneficial to representation learning, and the class-balanced sampling can be used to fine-tune the linear classifier. In light of the success of the two-stage method, a natural question is:

Can re-sampling benefit long-tail learning in the single-stage framework?

To answer this question, this paper empirically studies the re-sampling strategies and finds that re-sampling leads to opposite effects on long-tail datasets. Figure 1 gives a brief view of this phenomenon. Moreover, we deduce that if the training samples are highly semantically related to their target labels, class-balanced re-sampling can learn discriminative feature representations; otherwise, uniform sampling is even better than class-balanced re-sampling, as the latter suffers from oversampling redundant unrelated contexts. To verify this, we design a pair of synthetic benchmarks with the same content but different contexts, one containing irrele-





















 





























vant context in training samples and the other not. Experiments confirm that re-sampling achieves conspicuous different performances on these two benchmarks. In particular, when irrelevant context exists, class-balanced re-sampling learns poorer representations compared to uniform sampling, thus the irrelevant context negatively affects re-sampling methods.

We believe that re-sampling can benefit long-tail learning in the single-stage framework. It fails on some long-tail datasets mainly because it overfits the oversampled irrelevant contexts and learns unexpected spurious correlations. If such spurious correlations are avoided, re-sampling can be helpful for long-tail learning. Motivated by this, we propose a new context-shift augmentation module, which transfers well-separated context from head-class data to tail-class data. Specifically, it extracts the unrelated contexts (e.g. the backgrounds or the unrelated foreground objects) from head-class images and pastes them onto tail-class images to generate diverse novel samples. In this way, it encourages the model to learn more discriminative results for the tail classes. We conduct experiments on three long-tail datasets, CIFAR10-LT, CIFAR100-LT, and ImageNet-LT. The results show that the proposed module achieves competitive performance compared to the baseline methods. In summary, our main contributions are:

We conduct empirical analyses on different datasets and discover that re-sampling does not necessarily work or fail in long-tail learning.

We deduce that the failure of re-sampling may be attributed to overfitting on irrelevant contexts, and our empirical studies confirm our hypothesis.

We propose a new context-shift augmentation module to prevent re-sampling from overfitting to irrelevant contexts in a single-stage framework.

Extensive experiments verify the effectiveness of the proposed module against class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods.

The rest of the paper is organized as follows. Section 2 studies the effects of re-sampling approaches. Section 3 presents the proposed context-shift augmentation module. Section 4 briefly reviews related works. Section 5 concludes the paper.

2 A Closer Look at Re-sampling

2.1 Preliminaries

Given a training dataset D = { x i , y i } N i =1 , where x i is a training sample and y i ∈ C = [ K ] = { 1 , . . . , K } is the class label assigned to it. We assume that the training data follow a long-tail class

tail-classes

Training Set

Test Set

Training Set

Test Set

tail-classes

(a) Uniform sampling.

(b) Class-balanced re-sampling.

distribution where the class prior distribution P ( y ) is highly skewed so that many underrepresented classes have a very low probability of occurrence. Specifically, we define the imbalance ratio as ρ = max y P ( y ) / min y P ( y ) to indicate the skewness of data. Classes with high P ( y ) are referred to as head classes , while others are referred to as tail classes .

In practice, since the data distribution is unknown, Empirical Risk Minimization (ERM) uses the training data to achieve an empirical estimate of the underlying data distribution. Typically, one minimizes the softmax cross-entropy as following

where f y ( x ) denotes the predictive logit of model f on class y . However, this ubiquitous approach neglects the issue of class imbalance and makes the model biased toward head classes [8, 16].

To deal with the class-imbalance problem, the re-sampling strategy assigns a probability of being selected for each training sample according to its class frequency [6]. The probability of sampling a data point from class k can be written as:

where n k denotes the frequency of class k and q ∈ [0 , 1] . When q = 1 , Equation (2) denotes uniform sampling, where each training sample has an equal probability of being selected. When q = 0 , Equation (2) denotes the class-balanced re-sampling, which selects samples from every class k with the identical probability of 1 /K .

2.2 Exploring the Effect of Re-sampling

2.2.1 Re-sampling can learn discriminative representations

To better explore the effect of the re-sampling strategy, we conduct experiments on multiple long-tail datasets, including MNIST-LT, Fashion-LT, CIFAR100-LT [15], and ImageNet-LT [5]. We compare three different learning methods: 1) Cross-Entropy (CE) with uniform sampling; 2) Classifier Re-Training (cRT) which uses uniform sampling to learn the representation and class-balanced resampling to fine-tune the classifier; 3) Class-Balanced Re-Sampling (CB-RS) for the whole training process. We report the experimental results in Table 1.

Input

Uniform

CB-RS

Head

Classes

Tail

Classes

bear

fox

train

bicycle

bus

Input

Uniform

CB-RS















































































(a) Comparison of class accuracy.























(b) Overall accuracy.

According to the results, cRT performs best on CIFAR100-LT and ImageNet-LT, which is consistent with previous works [14, 6]. CE and cRT use the same representation but cRT achieves higher performances, which indicates that re-sampling can help for classifier learning. However, on MNISTLT and Fashion-LT, CB-RS surprisingly achieves the highest performance and outperforms CE and cRT by a large margin. Since cRT and CB-RS both use class-balanced re-sampling for classifier learning, the results indicate that CB-RS learns better representations than uniform sampling on MNIST-LT and Fashion-LT.

To further understand the effect of re-sampling, we visualize the learned representation on MNIST-LT in Figure 2. The figures show that with uniform sampling, the representation space is dominated by head classes, and the representations of tail classes are hard to distinguish. By applying class-balanced re-sampling, the representations of both head and tail classes are discriminative.

2.2.2 Re-sampling is sensitive to irrelevant contexts

We have demonstrated the generalization ability of the re-sampling strategy on MNIST-LT and Fashion-LT. Nevertheless, re-sampling performs unsatisfactorily on the other two datasets. Since the training samples and target labels on MNIST and Fashion are highly semantically correlated [18, 19], while samples on CIFAR and ImageNet contain complex contexts [20, 3], we hypothesize that re-sampling is sensitive to the contexts in training samples.

To support our hypothesis, we visualize the Grad-CAM of the representation learned by different sampling strategies in Figure 3. When training with a uniform sampler, models can distinguish the contexts from samples of head classes. However, when adopting a class-balanced re-sampler, the model tends to overfit the irrelevant context from the over-sampled tail data, which unexpectedly affects the representation of head classes. For example, when classifying different kinds of animals, re-sampling might focus on the posture rather than the appearance. Also, when classifying different vehicles, re-sampling is easily influenced by the human in tail-class images, and thus mistakenly focuses on the human in head-class images.

To further validate our point, we design a homogeneous benchmark of MNIST-LT termed ColoredMNIST-LT (CMNIST-LT). We inject colors into MNIST-LT to artificially construct irrelevant contexts.

Specifically, we design CMNIST-LT based on two considerations. First, head classes are prone to have rich contexts, so we inject different colors into the samples of each head class. Second, tail classes have limited contexts, so we inject an identical color into the samples of each tail class. We conduct uniform sampling, classifier re-training, and class-balanced re-sampling on MNIST-LT and CMNIST-LT. The experimental results are illustrated in Figure 4. The results show that when applied to MNIST-LT, CB-RS can boost the tail-class performance without degradation on head classes. However, for CMNIST-LT, CB-RS performs worse than uniform sampling and cRT on both head and tail classes, thus validating the negative impact of irrelevant contexts on re-sampling methods. Since re-sampling succeeds on MNIST-LT and fails on CMNIST-LT, we propose that re-sampling does not always fail, it can help for long-tail learning if avoiding the irrelevant contexts.

2.2.3 Proposed benchmark datasets

We follow the previous works [15, 14] to construct MNIST-LT, Fashion-LT and CIFAR100-LT and set the imbalance ratio to 100. ImageNet-LT is proposed by [5]. For MNIST-LT and Fashion-LT, we use LeNet [21] as the backbone network and add a linear embedding layer before the fully connected layer to project the representation into 2-dimensional space for better presentation. We use standard SGD with a mini-batch size of 128, an initial learning rate of 0.1 and a cosine annealing schedule to train the model for 8 epochs. When applying cRT, we retrain the last fully connected layer for 4 epochs by fixing the other layers. For CIFAR100-LT and ImageNet-LT, more details are in Section 3.3.

To construct CMNIST-LT, we follow the idea of CMNIST [22] to first randomly flip the label and then inject colors into the training samples. However, different from CMNIST which converts the MNIST to a binary classification dataset, we keep the ten classes to better simulate a long-tail class distribution. To generate flipped labels on the long-tail dataset MNIST-LT, we follow the method in [23] and set the flipping probability to 1 / 4 . We generate ten different colors using the seaborn 2 package. For the five head classes, we randomly inject one of these ten colors into each sample with equal probability. For the other five tail classes, we inject samples of each class with a single color.

3 A Simple Approach to Make Re-sampling Robust to Context-shift

3.1 Extracting Rich Contexts from Head-class Data

By studying the effects of re-sampling methods in different scenarios, we can draw a conclusion: when the training samples contain irrelevant contexts, simply over-sampling the tail-class samples might cause the model to unexpectedly focus on these redundant contexts, thereby resulting in the overfitting problem. However, the head classes have rich data to learn a model with good generalization ability. We naturally raise a question: can we utilize the rich contexts from head data to augment the over-sampled tail data to alleviate the negative impact of irrelevant contexts? Inspired by this motivation, we design a context-shift augmentation module by extracting the rich contexts from head-class data to enrich the over-sampled tail-class data.

To leverage the rich contexts in head-class data, we utilize a model f u trained with uniform sampling for context extraction. First, we select well-learned samples with fitting probability larger than a threshold δ to improve the extraction quality. The fitting probability for sample x i can be calculated by the Softmax function as follows

where z u i denotes the logits predicted by f u , i.e., z u i = [ z u i, 1 , . . . , z u i,K ] = f u ( x i ) . Then, we use off-the-shelf methods such as Grad-CAM [24, 17] to extract the image contexts, which is also used in previous works regarding open-set learning and adversarial learning [25, 26]. Specifically, given an image x i , we calculate its class activation map CAM ( x i | f u ) . Then, we inverse the map to get the background mask M i , i.e., M i = 1 -CAM ( x i | f u ) . Here M i is a matrix of the same size as x i , with values between 0 and 1. A higher value indicates that the corresponding pixel is more likely to be the background. Different from previous works that discretize the mask matrix to binary values [27, 28], we keep the floating values in the matrix to conserve more information.

After calculating the background mask M i of image x i , we paste the mask onto the original image by M i ⊙ x i to obtain a background image. In this way, we separate the semantically related contents from the images and keep the rest contexts for further augmentation. Finally, the extracted contexts are pushed into a memory bank Q for augmentation of re-sampled data.

For the training of the uniform module, we apply the conventional ERM algorithm. For each training sample x i , we calculate its loss by

where ℓ u can be any loss function. Generally, we use the standard cross-entropy loss.

3.2 Balanced Re-sampling with Context-shift Augmentation

Simply adopting balanced sampling might generate many repeated samples from the tail classes and lead to the overfitting problem. Therefore, we ask for background images from the context memory bank Q and paste them onto the re-sampled images. In this way, we generate more diverse novel samples by simulating each tail-class image within various contexts. Specifically, for a re-sampled training image ˜ x i , we ask for another image ˇ ˇ x i together with its mask M i from Q , and fuse it with ˜ x i to generate a novel sample, and calculate its training loss as follow:

Here λ is randomly generated between [0 , 1] to increase the diversity. Different from previous mixupbased methods[29, 30], our method does not change the target label, because the pasted background is not related to the semantics of any class labels.

To reduce the computational complexity, the uniform module and the balanced re-sampling module can be trained simultaneously by sharing the same feature extractor ϕ , and training their own linear classifiers ψ u and ψ b , i.e., f u ( · ) = ψ u ( ϕ ( · )) and f b ( · ) = ψ b ( ϕ ( · )) . Since the classifier is lightweight, it does not add much additional computational overhead. Moreover, the memory bank Q is designed as a first-in-first-out queue with a maximum volume of V for a convenient query. After extracting the context from the uniform module, we append the x i and M i pair into the context bank Q . When the size of Q reaches its maximized volume, the oldest contexts are pushed out. In practice, the volume size is set equal to the mini-batch size in order to minimize the overhead as well as ensure the querying requirements. Finally, the summarized loss function is

Figure 5 gives a brief overview of the proposed module. The detailed training procedure is given in the supplementary material due to the page limit.

In the inference phase, only the balanced re-sampling module is used. In other words, the uniform module only serves as an assistant to provide more rich contexts for the re-sampling module during the training phase. Formally, for a test data point x , we obtain the prediction by z = f b ( x ) , and then employ the Softmax function to obtain the predictive probabilities.

3.3 Empirical Results

We demonstrate the efficacy of the proposed module context shift augmentation by comparing it with different kinds of long-tail learning methods, including:

Re-sampling or re-weighting methods, such as Focal Loss [31], CB-Focal [7], CE-DRS [15], CE-DRW [15], LDAM-DRW [15], cRT [6], LWS [6], and BBN [14],

Head-to-tail knowledge transfer methods, such as M2m [32], OLTR [5], and FSA [28],

Long-tailed

Dataset

Diverse novel samples

①

…

Head classes

Tail classes

…

…

②

Re-sampling with Augmentation

Loss

𝐿 𝑢

Loss

𝐿 𝑏

Balanced sampler

…

Original image

Activation map

Grad-CAM

Masking

Background image

Context memory bank

…

Data augmentation methods, such as Mixup [29], Remix [33], CAM-BS [13], and CMO [27].

We conduct experiments on three long-tail datasets, including CIFAR10-LT [15], CIFAR100-LT [15], and ImageNet-LT [5]. CIFAR10-LT and CIFAR100-LT are the long-tail versions of CIFAR datasets by sampling from the raw dataset with an imbalance ratio ρ . Following previous works [15, 14], we conduct experiments with ρ ∈ { 100 , 50 , 10 } . ImageNet-LT is a long-tail version of the ImageNet [3], which contains 1000 classes, each with a number of samples ranging from 5 to 1280. For each long-tail training dataset, we evaluate the model on another corresponding class-balanced test set by calculating the overall prediction accuracy.

The results for CIFAR10-LT and CIFAR100-LT are summarized in Table 2. We report the results under imbalance ratio ρ = 100 , 50 , 10 . As shown in the table, our method achieves superior performance compared with the baseline methods in all settings.

Specifically, the methods based on re-sampling or re-weighting such as DRS, DRW, cRT, and LWS can ease the class-imbalanced problem to some degree but the performance gain is limited due to the neglect of the representation learning. The methods based on knowledge transfer and data augmentations, such as M2m and CMO, achieve higher performance.

Moreover, by combining cRT and LWS with mixup, the performance achieves an obvious improvement. However, such two-stage training methods are not end-to-end approaches. In contrast, the proposed context shift augmentation module enhances representation learning by enriching the contexts of samples and adopting the class-balanced re-sampling to ensure a balanced classifier. In this manner, it achieves more improvement with an end-to-end framework. Since cRT and the proposed module both use class-balanced sampling for classifier learning, the results indicate that the proposed context shift augmentation can achieve better representations.

We further conduct the experiments on a larger scale dataset ImageNet-LT. We calculate the accuracy of the overall test set and the average accuracy of the many-shot classes (more than 100 images in the training set), the medium-shot (20 ∼ 100 images), and the few-shot classes (less than 20 images). We report the results in Table 3. It shows that the proposed module is superior to most other baseline methods. The performance is similar to another data augmentation method CMO, but our method achieves higher accuracy on few-shot classes, which demonstrates the generalization ability of context-shift augmentation for tail-class data.

† denotes a longer training of 100 epochs.

We provide more detailed studies to analyze the effect of each component in the proposed module. Due to the page limit, we report the results in the supplementary material.

4 Related Work

4.1 Re-sampling and Re-weighting

Re-sampling is a widely used strategy in class-imbalanced learning [11, 12]. There are two main ideas of re-sampling: 1) Over-sampling by repeating data from the rare classes. 2) Under-sampling by abandoning a proportion of data from the frequent classes. However, when the class distribution is highly skewed, re-sampling methods often fail. Previous works point out that under-sampling may discard precious information which inevitably degrades the model performance, and over-sampling tends to cause the overfitting problem on the tail classes [14, 15].

Recent work [6] develops class-balanced re-sampling where samples from each class have an identical probability of being sampled. Class-balance re-sampling can bring performance gain for classifier learning but hurts representation learning. Therefore, two-stage approaches [15, 6, 14] adopt it at the late stage of the whole training process in order not to impact the representation.

Re-weighting aims to generate more balanced predictions by adjusting the losses for different classes [35, 36]. The most intuitive way is to weight each training sample by the inverse of its class frequency [4]. Similar to class-balanced re-sampling, re-weighting can achieve better results for tail classes but usually deteriorates its performance for head classes [7].

In contrast, our empirical study reveals that class-balanced re-sampling can be an effective method as long as there exist no irrelevant contexts. It fails in some cases mainly due to the unexpected overfitting

towards the over-sampled redundant contexts. When applied with context-shift augmentation, classbalanced re-sampling can achieve competitive performance on long-tail datasets.

4.2 Head-to-tail Knowledge Transfer

As the head classes have adequate training data while the tail classes have limited data, recent works aim to leverage the knowledge gained from head classes to enhance the generalization of tail classes. Feature transfer learning [37] utilizes the intra-class variance from head classes to guide the feature augmentation for tail classes. MetaModelNet [4] uses the head data to train a meta-network to predict many-shot model parameters from few-shot model parameters, then transfers the metaknowledge to the tail classes. Major-to-minor translation (M2m) [32] uses the over-sampling method and translates the head-class samples to replace the duplicated tail-class samples via adversarial perturbations. OLTR [5] maintains a dynamic meta-embedding between head and tail classes to transfer the semantic deep features from head to tail classes.

These methods assume that the head classes and the tail classes share some common knowledge such as the same intra-class variances, the same model parameters, or the same semantic features. In this work, we regard the contexts as such knowledge and transfer the contexts from head-class data to enrich the tail-class data.

4.3 Data Augmentation

Several data augmentation approaches have been proposed to improve the model generalization ability. In contrastive learning [38, 39], curriculum learning [40], meta-learning methods [41] and instance segmentation tasks [42], data augmentation strategies have been shown to effectively improve the generalization of tail classes. MiSLAS [34] studies the mixup [29] technology in long-tail learning and finds that mixup can have a positive effect on representation learning but a negative or negligible effect on classifier learning. Remix [33] adapts the mixup method to a re-balanced version. It assigns the mixed label in favor of the tail class by designing a disproportionately higher weight for the tail class. CMO [27] applies CutMix [30] by cutting out random regions of a sample from head classes and filling the removed regions with another sample from tail classes. By this means, it enriches the contexts of the tail data; but the random cutout operation does not necessarily separate the contents and contexts.

The attention or CAM-based methods have been proposed to improve long-tail learning via feature decomposition and augmentation. CAM-BS [13] separates the foreground and background of each sample, then augments the foreground part by flipping, translating, rotating, or scaling. Feature Space Augmentation (FSA) [28] uses CAM to decompose the features of each class into a class-generic component and a class-specific component, and generates novel samples in the feature space by combining the class-specific components from the tail classes and the class-generic components from the head classes. Attentive Feature Augmentation (AFA) [43] adopts feature decomposition and augmentation via the attention mechanism. Note that FSA and AFA can also be seen as head-to-tail knowledge transfer approaches. Nevertheless, these methods neglect that the learned model has limited generalization ability on tail classes, and most foregrounds (or class-specific components) of samples from tail classes are incredible. In comparison, our method applies CAM to separate related contents and unrelated contexts for samples mainly from head classes. It then pastes the contexts extracted from the head-class data onto the over-sampled tail-class data to enrich the contexts.

5 Conclusion

In this work, we study the re-sampling strategy for long-tail learning. Our empirical investigations reveal that the impact of re-sampling is highly dependent on the existence of irrelevant contexts and is not always harmful to long-tail learning. To reduce the influence of irrelevant contexts, we propose a new context-shift augmentation module that leverages the well-separated contexts from the head-class images to augment the over-sampled tail-class images. We demonstrate the superiority of the proposed module by conducting experiments on several long-tail datasets and comparing it against class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods.

Broader Impact and Limitations

This paper investigates the reasons behind the success/failure of re-sampling approaches in long-tail learning. In critical and high-stakes applications, such as medical image diagnosis and autonomous driving, the presence of imbalanced data poses the risk of producing biased predictions. By shedding light on this problem, we aim to inspire more research on safe and robust re-sampling approaches.

One may be concerned about combining the proposed module with other methods such as selfsupervised learning [44], logit adjustment [16]. We conduct additional experiments and report the results in the supplementary material due to the page limit. Nevertheless, the proposed module can not achieve comparable performance with the well-designed models [45, 46, 47], since our intention is not to achieve performance that is on par with state-of-the-art methods. Instead, we hope that our findings will inspire future research regarding re-sampling methods.

Data Availability Statement

The source code of our method is available at https://www.lamda.nju.edu.cn/code_CSA.ashx or https://github.com/shijxcs/CSA .

Acknowledgments and Disclosure of Funding

This research was supported by the National Key R&D Program of China (2022ZD0114803), the National Science Foundation of China (62176118, 61921006).

References

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems , volume 30, pages 7029-7039, 2017.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Largescale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2537-2546, 2019.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2020.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9268-9277, 2019.

Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems , volume 33, pages 4175-4186, 2020.

Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9495-9504, 2021.

Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems , volume 33, pages 19290-19301, 2020.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research , 16:321357, 2002.

Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 39(2):539550, 2008.

Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tailed visual recognition with deep convolutional neural networks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence , pages 3447-3455, 2021.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9719-9728, 2020.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems , volume 32, pages 1565-1576, 2019.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations , 2021.

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision , pages 618-626, 2017.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

AKrizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront , 2009.

Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems , 2, 1989.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. arXiv preprint arXiv:2108.11569 , 2021.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2921-2929, 2016.

Jie-Jing Shao, Xiao-Wen Yang, and Lan-Zhe Guo. Open-set learning under covariate shift. Machine Learning , pages 1-17, 2022.

Qibing Ren, Yiting Chen, Yichuan Mo, Qitian Wu, and Junchi Yan. Dice: Domain-attack invariant causal learning for improved data privacy protection and adversarial robustness. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , page 1483-1492, 2022.

Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6887-6896, 2022.

Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for longtailed data. In Proceedings of the 16th European Conference on Computer Vision , pages 694-710, 2020.

Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6023-6032, 2019.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision , pages 2980-2988, 2017.

Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via majorto-minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13896-13905, 2020.

Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. In Proceedings of the 16th European Conference on Computer Vision , pages 95-110, 2020.

Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16489-16498, 2021.

Charles Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence , pages 973-978, 2001.

Zhi-Hua Zhou and Xu-Ying Liu. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Transactions on Knowledge and Data Engineering , 18(1):63-77, 2006.

Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5704-5713, 2019.

Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In Proceedings of the 39th International Conference on Machine Learning , pages 27367-27377, 2022.

Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6908-6917, 2022.

Sumyeong Ahn, Jongwoo Ko, and Se-Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. In International Conference on Learning Representations , 2023.

Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5212-5221, 2021.

Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3457-3466, 2021.

Weiqiu Wang, Zhicheng Zhao, Pingyu Wang, Fei Su, and Hongying Meng. Attentive feature augmentation for long-tailed visual recognition. IEEE Transactions on Circuits and Systems for Video Technology , 32(9):5803-5816, 2022.

Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations , 2022.

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by routing diverse distribution-aware experts. In International Conference on Learning Representations , 2021.

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6626-6636, 2021.

Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 715-724, 2021.

Zhuoxun He, Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data. arXiv preprint arXiv:1909.09148 , 2019.

A Training Procedure

The training procedure of context-shift augmentation is summarized in Algorithm 1.

Algorithm 1 Training procedure of context-shift augmentation Input: training data D = { ( x i , y i ) } N i =1 ; context memory bank Q , maximum volume size V ; model parameters ϕ , f u , f b ; loss functions ℓ u , ℓ b ; Procedure:

B Implementation Details for context-shift augmentation

For experiments on CIFAR10-LT and CIFAR100-LT, we use ResNet-32 as the backbone network and train it using standard SGD with a momentum of 0.9, a weight decay of 2 × 10 -4 , a batch size of 128. The model is trained for 200 epochs. The initial learning rate is set to 0.2 and is annealed by a factor of 10 at 160 and 180 epochs. We train each model with 1 NVIDIA GeForce RTX 3090.

For experiments on ImageNet-LT, we implement the proposed method on ResNet-10 and ResNet-50. We use standard SGD with a momentum of 0.9, a weight decay of 5 × 10 -4 , and a batch size of 256 to train the whole model for a total of 90 epochs. We use the cosine learning rate decay with an initial learning rate of 0.2. We train each model with 2 NVIDIA Tesla V100 GPUs.

In all experiments, we first warm up the uniform module for 10 epochs and then train the uniform module and the balanced re-sampling module simultaneously for the rest epochs. For the uniform module, we follow the simple data augmentation used in [2] with only random crop and horizontal

flips. For the re-sampling module, we use the proposed context-shift augmentation method. We apply the trick proposed by [48] to disable the augmentation in the balanced re-sampling module at the last 3 epochs to obtain further improvements, which is also applied in other baseline methods [13, 27]. We set the threshold δ to 0.8.

C Additional Illustrations

For convenience in understanding context and content, we give an example in Figure 6. Context refers to the semantically unrelated parts in the images, and content refers to the semantically related parts. Moreover, We give a brief illustration of the generated dataset CMNIST-LT in Figure 7.

content

(bicycle)

context

Head Classes

Tail Classes

5k

#Examples

2k

1k

0

0

123456７

8

9

ClassIndex

D Additional Experimental Results

D.1 Effects of the context bank

The context bank Q is a novel component of context-shift augmentation which receives diverse contexts from the uniform module and provides them to augment the data in the re-sampling module. To verify the effectiveness of the context bank, we remove it from the framework and train the model on CIFAR100-LT with an imbalance ratio of 100. The results are reported in Table 4.

The results show that without context bank Q , the performance decreases by a large margin. The performance degradation mostly comes from the medium-shot classes and the few-shot classes, which indicates that the context bank can significantly improve the generalization of tail classes.

Moreover, we study the effect of different variants in the context bank. First, we study the threshold δ for sample selection and report the results in Table 5. On the one hand, if δ is too high, the selected samples will be very few. On the other hand, if δ is too small, the selected samples might be not well learned. Nevertheless, as the training process progresses, most samples will fit well, so our method is not sensitive to δ . We set δ = 0 . 8 considering its best performance.

Second, we study the influence of the volume size V of the context bank Q and report the results in Table 6. Since the bank Q is a first-in-first-out queue, the latest incoming contexts are more convincing. When the volume size is too large, the bank might contain more past samples. Besides, a larger size of V would bring more memory overhead. So we set the volume size V equal to the mini-batch size B in our method.

D.2 Influence of augmentation variants

We use a variant λ ∼ Uniform(0 , 1) for generating novel samples. The value of λ can result in different proportions of foreground and background in the novel sample. Also, the size of the sampling space affects the diversity of the novel images. To explore the effect of λ , we try λ ∼ Uniform( a, b ) and λ ∼ Beta( a, b ) to train context-shift augmentation on CIFAR100-LT with imbalance ratio 100 and report the results in Figure 8.

First, when λ is close to 0, the background merely takes effect, and the performance decreases a lot. Second, when λ = 1 , the background image might cover the important content in the foreground image. Also, the diversity of new samples is limited. Although the performance is better than that of λ = 0 , it is still unsatisfactory. Overall, choosing λ ∼ Uniform(0 , 1) or λ ∼ Beta(1 , 1) lead to the best performance.































































λ

∼

Uniform(

a, b

)

















































































λ

∼

Beta(

a, b

)

D.3 Comparison between different modules

In our framework, the uniform sampling module is only enabled in the training phase. While in the inference phase, we use the balanced re-sampling module to predict unseen instances. To verify the superiority of the re-sampling module, we compare the performance of these two modules as well as their ensemble. We report the results in Table 7. The results show that the re-sampling module is superior to the uniform module, and even achieves higher accuracy than the ensembled results. This also indicates the superiority of the proposed context-shift re-sampling method.

Moreover, we study the influence of different balance ratios on our re-sampling module and compare it with the vanilla re-sampling method. We report the results in Figure 9. For vanilla re-sampling, adopting a more balanced re-sampling would yield more severe performance degradation. In contrast, our method achieves higher performance through class-balanced resampling.





















































D.4 Comparison of the learned representation

We visualize the representation of CIFAR100-LT in Figure 10. Each color represents a class, with darker colors representing head classes, and lighter colors representing tail classes. Although the fine-grained colors make it difficult to distinguish some classes, it can still be seen that CB-RS learns worse representation compared with vanilla CE. Moreover, our proposed method can learn a more distinguishable representation. Moreover, we visualize the Grad-CAM for examples with our proposed context-shift augmentation. We choose the same examples in Figure 3 in the main paper and report the results in Figure 11. The results show that our method can alleviate the negative impact on head-class samples caused by the overfitting problem.

CE

CB-RS

Ours

Input

Uniform

CB-RS

Head

Classes

Tail

Classes

bear

fox

train

bicycle

bus

Input

Uniform

CB-RS

Ours

Ours

D.5 The influence of Grad-CAM

The Grad-CAM [24] is utilized to extract unrelated contexts in previous works such as open-set learning and adversarial learning [25, 26]. Also, we use Grad-CAM in the context-shift augmentation to generate diverse contexts for tail-class data. We compare Grad-CAM with CAM [17]. The results shown in Tab. 8 demonstrate that Grad-CAM is superior to CAM when applied to our method. One may be concerned with the generated activation map of tail-class images. We visualize some tail-class samples with predicted probabilities higher than δ in Figure 12, which shows that the model can still grap accurate activation maps for tail-class samples.

























D.6 Combination with self-supervised learning

It is interesting to combine self-supervised methods with context-shift augmentation . Inspired by this, we follow the self-supervised + fine-tune method, i.e., SimSiam+rwSAM in [44] and conduct extensive experiments on the CIFAR10-LT dataset. The results are shown in Tab. 9.

Note that in [44], the models are pre-trained with the long-tailed dataset, while fine-tuned with the balanced in-domain dataset. However, it is hard to achieve a balanced in-domain version of a long-tailed dataset in real-world scenarios. So we use the long-tailed dataset with a balancing method including class-balanced re-sampling (CB-RS), and re-sampling with context-shift augmentation . The results show that our method is superior to CB-RS.

D.7 Combination with the logit adjustment

Since our work aims to study the effectiveness of re-sampling in long-tail learning, we use the Class-Balanced Re-Sampling (CB-RS) in our method. We consider combining our method with other re-balancing methods, such as Logit Adjustment (LA) [16]. Specifically, we change the classbalanced re-sampling to the uniform sampling while adopting context-shift augmentation . Moreover, we consider combining class-balanced re-sampling and LA simultaneously. The comparison results are shown in Tab. 10. The results show that our method can be combined with logit adjustment to yield a higher performance. However, by applying the balanced loss and the balanced sampling, the model puts much focus on tail classes and results in a deterioration of overall accuracy.

D.8 Combination with supervised contrastive learning

Our proposed context-shift augmentation (abbreviated as CSA) can be integrated with the supervised contrastive learning method BCL [39] to further improve the generalization. In Table 11, we report the experimental results. We conduct experiments on CIFAR100-LT with varying imbalance ratios, showing that CSA consistently boosts the performance of BCL.

D.9 Computational cost analysis of context-shift augmentation .

The proposed context-shift augmentation is based on the widely-used dual-branch network. Moreover, a context-extracting module is designed to calculate Grad-CAM as well as obtain the contexts. The context-extracting module has a very small computational cost, as it only needs to apply the gradient backward once at the last layer of the network, and can be ignored compared to the global gradient backward for updating the whole model. Besides, it does not spend too much memory space to save the extracted contexts, since the size of the context bank is set equal to the mini-batch size. In Table 12, we report the training time cost per epoch of CIFAR100-LT using a single RTX3090, which also demonstrates that the proposed method does not lead to too much additional computational overhead.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest looking closer at the resampling strategy for long-tail recognition. The motivation comes from the observation that in recent works resampling brings rather negative than positive effects. Based on MNIST experiments, the authors demonstrate that context (background) imposes spurious correlations during training that prevents generalization on tail classes. To deal with such correlations, the authors suggest a context shift augmentation module to generate diverse training images for the tail classes. Experimental results show improvement over listed by the authors baselines.\n\nSTRENGTHS:\nS1: The authors suggest an interesting direction of analysis, namely why re-sampling fails to help generalization of the tail classes. The study is performed on the toy dataset with the aim to demonstrate that spurious correlations change drastically the predictions.\n\nS2: The overall structure and the storyline are nice; it is easy to follow the method section\n\nWEAKNESSES:\nW1: The synthetic experiment does not support the statements of the authors. The claim that spurious correlations hinder tail class accuracy is not presented in Fig.4 (a) on CMNIST-LT. Namely, starting from class 3, all the accuracies are jumping around independently of the number of samples in the class and the sampling strategy. “Overfit to the irrelevant context from the over-sampled tail data” is also not observed. Moreover, the generated dataset is not visualized. Further, the statement on L162-L165 sounds reasonable; however, I find the provided arguments not convincing and rather unrelated. \n\nW2: the authors claim to improve generalization on the tail classes; however, based on Table 3 the improvement comes mostly from “many” and “med” classes. \n\nW3: the tables overall are misleading. The authors simply do not include multiple baselines that outperform their method such as \n- Long-Tail Learning via Logit Adjustment  \n- Parametric Contrastive Learning   \n- Balanced Contrastive Learning for Long-Tailed Visual Recognition  \n\nW4: The authors briefly discuss differences to [28], but it looks like the novelty is rather limited wrt this method. \n\nW5: the presentation of the paper can be improved. Some sentences are hard to parse and connect to the previous context without reading further, e.g. L10, L44, L51-52. The caption of Fig.1 does not help to understand Fig.1, related text in the intro also is not helpful.\n\nQUESTIONS:\nQ1: Would it be possible to conduct a similar experiment on more complex data with a more in-depth analysis of the correlations? It would significantly help to improve the paper.\n\nQ2: Is there any adaptation/analysis of the spatial structure of the images, e.g. if the object is not in the center of the image or there are multiple objects of interest? \n\nQ3: in Eq. (7), the mask M relates to image x_double_hat but is unrelated to x_tilde. In this case not only background is affected by the augmentation. I didn’t find the discussion in this regard. \n\nQ4: Could the authors discuss more in detail the difference to [28] and provide quantitative confirmation of the improvements based purely on the proposed novelty?\n\nLIMITATIONS:\nThe authors mention that their method does not achieve comparable performance to papers presented at CVPR/ICCV21. No further discussion is added. \nOther limitation in regard medical images and autonomous driving is rather general than specific to the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors rethink the effect of re-sampling on long-tailed learning. Specifically, this paper leverages a synthetic experiment to claim that re-sampling is sensitive to irrelevant contexts. \nTo address this problem, this paper proposes a context-shift augmentation strategy to alleviate the overfitting to the irrelevant contexts in the tail classes. In the empirical part, this paper conducts performance comparisons on CIFAR-LT and ImageNet-LT to validate the proposed method.\n\nSTRENGTHS:\n- The paper is generally well-written and easy to follow.\n- The exploration on the intrinsic mechanism of re-sampling on long-tailed data is important and not addressed.\n- This paper claims that re-sampling is sensitive to the irrelevant context and presents a toy example to support the claim.\n\nWEAKNESSES:\n- The construction of CMNIST-LT seems an extreme case, which may not exhibit in real-world scenarios.\n- The proposed method still follows the multi-branch architecture. However, the computational cost of the proposed method is not discussed.\n- The experiments are not adequate. Some important long-tailed benchmarks or baselines are missing.\n- This paper lacks more ablation studies or further analysis to provide more insight of the proposed baseline.\n\nQUESTIONS:\n- The construction of CMNIST-LT seems unreasonable. The color injection introduces severe spurious correlations to the tail classes as each tail class is assigned with single color and each head class is assigned with multiple colors. It is not surprising that the class-balanced re-sampling will deteriorate the performance at this case as only the spurious feature (single color) is enough to perform the classification on the tail class. Furthermore, this hand-crafted distribution on the irrelevant contexts may not exhibit on CIFAR-10/100-LT, ImageNet-LT as the diversity of the tail-class irrelevant contexts is mainly proportional to their class cardinality, but rather than collapse to the single point in the distribution. \n- The author raises the question “Can re-sampling benefit long-tail learning in the single-stage framework” in line 41. However, the proposed method still requires multi-branch learning (even multi-stage learning) to perform the resampling. It is critical to compare the computational cost of the proposed method with baseline methods.\n- This paper mainly investigates the class-balanced re-sampling. Is this re-sampling strategy optimal at the case of irrelevant contexts? More discussions or empirical comparisons are recommended.\n- Intuitively, the Grad-CAM is not reliable at the early stages of the training. It seems multiple stages (warmup) are required to ensure the training stability. \n- The experiments are only conducted on CIFAR-LT and ImageNet-LT. It would be better to conduct experiments on more real-world long-tailed benchmark Places-LT and iNaturalist.\n- In the main text of this paper, only the empirical comparison results are presented. It would benefit a lot to provide more ablation studies or further analysis of the proposed method. \n- Some supervised long-tailed learning SOTA baselines are missing in this paper:\n\n[1] Menon, A. K., Jayasumana, S., Jain, H., Veit, A., Kumar, S., & Rawat, A. S. Long-tail learning via logit adjustment. ICLR 2020.\n\n[2] Cui, J., Zhong, Z., Liu, S., Yu, B., & Jia, J. (2021). Parametric contrastive learning. ICCV 2021.\n\n- Some related data augmentation baselines are missing in this paper:\n\n[3] Chu, P., Bian, X., Liu, S., & Ling, H. (2020). Feature space augmentation for long-tailed data. ECCV 2020.\n\n[4] Li, S., Gong, K., Liu, C. H., Wang, Y., Qiao, F., & Cheng, X. (2021). Metasaug: Meta semantic augmentation for long-tailed visual recognition. CVPR 2021.\n\n[5] Zang, Y., Huang, C., & Loy, C. C. (2021). Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. ICCV 2021.\n\n[6] Zhou, Z., Yao, J., Wang, Y. F., Han, B., & Zhang, Y. (2022). Contrastive learning with boosted memorization. ICML 2022.\n\n[7] Ahn, S., Ko, J., & Yun, S. Y. (2023). CUDA: Curriculum of Data Augmentation for Long-tailed Recognition. ICLR 2023.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article systematically investigates the effectiveness of resampling methods in long-tailed image classification problems. Resampling is a classical and important approach to addressing class imbalance, but it may not always be effective when applied to long-tailed datasets. The paper shed light on this phenomenon and provides a systematical analysis. It discovers that re-sampling the tail data can improve the performance, but may lead to overfitting on the irrelevant contexts. To address this issue, they propose a context-shift data augmentation method. The proposed method achieves superior performance on long-tailed datasets such as CIFAR10-LT, CIFAR100-LT, and ImageNet-LT.\n\nSTRENGTHS:\n1. This paper provides a well-defined and reasonable motivation. The authors show both success and failure cases of re-sampling on long-tailed datasets, which highlights the significance of addressing this problem.\n2. This paper extensively conducts experiments and ablation studies on widely-used datasets, which demonstrates the effectiveness of their method.\n3. This paper is well-writing and easy to follow. The source code is provided in a proper way, facilitating the reproducibility of their work.\n\nWEAKNESSES:\n1. A comparison with some data augmentation methods should be included. For example, Randaugment is a widely used method, which can improve the performance on long-tailed datasets.\n2. The name of the proposed method (context-shift augmentation) should be unified, as some parts include hyphens \"-\" while others do not. Also, some parts use italics while others do not.\n\nQUESTIONS:\n1. Does the data augmentation method of enhancing the background also applied to samples of the head class?\n2. Does the proposed context-shift augmentation require additional computational cost?\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLong-tailed Learning has drawn great attention in recent development of deep learning due to the non-negligible challenges on tail classes during learning. The re-sampling method that should make an effect to improve the training usually fails in a lot of practices. The authors dive into the phenomenon and finds some new insights towards this problem, and then design a new method to alleviate the negative impact of re-sampling on some datasets, which shows the final promise compared with the recent SOTAs.\n\nSTRENGTHS:\nThe strengths of this submission can be summarized as follows,\n\n1) The authors rethinks an interesting problem about the re-sampling method, which are under-estimated or mistaken in previous studies. That is, re-sampling mainly works in the classifier layer through a two-state decoupling way. How to make the re-sampling works in a single-stage framework is an unexplored yet important challenge.\n\n2) The observation and interpretation are insightful, as two different types of datasets contracts on the final performance for re-sampling. It means actually re-sampling can help to extract the discriminative representation but just sensitive to the irrelevant contexts, i.e., overfitting to the spurious correlations in some cases. \n\n3) Based on the conjecture, the authors design a context-shift augmentation method along with re-sampling to alleviate the negative impact. Specially, they construct a context memory bank to boost the augmentation of samples in diversity and avoid overfitting the spurious correlation from backgrounds. The experimental results demonstrate the promise of the proposed method.\n\n4) The whole paper is well written, in which the motivation parts are well justified with a range of experiments to show the intuition, making the whole parts clearly understood.\n\nWEAKNESSES:\nAlthough the submission are well organized and presented, there are still some points that can be improved by following the below advices.\n\n1) In Figure 2, although it is hard to visual the representation distribution on CIFAR100-LT and ImageNet-LT, it is still possible to randomly select some sub-classes, e.g., downsample 10 classes from the super classes of CIFAR100-LT, to visualize the representation. This can shows whether on some datasets that contains many class-irrelevant contexts, the representation are not well learnt, i.e., not so discriminative, compared with that on datasets that contains few class-irrelevant contexts.\n\n2) Similarly, on basis of Figure 3, it is helpful to show with the context-shift augmentation, the method can efficiently avoid overfitting the background spurious correlations, by tracing the Grad-CAM map of some same examples.\n\n3) It will be more solid to give some interpretation from the theoretical analysis: why the re-sampling encourages the method to learn more spurious correlations compared the method without re-sampling, or why the two-stage methods reduce the overfitting of spurious correlation in re-sampling. What the benefits or lessons we get in re-sampling from some theoretical perspectives.\n\n4) It lacks of more experiments about some hyperparameters like \\lambda of Eq.(7). Especially, the hyperparameter can be tuned based on different datasets and even different classes. The authors can strengthen this parts to avoid the concerns about the hyperparameter selection or heuristics.\n\nQUESTIONS:\n1) How is the visualization of representation on datasets that contains more class-irrelevant contexts with or without resampling? And how is the visualization of Grad-CAM for examples with or without context-shift augmentation?\n\n2) How is the sensitivity of the hyparameter lambda in Eq. (7)? How is the performance of the proposed method applied to some recent SOTAs, i.e., their compatibility with recent SOTAs?\n\n 3) Why do not consider the SOTAs like LA or BCL, PaCo in the experimental parts? The authors can have some more explanations in choosing the baselines.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}

comments: SUMMARY:
This paper studies the effect of re-sampling for long-tail learning. It finds that re-sampling is highly related with the contexts of images. And it does not necessarily work of fail in long-tail learning. Based on these findings, a context-shift augmentation module is proposed. Experiments verify the effectiveness of the proposed module.

STRENGTHS:
The paper studies an important task and proposes a unique solution for the found issues of re-sampling.

The paper is well-written and easy to follow.

The paper comprehensively introduces the preliminary of the proposed method.

WEAKNESSES:
The concept of context and content appear a lot in the paper, but there is no formal introduction or definition for these two concepts.

The paper illustrates that the irrelevant context issue occurs for MNIST and Fashion. And for IMAGENET, the issue is not significant. So is the proposed method more suitable for those simple datasets like MNIST and Fashion? If yes, this will highly affect the application of the proposed method for real-world tasks.

The paper uses Grad-CAM to determine the foreground and background for training images. However, for tail-class images, since the model may baise to head class, the generated activation map may not be accurate, which will affect the final results.

QUESTIONS:
More examples are encouraged to be attached in Introduction to better illustrate the found issue and the basic motivation of the paper.

The paper illustrates that the irrelevant context issue occurs for MNIST and Fashion. And for IMAGENET, the issue is not significant. So is the proposed method more suitable for those simple datasets like MNIST and Fashion? If yes, this will highly affect the application of the proposed method for real-world tasks.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-24