How Re-sampling Helps for Long-Tail Learning?
Authors: Jiang-Xin Shi
Tong Wei
Yuke Xiang
Yu-Feng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: long-tail learning
class-imbalanced learning
re-sampling
TL;DR: This paper investigates the reasons behind the success or failure of re-sampling approaches in long-tail learning.

Abstract: Long-tail learning has received significant attention in recent years due to the challenge it poses with extremely imbalanced datasets. In these datasets, only a few classes (known as the head classes) have an adequate number of training samples, while the rest of the classes (known as the tail classes) are infrequent in the training data. Re-sampling is a classical and widely used approach for addressing class imbalance issues. Unfortunately, recent studies claim that re-sampling brings negligible performance improvements in modern long-tail learning tasks. This paper aims to investigate this phenomenon systematically. Our research shows that re-sampling can considerably improve generalization when the training images do not contain semantically irrelevant contexts. In other scenarios, however, it can learn unexpected spurious correlations between irrelevant contexts and target labels. We design experiments on two homogeneous datasets, one containing irrelevant context and the other not, to confirm our findings. To prevent the learning of spurious correlations, we propose a new context shift augmentation module that generates diverse training images for the tail class by maintaining a context bank extracted from the head-class images. Experiments demonstrate that our proposed module can boost the generalization and outperform other approaches, including class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods. The source code is available at https://www.lamda.nju.edu.cn/code_CSA.ashx.

How Re-sampling Helps for Long-Tail Learning?
Jiang-Xin Shil* Tong Wei Yuke Xiang² Yu-Feng Li
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China School of Computer Science and Engineering, Southeast University, Nanjing. China Coonsmmer BG, Huawei Technologies, Shenzhen, China 1iy nju, edu. en, weit@seu, eduaseuuedu..ce cn, yuke. xi xiang@huaweii.com com
Abstract
Long-tail learning has received significant attention in recent years due the challenge it poses with extremely imbalanced datasets. In these datasets, only few classes (known the head classes) have an adequate number of training samples, while the rest of the classes (known as the tail classes) are infrequent in the training data. Re-sampling
classical and widely used approach for addressing class imbalance issues. Unfortunately, recent studies claim that re-sampling brings negligible performance improvements in modern long-tail learning tasks. This paper aims to investigate this phenomenon systematically. Our research shows that re-sampling can considerably improve generalization when the training images do not contain semantically irrelevant contexts. In other scenarios, however, it can learn unexpected spurious correlations between irrelevant contexts and target labels. We design experiments two homogeneous datasets, context and the other not, to confirm our findings. To prevent the learning spurious correlations, generates diverse training images for the tail class by maintaining a context bank extracted from the head-class images. Experiments demonstrate that proposed module can boost the generalization and outperform other approaches, including class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods. The source code is available at https: / //www/ www.lamde lamda. njju. edu. cn/ code_ Site__SA.aas ashx
containing irrelevant
propose new context shift augmentation module that
Introduction
Deep neural networks have achieved great success by applying well-designed models on large-scale elaborated datasets I 2 33. However, real-world data often exhibits long-tail class distribution [4] 6. Learning from long-tail datasets has two main challenges, one the class-imbalanced problem which causes the model biased towards the dominated head classes, and another is the data scarcity problem leading the poor generalization on those rare tail classes
I0].
One simple and intuitive approach to deal with the class-imbalanced problem is re-sampling 11] ie., create the replicate of the dataset to estimate the model parameters. Unfortunately, it has been reported that re-sampling methods achieve limited effects when applied to most long-tail datasets 66 [3]. Currently, there still few concrete and comprehensive explanations for this observation. Existing works mainly conclude that re-sampling will lead to the overfitting problem, and thus will be harmful to long-tail representation learning 6 [4]
Recently, many two-stage approaches have been proposed to improve the tail-class performance by adopting re re-sampling in the second training stage. For instance, DRS [5] adopts re-sampling
"quall contribution corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

schedule at the last several episodes of the training process. cRT [6] first trains a preliminary model using the uniform sampler. then fixes the representations and re-trains the linear classifier using a class-balances sampler. Last but not least, BBN [14] adjusts the whole model to first learn from the conventional learning branch and dynamically move to the re-balancing branch. Overall, the two-stage method has attracted widespread attention due to its basic hypothesis that uniform sampling is beneficial to representation learning, and the linear classifier. In light of the success of the two-stage method, a natural question is:
sampling can be used to fine-tune the
Can re-sampling benefit long- taig learning in the single-stage framework?
To answer this question, this paper empirically studies the re-sampling strategies and finds that re-sampling leads to opposite effects on long-tail datasets. Figure [ gives a brief view of this phe- nomenon. Moreover, ing samples are highly semantically related to their target labels, class-balanced re-sampling can learn discriminative feature representations; otherwise, uniform sampling is even better than class-balanced re-sampling, as the latter suffers from oversampling redundant unrelated contexts. To verify this, de- sign pair of synthetic benchmarks with the same content but different contexts, one containing irrele- vant context in training samples and the other not. Experiments confirm that re-sampling achieves conspicuous different performances on these two benchmarks. particular, when irrelevant context exists, balsnee re-sampling learns poorer representations compared to uniform sampling, thus the irrelevant context negatively affects re-sampling methods.
MNIST-LT
CIFARI00-LT
Tost Accurncy (%)
84.8
89.2
90.2
Test, Accurncy (9)
39.1
38.3
37.6
75.5
36.2
deduce that if the train-
65.8
34.1
0.25
0.75
0.25
0.5 0.75
Balance Ratio
Balance Ratio Y
Figure 1: Performance of re-sampling on two long-tail datasets. The sampling weight for data point (æ. y) is defined as n, where my denotes the class frequency of class y.
We believe that re-sampling can benefit long-tail learning in the single-stage framework. It fails on some long-tail datasets mainly because it overfits the oversampled irrelevant contexts and learns unexpected spurious correlations. If such spurious correlations are avoided, re-sampling can be helpful for long-tail learning. Motivated by this, we propose new context-shift augmentation module, which transfers well-separated context from head-class data to tail-class data. Specifically, it extracts the unrelated contexts (e.g. the backgrounds or the unrelated foreground objects) from head-class images and pastes them onto tail-class images to generate diverse novel samples. In this way, it encourages the model to learn more discriminative results for the tail classes. We conduct experiments three long-tail datasets, CIFAR10-L1 CIFAR100-LT,, and ImageNet-LT. The results show that the proposed module achieves competitive performance compared the baseline methods. In summary, our main contributions are:
We conduct empirical analyses on different datasets and discover that re-sampling does not necessarily work fail in long-tail tail learning.
We deduce that the failure of re-sampling may be attributed to overfitting on irrelevant contexts, and our empirical studies confirm our hypothesis.
We propose a new context-shift augmentation module to prevent re-sampling from overfitting to irrelevant contexts in a single-stage framework.
Extensive experiments verify the effectiveness of the proposed module against class-balanced re-sampling, decoupled classifier re-training, and data augmentation methods.
The rest of the paper organized as follows. Section studies the effects of re-sampling approaches. Section 3 presents the proposed context-shift augmentation module. Section 4 briefly reviews related
works. Section 5 concludes the paper.
A Closer Look at Re-sampling
2.1 Preliminaries
Given a training dataset D (x,, yi ) where a; training sample and yi E = [K] = (l..... K} is the class label assigned to it. We assume that the training data follow a long-tail class

---

Table 1: Test accuracy (%) of CE with uniform sampling, classifier re-training (cRT), and class- balanced ramsaip (CB-RS) on four long-tail tail benchmarks. We report the accuracy in terms of all, many-shop medium-shot, and few-shot classes.
MNIST-LT
Fashion-LT Many
CIFARI00-LT
ImageNet-LT Med.
All
Many Med. Few
All
Med.
Few
All
Many
Med.
Few
All
Many
Few
CE
65.8
99.1
89,9
0.0
45.6
94.7
43.1
0.0
39,1
65.8
36.8
8,8
35.0
57.7
26.5
4.7 23.6
cRT
82.5
96.6
89.4
58.8
60.3
77.1
61.4
42.1
41.6
63.0
40.4
16.5
41.9
52.9
39.2
CB-RS
90.8
98.7
94.4
77.7
80.5
86.6
74.3
82.8
34.1
59.5
31.1
6.2
37.6
47.5
36.5
16.7
Training Set
Test Set
Training Set.
Test Set.
tall-classes
tadl classss
(a) Uniform sampling.
(b) Class-balances re-sampling.
Figure 2: Visualization of learned representation of training and test MNIST-LT. Using class-
balanced re-sampling yields more discriminative and balanced representations.
distribution where the class prior distribution Pty) is highly skewed that many underrepresented classes have very low probability of occurrence. Specifically, we define the imbalance ratio as p max, /v)) min, Pyy) indicate the skewness data. Classes with high Pyy) are referred to as head classes, while others are referred to as tail classes.
In practice, since the data distribution is unknown, Empirical Risk Minimization (ERM) uses the training data to achieve an empirical estimate of the underlying data distribution. Typically, one minimizes the softmax cross-entropy following
(xp(f((x))
Ely, f(x)) = log
(I)
Eye(k) exp(fy
where (z) denotes the predictive logit of model f on class y. However, this ubiquitous approach neglects the issue of class imbalance and makes the model biased toward head classes [6]]]]
To deal with the class-imbalance problem, the re-sampling strategy assigns a probability of being selected for each training sample according to its class frequency (6]. The probability of sampling data point from class &t can be written as:
T2
Ph EME[KK] D
where ny denotes the frequency of class & and q E [0, 1]. When q = 1, Equation 2 denotes uniform sampling, where each training sample has an equal probability of being selected. When 0, Equation (2) denotes the balsnca re-sampling, which selects samples from every class k with the identical probability of 1/K.
2.2 Exploring the Effect of Re-sampling
2.2.1 Re-sampling can learn discriminative representations
To better explore the effect of the re-sampling strategy, we conduct experiments on multiple long-tail datasets, including MNIST-LT, Fashion-LT, CIFAR100 100-LT [15]. and ImageNet-LT [5]. We compare three different learning methods: 1) Cross-Entropy (CE) with uniform sampling: 2) Classifier Re-Training (cRT) which uses uniform sampling to learn the representation and class-balanced re- sampling to fine-tune the classifier; 3) Class-Balanced Re-Sampling (CB-RS) for the whole training process. We report the experimental results in Table [

---

Ingut
Uniform
CB-RS
Inpat
Uniform
CB-RS
bear
bicycle
Head Classes
fox
bus
Tail Classes
wolf
train
Figure 3: Visualization of features with Grad-CAM 17 on CIFARIO0-LT, Uniform sampling mainly
learns label-relevant features, while re-sampling overfits the label-irrelevant features.
MNIST-LT
CMNIST-LT
Class Accuney (%6
Class Accurey (%)
Overall (%)
Uniform
75
75
80
eRT
50-
50
60
25
Uniform
25
eRT
CB-RS
3
2
8
20
Index
Index
MNIST-LT
CMNIST-LT
(a) Comparison of class accuracy.
(b) Overall accuracy.
Figure 4: Comparison of Uniform sampling, cRT, and CB-RS MNIST-LT and CMNIST-LT.
According to the results, cRT performs best on CIFARIO0-LT 100-LT and ImageNet-LT, which is consistent with previous works 14 6 CE and cRT use the same representation but cRT achieves higher performances, which indicates that re-sampling can help for classifier learning. However, on MNIST- LT and Fashion-LT, CB-RS surprisingly achieves the highest performance and outperforms CE and cRT by a large margin. Since cRT and CB-RS both use learning, the results indicate that CB-RS learns better representations than uniform sampling MNIST-LT and Fashion-L.T.
re-sampling for classifier
To further understand the effect of re-sampling, we visualize the learned representation on MNIST-LT Figure 2 The figures show that with uniform sampling, the representation space is dominated by head classes, and the representations of tail classes are hard to distinguish. By applying class-balanced re-sampling, the representations of both head and tail classes are discriminative.
2.2.2 Re-sampling is sensitive to irrelevant contexts
We have demonstrated the generalization ability of the re-sampling strategy MNIST-LT and Fashion-LT. Nevertheless, re-sampling performs unsatisfactorily on the other two datasets. Since the training samples and target labels on MNIST and Fashion are highly semantically correlated [18] 19) while samples on CIFAR and ImageNet contain complex contexts 20 3, re-sampling is sensitive the contexts in training samples.
we hypothesize that
To support our hypothesis, we visualize the Grad-CAM of the representation learned by different sampling strategies in Figure When training with uniform sampler, models can distinguish the contexts from samples of head classes. However, when adopting a class-balanced re-sampler, the model tends to overfit the irrelevant context from the over-sampled tail data, which unexpectedly affects the representation of head classes. For example, when classifying different kinds of animals, re- sampliing might focus the posture rather than the appearance. Also, when classifying different vehicles, re-sampling easily influenced by the human in tail-class images, and thus mistakenly focuses the human in head-class images.
To further validate our point, we design a homogeneous benchmark of MNIST-LT termed Colored- MNIST-LT (CMNIST-LT), We inject colors into MNIST-LT to artificially construct irrelevant contexts.

---

Specifically, we design CMNIST-LT based on two considerations. First, head classes are prone to have rich contexts, so we inject different colors into the samples of each head class. Second, tail classes have limited contexts, so we inject an identical color into the samples of each tail class. We conduct uniform sampling, classifier re-training, and class-balanced re-sampling on MNIST-LT and CMNIST-LT. The experimental results are illustrated in Figure | The results show that when applied to MNIST-LT, CB-RS can boost the tail-class performance without degradation on head classes. However, for CMNIST-LT, CB-RS performs worse than uniform sampling and cRT on both head and tail classes, thus validating the negative impact of irrelevant contexts on re-sampling methods. Since re-sampling succeeds on MNIST-LT and fails on CMNIST-LT, we propose that re- re-sampling does not always fail, it can help for long-tail tail learning if avoiding the irrelevant contexts.
2.2.3 Proposed benchmark datasets
We follow the previous works [5 14 to construct MNIST-LT, Fashion-I.T and CIFART00-LT and set the imbalance ratio to 100. ImageNet- LT is proposed by [5]. For MNIST-LT and Fashion-LT, use LeNet [21] as the backbone network and add a linear embedding layer before the fully connected layer to project the representation into 2-dimensional space for better presentation. We use standard SGD with a mini-batch size of 128, an initial learning rate of 0.1 and a cosine annealing schedule to train the model for 8 epochs. When applying cRT, we retrain the last fully connected layer for 4 epochs by fixing the other layers. For CIFAR100-LT and ImageNet-LT, more details are in Section 3.3]
To construct CMNIST-LT, we follow the idea of CMNIST 22 to first randomly flip the label and then inject colors into the training samples. However, different from CMNIST which converts the MNIST to binary classification dataset, we keep the ten classes to better simulate a long-tail class distribution. To generate flipped labels on the long-tail dataset MNIST-LT, we follow the method in (23] and set the flipping probability to 1/4. We generate ten different colors using the scaborr") package. For the five head classes, we randomly inject one of these ten colors into each sample with equal probability. For the other five tail classes, we inject samples of each class with single color.
3 A Simple Approach to Make Re-sampling Robust to Context-shift
3.1 Extracting Rich Contexts from Head-class Data
By studying the effects of re-sampling methods in different scenarios, we can draw a conclusion: when the training samples contain irrelevant contexts, simply over-sampling the tail-class samples might cause the model to unexpectedly focus on these redundant contexts, thereby resulting in the overfitting problem. However, the head classes have rich data learn model with good generalization ability. We naturally raise a question: can we utilize the rich contexts from head data to augment the over-sampled tail data to alleviate the negative impact of irrelevant contexts? Inspired by this motivation, we design context-shift augmentation module by extracting the rich contexts from head-class data to enrich the over-sampled tail-class data.
To leverage the rich contexts in head-class data, we utilize a model f trained with uniform sampling for context extraction. First, we select well-learned samples with fitting probability larger than threshold 8 to improve the extraction quality. The fitting probability for sample æ; can be calculated by the Softmax function as follows
exp(2%.y)
ply 0i x., 1") Ey' exp( (2
where z. denotes the logits predicted by f" i.e., x = [2 ,,........ 24 = (x; Then, we use off-the-shelf methods such as Grad-CAM [24] [7] extract the image contexts, which is also used in previous works regarding open-set learning and adversarial learning (25) 26). Specifically, given an image Bi, we calculate its class activation map CAM(#: l"). Then, we inverse the map get the background mask M,, i.e., M, = =1 CAM(2, f"). Here M, is a matrix of the same size as Tt, with values between 0 and 1. A higher value indicates that the corresponding pixel is more likely to be the background. Different from previous works that discretize the mask matrix to binary values (27 28 we keep the floating values in the matrix to conserve more information.
http://seaborn, http:///wwtt......ee.////aaoaaaa
5

---

After calculating the background mask M, of image a; we paste the mask onto the original image by M, z; to obtain background image. this way, we separate the semantically related contents from the images and keep the rest contexts for further augmentation. Finally, the extracted contexts are pushed into a memory bank Q for augmentation of re-sampled data.
For the training of the uniform module, we apply the conventional ERM algorithm. For each training sample 2;, we calculate its loss by
f (x. C = C (2, v4)
(4)
(5)
where can be any loss function. Generally, we use the standard cross-entropy loss.
3.2 Balanced Re-sampling with Context-shift Augmentation
Simply adopting balanced sampling might generate many repeated samples from the tail classes and lead to the overfitting problem. Therefore, we ask for background images from the context memory bank Q and paste them onto the re-sampled images. In this way, we generate more diverse novel samples by simulating each tail-class image within various contexts. Specifically, for re-sampled training image D;, we ask for another image á together with its mask M, from Q, and fuse it with à; to generate a novel sample, and calculate its training loss as follow:
À Uniform (0, 1)
(6)
3. XM, Gb. (1 AM,) e a.
(7)
2 = (x:):
(8)
C. = ehh (z), 31)
(9)
Here is randomly generated between [0, 1] to increase the diversity. Different from previous mixup- based methods 29 (0), method does not change the target label, because the pasted background is not related to the semantics of any class labels.
To reduce the computational complexity, the uniform module and the balanced re-sampling module can be trained simultaneously by sharing the same feature extractor c) and training their own linear classifiers 16th and & i..., f"(+) 26th (Ø(-)) and (*(.) (ó(-).. Since the classifier is lightweight, it does not add much additional computational overhead. Moreover, the memory bank Q is designed as firs-in-first-our queue with maximum volume of V for a convenient query. After extracting the context from the uniform module, we append the 2; and M; pair into the context bank Q. When the size of reaches its maximized volume, the oldest contexts volume size is set equal to the mini- batch size in order to minimize the overhead as well as ensure the querying requirements. Finally, the summarized loss function is
pushed out. In practice, the
N
N C
C C L
(10)
N
Figure gives brief overview of the proposed module. The detailed training procedure is given in the supplementary material due to the page limit.
In the inference phase, only the balanced re-sampling module is used. In other words, the uniform module only serves as an assistant to provide more rich contexts for the re-sampling module during the training phase. Formally, for test data point æ. obtain the prediction by fx)) and then employ the Softmax function to obtain the predictive probabilities.
3.3 Empirical Results
We demonstrate the efficacy of the proposed module context shift augmentation by comparing it with different kinds of long-tail learning methods, including:
Re-sampling or re-weighting methods, such as Focal Loss CE-DRW (15), LDAM-DRW (15), cRT (6. LWS 6. and BBN [4].
31. CB-Focal [7]. CE-DRS 15
Head-to-tail knowledge transfer methods, such as M2m [32]
OLTR (5] and FSA 28

---

Long-tailed Dataset
Uniform Module
Activation map
Extracting Rich Contexts
Cind
Masking
Context memory bank
Loss 1"
Head cleesss
Original image
Background image
Re-sampling with Augmentation
Balanced sampler
Diverse novel samples
Tail
Loss 1
Figure 5: An overview of the proposed method.
Data augmentation methods, such as Mixup 229 Remix 33 CAM-BS 13 and CMO
27
conduct experiments on three long-tail datasets, including CIFARIO-LT [5] CIFARI00-L [5] and ImageNet-LT [5]. CIFARI0-1 10-LT and IOOATO are the long-tail versions of CIFAR datasets by sampling from the raw dataset with an imbalance ratio p. Following previous works 15 [4], we conduct experiments with p {100, 50, 10}. ImageNet-LTT is long-tail version of the ImageNet (3), which contains 1000 classes, each with a number of samples ranging from 5 to 1280. For each long-tail training dataset, evaluate the model another corresponding class-balanced test by calculating the overall prediction accuracy.
The results for CIFAR1O-LI and CIFAR100-LT are summarized in Table We report the results under imbalance ratio P 100, 50, 10. As shown in the table, our method achieves superior performance compared with the baseline methods in all settings.
Table 2: Test accuracy (%) on CIFAR datasets with various imbalanced ratios.
Dataset
CIFAR100-LT
CIFARIOA IO-LT
Imbalance Ratio
100
50
10
100
50
10
CE
38.3
43.9
55.7
70.4
74.8
86.4
Focal Loss
38.4
44.3
55.8
70.4
76.7
86.7
CB-Focal
39.6
45.2
58.0
74.6
79.3
87.1
CE-DRS [15]
41.6
45.5
58.1
75.6
79.8
87.4
CE-DRW [15
41.5
45.3
58.1
76.3
80.0
87.6
LDAM-DRW
11
42.0
46.6
58.7
77.0
81.0
88.2
cRT (6
42.3
46.8
58.1
75.7
80.4
88.3
LWS 6
42.3
46.4
58.1
73.0
78.5
87.7
BBN 14
42.6
47.0
59.1
79.8
82.2
88.3
mixup
29
39.5
45.0
58.0
73.1
77.8
87.1
Remix
33
41.9
59.4
75.4
88.2
M2m
32
43.5
57.6
79.1
87.5
CAM-BS
13
41.7
46.0
75,4
81.4
CMO 27
43.9
48.3
59.5
cRT+mixup
45.1
50.9
62.1
79.1
84.2
89.8
LWS+mixup
34
44.2
50.7
62.3
76.3
82.6
89.6
CSA (ours)
45.8
49.6
61.3
80.6
84.3
89.8
CSA + mixup (ours)
46.6
51.9
62.6
82.5
86.0
90.8
Specifically, the methods based on re-sampling or re-weighting such as DRS, DRW, cRT, and LWS can ease the class-imbalanced problem to some degree but the performance gain is limited due to the neglect of the representation learning. The methods based on knowledge transfer and data augmentations, such as M2m and CMO, achieve higher performance.

---

Moreover, by combining cRT and LWS with mixup, the performance achieves an obvious improve- ment. However, such two-stage training methods are end-to-end approaches. In contrast, the proposed context shift augmentation module enhances representation learning by enriching the con- texts of samples and adopting the class-balanced re-sampling to ensure balanced classifier. manner, it achieves more improvement with an end-to-end framework. Since cRT and the proposed module both use Blass-balanced sampling for classifier learning, the results indicate that the proposed context shift augmentation can achieve better representations.
this
We further conduct the experiments on a larger scale dataset ImageNet-LT. We calculate the accuracy of the overall test set and the average accuracy of the many-shot classes (more than 100 images in the training set), the medium-shor (20~100 images), and the few-shot classes (less than 20 images). We report the results in Table It shows that the proposed module is superior to most other baseline methods. The performance is similar to another data augmentation method CMO, our method achieves higher accuracy on few-shot classes, which demonstrates the generalization ability of context-shift augmentation for tail-class data.
Table 3: Test accuracy (%) on ImageNet-LT dataset.
ResNet-10 (All)
ResNet-50
All
Many
Med.
Few
CE
34.8
41.6
64.0
33.8
5.8
Focal Loss
31
30.5
OLTR 5
35.6
FSA 28
35.2
cRT 6
41.8
47.3
58.8
44.0
26.1
LWS 6
41.4
47.7
57.1
45.2
29.3
BBN
14
48.3
CMO
27
49.1
67.0
42.3
20.5
CSA (ours)
42.7
49.1
62.5
46.6
24.1
CSA (ours)
43.2
49.7
63.6
47.0
23.8
denotes longer training of 100 epochs.
We provide more detailed studies to analyze the effect of each component in the proposed module. Due to the page limit, we report the results in the supplementary material.
4 Related Work
4.1 Re-sampling and Re-weighting
Re-sampling is a widely used strategy in class-imbalanced learning [2] There are two main ideas of ramsampii 1) Over-sampling by repeating data from the rare classes. 2) Under-sampling by abandoning proportion of data from the frequent classes. However, when the class distribution is highly skewed, re-sampling methods often fail. Previous works point out that mnde--samplig may discard precious information which inevitably degrades the model performance, and over-sampling tends to cause the overfitting problem on the tail classes [1 15
Recent work 6 develops class-balanced re-sampling where samples from each class have an identical probability of being sampled. Class-balance re-sampling can bring performance gain for classifier learning but hurts representation learning. Therefore, two-stage approaches [15] 6 [4] adopt it at the late stage of the whole training process order impact the representation.
Re-weighting aims to generate more balanced predictions by adjusting the losses for different classes 35 (3). The most intuitive way is to weight each training sample by the inverse of its class frequency Similar to balanced re-sampling, re- weighting can achieve better results for tail classes but usually deteriorates its performance for head classes
In contrast, our empirical study reveals that class-balanced re-sampling can effective method as long as there exist no irrelevant contexts. It fails in some cases mainly due to the unexpected overfitting

---

towards the ver-sa redundant contexts. When applied with context-shif augmentation, class- balanced re-sampling can achieve competitive performance on long-tail datasets.
4.2 Head-to-tail Knowledge Transfer
As the head classes have adequate training data while the tail classes have limited data, recent works aim to leverage the knowledge gained from head classes enhance the generalization of tail classes. Feature transfer learning [7 utilizes the intra-class variance from head classes to guide the feature augmentation for tail classes. MetaModelNNt 4 uses the head data train a meta-network to predict many-shot model parameters from few-shot model parameters, then transfers the meta- knowledge to the tail classes. Major-to-minor translation (M2m) 32 uses the over-sampling method and translates the head-class samples to replace the duplicated tail-class samples via adversarial perturbations. OLTR [3] maintains dynamic meta-embedding between head and tail classes to transfer the semantic deep features from head to tail classes.
These methods assume that the head classes and the tail classes share some common knowledge such as the same intra-class variances, the same model parameters, or the same semantic features. In this work, we regard the contexts such knowledge and transfer the contexts from head-class data to enrich the tail-class data.
4.3 Data Augmentation
Several data augmentation approaches have been proposed to improve the model generalization ability. In contrastive learning [38] 39). curriculum learning [40], meta-learning methods [41] and instance segmentation tasks 42], data augmentation strategies have been shown to effectively improve the generalization of tail classes. MiSLAS [4] studies the mixup [29] technology in long-tail learning and finds that mixup have a positive effect representation learning but negative negligible effect on classifier learning. Remix 33 adapts the mixup method to re- re-balanced version. assigns the mixed label in favor of the tail class by designing disproportionately higher weight for the tail class, CMO 277 applies CutMix (30) by cutting out random regions of a sample from head classes and filling the removed regions with another sample from tail classes. By this means, it enriches the contexts of the tail data; but the random cutout operation does not necessarily separate the contents and contexts.
The attention or CAM-based methods have been proposed to improve long-tail learning via feature decomposition and augmentation. CAM-BS [13 separates the foreground and background of each sample, then augments the foreground part by flipping, translating, rotating, or scaling. Feature Space Augmentation (FSA) (28) uses CAM decompose the features of each class into class generic component and a -lass-specific component, and generates novel samples in the feature space by combining the class-specific components from the tail classes and the class-generic components from the head classes. Attentive Feature Augmentation (AFA) [43 adopts feature decomposition and head-to-tail
augmentation via the attention mechanism. Note that FSA and AFA can also be knowledge transfer approaches. Nevertheless, these methods neglect that the learned model has limited generalization ability on tail classes, and most foregrounds (or class-specific components) of samples from tail classes are incredible. In comparison, our method applies CAM to separate related contents and unrelated contexts for samples mainly from head classes. It then pastes the contexts extracted from the head-class data onto the over- sampedd tail-class data to enrich the contexts.
5 Conclusion
In this work, we study the re-sampling strategy for long-tail learning. Our empirical investigations reveal that the impact of re-sampling is highly dependent on the existence of irrelevant contexts and is not always harmful to long-tai tail learning. To reduce the influence of irrelevant contexts, we propose a new context-shift augmentation module that leverages the well-separated contexts from the head-class images to augment the over-sampled tail-class images. We demonstrate the superiority of the proposed module by conducting experiments on several long-tail datasets and comparing re- re-sampling, decoupled classifier re-training, and data augmentation methods.
against

---

Broader Impact and Limitations
This paper investigates the reasons behind the success/failure of re-sampling approaches long-tail learning. In critical and high-stakes applications, such medical image diagnosis and autonomous driving, the presence of imbalanced data poses the risk of producing biased predictions. By shedding light on this problem, we aim inspire more research safe and robust re-sampling approaches.
One may be concerned about combining the proposed module with other methods such as self- supervised learning [44,, logit adjustment [T6]. We conduct additional experiments and report the results in the supplementary material due to the page limit. Nevertheless, the proposed module can not achieve comparable performance with the well-designed models [45] 46 47), since our intention not to achieve performance that is on par with sate-of-ttheaar methods. Instead, we hope that our findings will inspire future research regarding re-sampling methods.
Data Availability Statement
The source code of our method is available at at/https:/www. hwtp://wwwww..aa..mmmmmma .nj. adj. edu. en/code_Coo CSAA ashx or https://gittub. httpg/ttttbbcoo.oeeee sh sttpm//ssiiss///s/eo/sso jxcs/CSc
Acknowledgments and Disclosure of Funding
This research was supported by the National Key R&D Program of China (2022ZD01 the National Science Foundation of China (62176118, 61921006).
References
[1] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444 2015.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-778, 2016.
[3] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Puttern Recognition, pages 248-255. 2009.
[4] Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, volume 30, pages 7029-7039, 2017.
[5] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[6] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
[7] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based effective number of samples. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9268-9277, 2019.
[8] Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. Advances in Neural Information Processing Systems, volume 33, pages 4175-4186, 2020.
[9] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 9495-9504, 2021.
[10] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. In Advances in Neural Information Processing Systems, volume 33, pages 19290-19300 2020.
10

---

[11] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321- 357, 2002.
[12] YunYing Liu, Jianxin Wu, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539- 550, 2008.
[13] Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-taine toigetailed visual recognition with deep convolutional neural networks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 3447-3455, 2021.
[14) Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9719-9728, 2020.
[15] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbal- anced datasets with label-distribution-avaaa margin loss. In Advances in Neural Information Processing Systems, volume 32, pages 1565-1576, 2019.
[16] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiy Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021.
[17] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618-626, 2017.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 96(11):2278-2)) -2244, 1998.
[19] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:170..07744 2017.
[20] A Krizhevsky. Learning multiple layers of features from tiny images. Master's 's thesis, University of Tront, 2009.
[21] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with back-propagation network. Advances in Neural Information Processing Systems, 1989.
[22] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint 1907.09997700293 2019.
[23] Tong Wei, Jiang-Xin Shi, Wei-Wei Tu, and Yu-Feng Li. Robust long-tailed learning under label noise. arXiv preprint arXiv:2108. //669, 2021.
[24] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
[25] Jie-Jing Shao, Xiao-War Wen Yang. and Lan-Zhe Guo. Open-set learning under covariate shift. Machine Learning, pages 1-17, 2022.
[26] Qibing Ren, Yiting Chen, Yichuan Mo, Qitian Wu, and Junchi Yan. Dice: Domain-attad invariant causal learning for improved data privacy protection and adversarial robustness. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 1483-1492, 2022.
[27] Seulki Park, Youngkyu Hong. Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6887-6896, 2022.
[28] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long- tailed data. In Proceedings of the 16th European Conference on Computer Vision, pages 694-710, 2020.
[29] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
11

---

[30] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6023-6032. 2019.
[31] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-2988, 2017.
[32] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13896-13905, 2020.
[33] Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. In Proceedings of the 16th European Conference on Computer Vision, pages
95-110, 2020.
[34] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16489-16498, 2021.
[35] Charles Elkan. The foundations of cost-sensitive learning. In Proceedings of the 17th Interna- tional Joint Conference on Artificial Intelligence, pages 973-978, 2001.
[36] Zhi-Hua Zhou and Xu- Ying Liu. Training cost-sensitive neural networks with methods address- ing the class imbalance problem. IEEE Transactions on Knowledge and Data Engineering,
18(1):63-77, 2006.
[37] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with Conference on Computer Vision and Pattern Recognition, pages 5704-5715 2019.
data. In Proceedings of the IEEE/CVF
[38] Zhihan Zhou, Jiangehao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In Proceedings of the 39th International Conference on Machine Learning, pages 27367-27377 2022.
[39] Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Bal- anced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 6908 (998-6997,20 2022.
[40] Sumyeong Ahn, Jongwoo Ko, and Se- Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. In International Conference on Learning Representations, 2023.
[41] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5212-5221, 2021.
[42] Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 3457-3466, 2021.
[43] Weiqiu Wang, Zhicheng Zhao, Pingyu Wang, Fei Su, and Hongying Meng. Attentive feature augmentation for long-tailes visual recognition. IEEE Transactions on Circuits and Systems for Video Technology, 32(9):5803-58 5816, 2022.
[44] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust dataset imbalance. In International Conference on Learning Representations, 2022.
[45] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recogni- tion by routing diverse distribution-awann experts. In International Conference on Learning Representations, 2021.
[46] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6626-6636 2021.
[47] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 715-724, 2021.
12

---

[48] Zhuoxun He. Lingxi Xie, Xin Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Data augmentation revisited: Rethinking the distribution gap between clean and augmented data, arXiv preprint arX9v:1909.099 091409 2019.
13

---

A Training Procedure
The training procedure of context-shift augmentation is summarized in Algorithm |
Algorithm Training procedure of context-shiff augmentation
Input:
training data D { ys) context memory bank Q, maximum volume size V; model parameters Φ. _ f loss functions f", lb:
Procedure:
1: Initialize model parameters 6, f":
2: Re-sampling
dataset D {(x;, Ti))
3: Empty memory bank Q:
4: for epoch ,, do
5: repeat
6:
Draw a mini-batch (æ,, 1=) from D;
7:
Draw a mini-batch (æ,, ū4 ! from D; I/ uniform module
8:
9:
for i = 1,.... do
10:
Calculate x ()()))) and D = (=",,,))) if p(y = y/ 3i (") then Calculate background mask M; æ;; Push (æ;, M;) into Q;
11:
12:
14:
end end for
Calculate L = EE C.;
16:
I/ balanced re- ra-samlinng module if Size of Q reaches V then Obtain contexts (ă:, M.) from Q:
18:
20:
À ~ Uniform 1); for i 1..... B do
22:
8. XM, 3 d; + (1 AM;) o ã.;
23:
Calculate 2 = fo (o(z:)) and L (z).
end for
25:
Calculate C B Σ C.:
else
27:
Assign L 00
end if
29:
// total objective function
30:
Calculate - C" Lb:
31:
Update model parameters é. f" fd with L;
32:
until all training data are traversed.
33:
end for
B Implementation Details for context-shift augmentation
For experiments CIFARIO-LT and 100-L,, and train it using standard SGD with a momentum of 0.9, a weight decay of x 10-1, a batch size of 128. The model trained for 200 epochs. The initial learning rate factor of 10 at 160 and 180 epochs. We train each model with NVIDIA GeForce RTX 3090.
ResNet-32 the backbone network
0.2 and is annealed by
For experiments We use standard SGD with a momentum of 0.9, a weight decay of 5 x 10-4 and a batch size 256 to train the whole model for total of 90 epochs. We use the cosine learning rate decay with initial learning rate of 0.2. We train each model with 2 NVIDIA Tesla V100 GPUs.
ImageNet-LT,
implement the proposed method ResNet-10 and ResNet-50.
In all experiments, we first warm up the uniform module for 10 epochs and then train the uniform module and the balanced ra-sampling module simultaneously for the rest epochs. For the uniform module, we follow the simple data augmentation used in 2 with only random crop and horizontal
14

---

flips. For the re-sampling module, we use the proposed context-shifi augmentation method. We apply the trick proposed by [48] disable the augmentation in the balanced re-sampling module the last 3 epochs to obtain further improvements, which is also applied in other baseline methods 13 27 We set the threshold 5 to 0.8.
C Additional Illustrations
For convenience in understanding context and content, we give an example in Figure 6 Context refers to the semantically unrelated parts in the images, and content refers to the semantically related parts, Moreover, We give brief illustration of the generated dataset CMNIST-LT Figure
Head Closses
5k
context
4k
Frammles
3k
Tall Closses
content (bicycle)
2k
1k
Figure 6: Illustration of context and content. Taking photo of the bicycle as an example.
0
1
2
3
9
Class Index
Figure 7: Illustration of the CMNIST-LT benchmark.
D Additional Experimental Results
D.1 Effects of the context bank
The context bank Q is a novel component context-shift augmentation which receives diverse contexts from the uniform module and provides them to augment the data in the re-sampling module. To verify the effectiveness of the context bank, we remove it from the framework and train the model on CIFAR100-LT with an imbalance ratio of 100. The results reported in Table
Table 4: Ablation study on the context bank Q.
All
Many
Med.
Few
Ours
45.8
64.3
49.7
18.2
41.2
65.1
41.9
10.7
Ours w/o Q
(-4.6)
(+0.8)
(-7.8)
(-7.5)
The results show that without context bank Q, the performance decreases by a large margin. The performance degradation mostly comes from the medium-shot classes and the few-shot classes, which indicates that the context bank significantly improve the generalization of tail classes.
Morcover, we study the effect of different variants in the context bank. First, we study the threshold for sample selection and report the results in Table On the one hand, if 8 is too high, the selected samples will be very few. On the other hand, if is too small, the selected samples might be not well learned. Nevertheless, the training process progresses, most samples will fit well, our method is not sensitive to 8. We set 8 = 0.8 considering its best performance.
Second, we study the influence of the volume size V of the context bank Q and report the results Table 6 Since the bank Q is a first-in-first-out queue, the latest incoming contexts are more convincing. When the volume size is too large, the bank might contain more past samples. Besides, a larger size of V would bring more memory overhead. So we set the volume size V equal to the mini-bate size B in our method.
15

---

Table 5: Influence of the threshold 5.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Accuracy
45.47
45.37
45.55
44.83
45.52
45.59
45,42
45.08
45.83
44.93
Table 6: Influence of the bank volume size (compared with the mini-batch size B).
Volume
X
x2
x4
x8
16
x32
X64
Accuracy
45.83
45.60
45.57
45.32
45.55
45.37
45.26
D.2 Influence of augmentation variants
We use variant À Uniform(O, 1) for generating novel samples. The value of À can result in different proportions foreground and background in the novel sample. Also, the size the sampling space affects the diversity of the novel images. To explore the effect of X, we try À ~ Uniform(e (a, b) and À ~ Beta(a, a, b) to train context-shiff shift augmentation on CIFARI00-LT with imbalance ratio 100 and report the results in Figure
First, when close to O, the background merely takes effect, and the performance decreases a lot. Second, when À = the background image might cover the important content in the foreground image. Also, the diversity new samples is limited, Although the performance is better than that of À 0, it is still unsatisfactory. Overall, choosing À ~ Uniform(O, 1) or À ~ Beta(1, 1) lead to the best performance.
0.25
0.50
0.75
1.00
1.0
3.0
4.0
5.0
1.0
45.8
45.1
45.0
44.4
44.2
44.55
-45
2.0
45.4
45.0
45.5
43.8
44.7
44
44.55
0.50
43.7
44.1
3.0
44.2
45.4
45.5
43.4
45.0
-1
0.25
41.8
42.3
-43.5
41.2
X Uniform b)
Beta(a, b)
Figure 8: Influence of variants X.
D.3 Comparison between different modules
In our framework, the uniform sampling module is only enabled the training phase. While in the inference phase, use the balanced re-sampling module to predict unseen instances. To verify the superiority the re-sampling module, we compare the performance these two modules as well as their ensemble. We report the results in Table 7 The results show that the re-sampling module is superior to the uniform module, and even achieves higher accuracy than the ensembled results. This also indicates the superiority of the proposed context-shift re-sampling method.
Moreover, we study the influence of different balance ratios on our re-sampling module and compare it with the vanilla re-sampling method. We report the results in Figure 9 For vanilla re-sampling. adopting a more balanced re-sampling would yield more severe performance degradation. In contrast, method achieves higher performance through class-balances resampling.
16

---

Table 7: Comparison between the uniform module and the re-sampling module in context-shift angmentation.
All
Many
Med.
Few
Uniform module
39.4
68.3
37.3
6.1
Re-sampling module
45.8
64.3
49.7
18.2
Ensemble results
43.0
67.5
44.1
11.4
Test Accuracy (%)
Re-sampling
Test Accuracy (%)
Ours
39.1 * 37.6
45.8
38.3
44.5
43.8
36.2
42.9
34.1
41.4
0
0.25
0.5 0.75
0.75 Balanee Ratio
0.25
0.5
Balance Ratio y
Figure 9: Comparison re-sampling and our method under different balance ratios 7.
D.4 Comparison of the learned representation
We visualize the representation of CIFAR100) in Figure T0 Each color represents a class, with darker colors representing head classes, and lighter colors representing tail classes. Although the fine-grained colors make it difficult to distinguish some classes, it can still be seen that CB-RS learns worse representation compared with vanilla CE. Moreover, our proposed method can learn more distinguishable representation. Moreover,
visualize the Grad-CAM for examples with our proposed context-shift augmentation. We choose the same examples in Figure S in the main paper and report the results in Figure II The results show that our method can alleviate the negative impact on head-class samples caused by the overfitting problem.
CBRS
Figure 10: Visualization of learned representation on 100-LT
Ingut
Uniform
CB- RS
Ours
Input
Uniform
CB- RS
Ours
bear
bicycle
Head Closes
Fox
bus
Tall
WoLE
Figure 1I: Visualization features with Grad-CAM on CIFARIOO-LT. Our method can alleviate negative impact on head-class samples caused by the overfitting problem.
17

---

D.5
The influence of Grad-CAM
The Grad-CAM 24 is utilized extract unrelated contexts in previous works such as open-set learning and adversarial learning [25] [25]26] Also. we use Grad-CAM in the context-shiff augmentation generate diverse contexts for tail-class data. We compare Grad-CAM with CAM [17]. The results shown in Tab. Tab. demmnstratee that Grad-CAM is superior to CAM when applied to our method. One may be concerned with the generated activation map of tail-class images. We visualize some tail-class samples with predicted probabilities higher than ó in Figure 12 which shows that the model can still grap accurate activation maps for tail-class samples.
Table 8: Comparison between CAM and Grad-CAM in context-shifi augmentation
All
Many
Med.
Few
Ours w/ CAM Ours w/ Grad-CAM
45,1 45.8
63.8
48.1
18.0
64.3
49.7
18.2
clanc train n-54 94%
clam: tou4l Twrbh 01%
clas: salpp p-99.41%
clam: wundube p-4 82%
class whale p-18 22%
clas: wolf 2-12 99%
Figure 12: Visualization of features with Grad-CAM. For tail-class samples with predicted probabili- ties higher than the threshold (0.8), the model can also grap accurate activation maps.
D.6 Combination with self-supervised learning
It is interesting combine self-supervised methods with context-shifi augmentation. Inspired by this, we follow the self-supervised + fine-tune method, i.c. SimSiam-rwSAM in [44] and conduct extensive experiments the CIFARIO-LA dataset. The results are shown in Tab,
Table 9: Combing self-supervise method SimSiam+rWSSAM with different re- re-balancing methods
Pre-train model
Fine-tune method
Accuracy (%)
SinSiam+rWSAM SimSiam+rWSAM
CE
69.5
CB-RS Ours
72.8
SinSiamtrWSAM
75.5
Note that in 421 the models pre-trained with the long-tailed dataset, while fine-tuned with the balanced in-domain dataset. However, it is hard to achieve a balanced in-domain version of a long-tailed tailed dataset in real-world scenarios. So we the long-tailed dataset with balancing method including class-balanced re-sampling (CB-RS), and re-sampling with context-shifi augmentation. The results show that method is superior CB-RS.
D.7
Combination with the logit adjustment
Since our work aims to study the effectiveness of re-sampling long-tail learning, we use the Class-Balanced Re-Sampling (CB-RS) in our method. We consider combining our method with other re-balancing methods, such Logit Adjustment (LA) [66] Specifically, we change the class- balanced remaalnn to the uniform sampling while adopting context-shift augmentation. Moreover, we consider combining blaasballaccdd re-sampling and LA simultaneously. The comparison results are shown in Tab. 10 The results show that our method can be combined with logit adjustment to yield higher performance. However, by applying the balanced loss and the balanced sampling, the model puts much focus tail classes and results in deterioration of overall accuracy.
18

---

Table 10: Combination with Logit Adjustment (LA). Accuracy results (%) on CIFARI00-LT with an imbalance ratio of 100 are reported.
All
Many
Med.
Few
w/ CB-RS
45.8
64.3
49.7
18.2
w/ LA
47.2
62.4
48.8
26.1
w/ both
45.0
48.7
51.0
31.4
D.8
Combination with supervised contrastive learning
Our proposed context-shipp augmentation (abbreviated as CSA) be integrated with the supervised contrastive learning method BCL [39] to further improve the generalization. In Table we report the experimental results. We conduct experiments CIFAR RO0-LT with varying imbalance ratios, showing that CSA consistently boosts the performance of BCL..
Table 11: Accuracy (%) CIFARI00-LT by integrating the proposed CSA into BCL
Imbalance Ratio
100
50
10
BCL
51.9
56.6
64.9
BCL w/ CSA
52.6
57.1
65.8
D.9 Computational cost analysis of context-shift augmentation.
The proposed context-shifi augmentation based on the widely-used dual- network. Moreover, a context-extracting module is designed to calculate Grad-CAM as well as obtain the contexts. The context-extracting module has very small computational cost, as it only needs apply the gradient backward once at the last layer of the network, and can be ignored compared to the global gradient backward for updating the whole model. Besides, does not spend too much memory space to save the extracted contexts, since the size of the context bank is set equal to the mini-bated size. In Table we report the training time cost per epoch of CIFAR100-LT using a single RTX3090, which also demonstrates that the proposed method does not lead to too much additional computational overhead.
Table 12: Training time cost per epoch CIFARIOO-LT.
w/ CE
w/ BCL
Single-Bres Branch
2.04 s
4.76 s
Dual-Branch
2.38
4.85
Ours
2.98 s
5.10
19


Reviews:
[{"review_id": "VOXWofTVn4", "reviewer": "Reviewer_jD4D", "summary": "This paper studies the effect of re-sampling for long-tail learning. It finds that re-sampling is highly related with the contexts of images. And it does not necessarily work of fail in long-tail learning. Based on these findings, a context-shift augmentation module is proposed. Experiments verify the effectiveness of the proposed module.", "questions": "More examples are encouraged to be attached in Introduction to better illustrate the found issue and the basic motivation of the paper.\n\nThe paper illustrates that the irrelevant context issue occurs for MNIST and Fashion. And for IMAGENET, the issue is not significant. So is the proposed method more suitable for those simple datasets like MNIST and Fashion? If yes, this will highly affect the application of the proposed method for real-world tasks.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper studies an important task and proposes a unique solution for the found issues of re-sampling.\n\nThe paper is well-written and easy to follow.\n\nThe paper comprehensively introduces the preliminary of the proposed method.", "weaknesses": "The concept of context and content appear a lot in the paper, but there is no formal introduction or definition for these two concepts.\n\nThe paper illustrates that the irrelevant context issue occurs for MNIST and Fashion. And for IMAGENET, the issue is not significant. So is the proposed method more suitable for those simple datasets like MNIST and Fashion? If yes, this will highly affect the application of the proposed method for real-world tasks.\n\nThe paper uses Grad-CAM to determine the foreground and background for training images. However, for tail-class images, since the model may baise to head class, the generated activation map may not be accurate, which will affect the final results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments, and are encouraged the reviewer comments that the paper is 'well-written'. We will address the concerns below.\n\n**Concern #1 (Weakness #1 & Question #1): There is no formal introduction or definition for context and content.**\n\n**Response:** Context refers to the semantically unrelated parts in the images, and content refers to the semantically related parts. We give an example in Figure (r4) in the Rebuttal PDF file. Thank you for pointing out this concern. We will revise the miswriting and standardize the definitions in the next version.\n\n**Concern #2 (Weakness #2 & Question #2): The irrelevant context issue occurs only for simple datasets like MNIST and Fashion. And for IMAGENET, the issue is not significant.**\n\n**Response:** There might be some misunderstanding about this concern. The irrelevant context issue **does not occur** for simple datasets like MNIST-LT and Fashion-LT, because training samples and target labels on these datasets are highly semantically correlated (even do not contain any backgrounds). Meanwhile, the irrelevant context issue **occurs** for real datasets like CIFAR-LT and IMAGENET-LT, because these datasets contain many semantically unrelated elements, such as backgrounds and irrelevant objects. In Table 1, both cRT (classifier re-training) and CB-RS (class-balanced re-sampling) are classical learning methods for long-tail learning. CB-RS learns well on MNIST-LT and Fashion-LT but fails on CIFAR-LT and IMAGENET-LT, demonstrating that the irrelevant context has negative impacts. So we propose a context-shift augmentation to alleviate the problem. Experimental results on CIFAR-LT and IMAGENET-LT validate the effectiveness of our proposed method.\n\n**Concern #3 (Weakness #3): The generated activation map of tail-class images may not be accurate.**\n\n**Response:** First, we generate the activation maps from the uniform sampling of the long-tailed data. In this sampling manner, most data are from head classes, and tail-class images are rarely sampled. Second, we use a threshold $\\delta$ to filter the well-learned training samples to ensure the quality of extracted backgrounds, as is mentioned in Section 3.1. Moreover, we visualize some tail-class samples with predicted probabilities higher than $\\delta$ in Figure (r3) in the Rebuttal PDF file, which shows that the model can still grap accurate activation maps for tail-class samples."}, {"review_id": "6qLcjsDNoK", "reviewer": "Reviewer_gT6X", "summary": "Long-tailed Learning has drawn great attention in recent development of deep learning due to the non-negligible challenges on tail classes during learning. The re-sampling method that should make an effect to improve the training usually fails in a lot of practices. The authors dive into the phenomenon and finds some new insights towards this problem, and then design a new method to alleviate the negative impact of re-sampling on some datasets, which shows the final promise compared with the recent SOTAs.", "questions": "\n1) How is the visualization of representation on datasets that contains more class-irrelevant contexts with or without resampling? And how is the visualization of Grad-CAM for examples with or without context-shift augmentation?\n\n2) How is the sensitivity of the hyparameter lambda in Eq. (7)? How is the performance of the proposed method applied to some recent SOTAs, i.e., their compatibility with recent SOTAs?\n\n 3) Why do not consider the SOTAs like LA or BCL, PaCo in the experimental parts? The authors can have some more explanations in choosing the baselines.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this submission can be summarized as follows,\n\n1) The authors rethinks an interesting problem about the re-sampling method, which are under-estimated or mistaken in previous studies. That is, re-sampling mainly works in the classifier layer through a two-state decoupling way. How to make the re-sampling works in a single-stage framework is an unexplored yet important challenge.\n\n2) The observation and interpretation are insightful, as two different types of datasets contracts on the final performance for re-sampling. It means actually re-sampling can help to extract the discriminative representation but just sensitive to the irrelevant contexts, i.e., overfitting to the spurious correlations in some cases. \n\n3) Based on the conjecture, the authors design a context-shift augmentation method along with re-sampling to alleviate the negative impact. Specially, they construct a context memory bank to boost the augmentation of samples in diversity and avoid overfitting the spurious correlation from backgrounds. The experimental results demonstrate the promise of the proposed method.\n\n4) The whole paper is well written, in which the motivation parts are well justified with a range of experiments to show the intuition, making the whole parts clearly understood.", "weaknesses": "\nAlthough the submission are well organized and presented, there are still some points that can be improved by following the below advices.\n\n1) In Figure 2, although it is hard to visual the representation distribution on CIFAR100-LT and ImageNet-LT, it is still possible to randomly select some sub-classes, e.g., downsample 10 classes from the super classes of CIFAR100-LT, to visualize the representation. This can shows whether on some datasets that contains many class-irrelevant contexts, the representation are not well learnt, i.e., not so discriminative, compared with that on datasets that contains few class-irrelevant contexts.\n\n2) Similarly, on basis of Figure 3, it is helpful to show with the context-shift augmentation, the method can efficiently avoid overfitting the background spurious correlations, by tracing the Grad-CAM map of some same examples.\n\n3) It will be more solid to give some interpretation from the theoretical analysis: why the re-sampling encourages the method to learn more spurious correlations compared the method without re-sampling, or why the two-stage methods reduce the overfitting of spurious correlation in re-sampling. What the benefits or lessons we get in re-sampling from some theoretical perspectives.\n\n4) It lacks of more experiments about some hyperparameters like \\lambda of Eq.(7). Especially, the hyperparameter can be tuned based on different datasets and even different classes. The authors can strengthen this parts to avoid the concerns about the hyperparameter selection or heuristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful reviews, and for noting that the observation and interpretation are 'insightful', and the whole paper is 'well written'. We will address the concerns below.\n\n**Concern #1 (Weakness #1): Visualize the representation distribution on CIFAR100-LT.**\n\n**Response:** Following your suggestion, we visualize the representation of CIFAR100-LT in Figure (r1) in the Rebuttal PDF file. Each color represents a class, with darker colors representing head classes, and lighter colors representing tail classes. Although the fine-grained colors make it difficult to distinguish some classes, it can still be seen that CB-RS learns worse representation compared with vanilla CE. Moreover, our proposed method can learn a more distinguishable representation.\n\n**Concern #2 (Weakness #2 & Question #1): Visualize the Grad-CAM for examples with context-shift augmentation.**\n\n**Response:** We follow your suggestion and visualized the Grad-CAM for examples with our proposed context-shift augmentation. We choose the same examples in Figure 3 in the main paper, and report the results in Figure (r2) in the Rebuttal PDF file. The results show that our method can alleviate the negative impact on head-class samples caused by the overfitting problem.\n\n**Concern #3 (Weakness #3): Give some interpretation from the theoretical analysis.**\n\n**Response:** Thank you for your suggestions. We conduct our research mainly from an empirical perspective since our work is the first to study the reason for overfitting caused by re-sampling. It is a good suggestion to analyze from the theoretical aspect, and we will leave this for future research.\n\n**Concern #4 (Weakness #4  & Question #2): Experiments about some hyper-parameters like $\\lambda$ of Eq.(7).**\n\n**Response:** We have done experiments about hyperparameters. We report the results in the supplementary material due to the page limit. Please refer to Section C for more details.\n\n**Concern #5 (Question #2 & Question #3): Consider more recent SOTAs.**\n\n**Response:** We have combined our method with LA and reported the results in Section C.7 in the supplementary material. Following are the experimental results.\n\n|            | Overall | Many | Medium | Few  |\n| ---------- | ------- | ---- | ------ | ---- |\n| Ours       | 45.8    | 64.3 | 49.7   | 18.2 |\n| Ours w/ LA | 47.2    | 62.4 | 48.8   | 26.1 |\n\nMoreover, our proposed Context-Shift Augmentation (abbreviated as CSA) can be integrated with the contrastive-learning-based method BCL to further improve the generalization. Following are the experimental results.\n\n| CIFAR100-LT         | ir=100 | ir=50 | ir=10 |\n| ------------------- | ------ | ----- | ----- |\n| BCL                 | 51.9   | 56.6  | 64.9  |\n| BCL w/ proposed CSA | 52.6   | 57.1  | 65.8  |"}, {"review_id": "0tbYoKRx4A", "reviewer": "Reviewer_gDP8", "summary": "This article systematically investigates the effectiveness of resampling methods in long-tailed image classification problems. Resampling is a classical and important approach to addressing class imbalance, but it may not always be effective when applied to long-tailed datasets. The paper shed light on this phenomenon and provides a systematical analysis. It discovers that re-sampling the tail data can improve the performance, but may lead to overfitting on the irrelevant contexts. To address this issue, they propose a context-shift data augmentation method. The proposed method achieves superior performance on long-tailed datasets such as CIFAR10-LT, CIFAR100-LT, and ImageNet-LT.", "questions": "1. Does the data augmentation method of enhancing the background also applied to samples of the head class?\n2. Does the proposed context-shift augmentation require additional computational cost?\n", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides a well-defined and reasonable motivation. The authors show both success and failure cases of re-sampling on long-tailed datasets, which highlights the significance of addressing this problem.\n2. This paper extensively conducts experiments and ablation studies on widely-used datasets, which demonstrates the effectiveness of their method.\n3. This paper is well-writing and easy to follow. The source code is provided in a proper way, facilitating the reproducibility of their work.\n", "weaknesses": "1. A comparison with some data augmentation methods should be included. For example, Randaugment is a widely used method, which can improve the performance on long-tailed datasets.\n2. The name of the proposed method (context-shift augmentation) should be unified, as some parts include hyphens \"-\" while others do not. Also, some parts use italics while others do not.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the comments and for mentioning that this paper has a 'well-defined' and 'reasonable' motivation. We address the concerns below.\n\n**Concern #1 (Weakness #1): Comparison with some data augmentation methods such as Randaugment.**\n\n**Response:** Thanks for your suggestion. AutoAugment [1] and Cutout [2] can be applied to further improve the performance. Following are the results. \n\n| CIFAR100-LT (ir=100) | Overall | Many | Medium | Few  |\n| -------------------- | ------- | ---- | ------ | ---- |\n| CE                   | 44.1    | 72.5 | 45.1   | 7.7  |\n| cRT                  | 49.4    | 66.7 | 50.7   | 26.3 |\n| Ours                 | 50.4    | 64.6 | 55.7   | 26.5 |\n\nMoreover, Randaugment is usually used in contrastive learning methods. Our proposed Context-Shift Augmentation (abbreviated as CSA) can be integrated with the contrastive-learning-based method BCL to further improve the generalization. The results are also shown as follows.\n\n| CIFAR100-LT         | ir=100 | ir=50 | ir=10 |\n| ------------------- | ------ | ----- | ----- |\n| BCL                 | 51.9   | 56.6  | 64.9  |\n| BCL w/ proposed CSA | 52.6   | 57.1  | 65.8  |\n\n**Concern #2 (Weakness #2): The name of the proposed method should be unified.**\n\n**Response:** Thanks for your kind suggestion. We will revise the miswriting in the next version.\n\n**Concern #3 (Question #1): Does the data augmentation method of enhancing the background also applied to samples of the head class?**\n\n**Response:** Yes, we extract backgrounds from the uniform sampling and apply it to the balanced sampling. In this manner, all samples are used for augmentation.\n\n**Concern #4 (Question #2): Does the proposed context-shift augmentation require additional computational cost?**\n\n**Response:** Our method is based on the widely-used dual-branch network. Apart from this, we add a context-extracting module to calculate Grad-CAM and get the backgrounds. This module has a very small computational cost, as it only needs to apply the gradient backward once at the last layer of the network. Compared to the global gradient backward for updating the whole model, this cost can be ignored. Also, we do not need too much space to save the extracted backgrounds, since we set the size of the context bank equal to the mini-batch size. Moreover, we calculate the time cost per epoch of CIFAR100-LT using a single RTX3090. Following are the results.\n\n| Loss          | CE     | BCL    |\n| ------------- | ------ | ------ |\n| Single-Branch | 2.04 s | 4.76 s |\n| Dual-Branch   | 2.38 s | 4.85 s |\n| Ours          | 2.98 s | 5.10 s |\n\n[1] Autoaugment: Learning augmentation strategies from data.\n\n[2] Improved regularization of convolutional neural networks with cutout.\n"}, {"review_id": "dP7DKk8X0y", "reviewer": "Reviewer_qgru", "summary": "In this paper, the authors rethink the effect of re-sampling on long-tailed learning. Specifically, this paper leverages a synthetic experiment to claim that re-sampling is sensitive to irrelevant contexts. \nTo address this problem, this paper proposes a context-shift augmentation strategy to alleviate the overfitting to the irrelevant contexts in the tail classes. In the empirical part, this paper conducts performance comparisons on CIFAR-LT and ImageNet-LT to validate the proposed method.\n", "questions": "- The construction of CMNIST-LT seems unreasonable. The color injection introduces severe spurious correlations to the tail classes as each tail class is assigned with single color and each head class is assigned with multiple colors. It is not surprising that the class-balanced re-sampling will deteriorate the performance at this case as only the spurious feature (single color) is enough to perform the classification on the tail class. Furthermore, this hand-crafted distribution on the irrelevant contexts may not exhibit on CIFAR-10/100-LT, ImageNet-LT as the diversity of the tail-class irrelevant contexts is mainly proportional to their class cardinality, but rather than collapse to the single point in the distribution. \n- The author raises the question “Can re-sampling benefit long-tail learning in the single-stage framework” in line 41. However, the proposed method still requires multi-branch learning (even multi-stage learning) to perform the resampling. It is critical to compare the computational cost of the proposed method with baseline methods.\n- This paper mainly investigates the class-balanced re-sampling. Is this re-sampling strategy optimal at the case of irrelevant contexts? More discussions or empirical comparisons are recommended.\n- Intuitively, the Grad-CAM is not reliable at the early stages of the training. It seems multiple stages (warmup) are required to ensure the training stability. \n- The experiments are only conducted on CIFAR-LT and ImageNet-LT. It would be better to conduct experiments on more real-world long-tailed benchmark Places-LT and iNaturalist.\n- In the main text of this paper, only the empirical comparison results are presented. It would benefit a lot to provide more ablation studies or further analysis of the proposed method. \n- Some supervised long-tailed learning SOTA baselines are missing in this paper:\n\n[1] Menon, A. K., Jayasumana, S., Jain, H., Veit, A., Kumar, S., & Rawat, A. S. Long-tail learning via logit adjustment. ICLR 2020.\n\n[2] Cui, J., Zhong, Z., Liu, S., Yu, B., & Jia, J. (2021). Parametric contrastive learning. ICCV 2021.\n\n- Some related data augmentation baselines are missing in this paper:\n\n[3] Chu, P., Bian, X., Liu, S., & Ling, H. (2020). Feature space augmentation for long-tailed data. ECCV 2020.\n\n[4] Li, S., Gong, K., Liu, C. H., Wang, Y., Qiao, F., & Cheng, X. (2021). Metasaug: Meta semantic augmentation for long-tailed visual recognition. CVPR 2021.\n\n[5] Zang, Y., Huang, C., & Loy, C. C. (2021). Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. ICCV 2021.\n\n[6] Zhou, Z., Yao, J., Wang, Y. F., Han, B., & Zhang, Y. (2022). Contrastive learning with boosted memorization. ICML 2022.\n\n[7] Ahn, S., Ko, J., & Yun, S. Y. (2023). CUDA: Curriculum of Data Augmentation for Long-tailed Recognition. ICLR 2023.\n", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is generally well-written and easy to follow.\n- The exploration on the intrinsic mechanism of re-sampling on long-tailed data is important and not addressed.\n- This paper claims that re-sampling is sensitive to the irrelevant context and presents a toy example to support the claim.\n", "weaknesses": "- The construction of CMNIST-LT seems an extreme case, which may not exhibit in real-world scenarios.\n- The proposed method still follows the multi-branch architecture. However, the computational cost of the proposed method is not discussed.\n- The experiments are not adequate. Some important long-tailed benchmarks or baselines are missing.\n- This paper lacks more ablation studies or further analysis to provide more insight of the proposed baseline.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the thoughtful reviews, and for the comments that the paper is 'well-written'. We will address the concerns below.\n\n**Concern #1 (Weakness #1 & Question #1): The construction of CMNIST-LT seems an extreme case and unreasonable.**\n\n**Response:** Thanks for your concern. The CMNIST-LT benchmark is inspired by the widely-used CMNIST [1-4], which is also designed to simulate biased data. Your understanding is correct since the injected color is the single factor that affects the learning results. In real-world data, the class-balanced re-sampling (CB-RS) will also deteriorate the performance if some spurious features are enough to perform the classification on the tail class. It might be not surprising, but we just intend to reveal the insights from a simple case. Moreover, we also validate this idea by conducting empirical studies on real-world datasets CIFAR-LT and IMAGENET-LT in Table 1 and Figure 3 in the main paper.\n\n**Concern #2 (Weakness #2 & Question #2): The computational cost of the proposed method is not discussed.**\n\n**Response:** Our method is based on the widely-used dual-branch network. Moreover, we add a context-extracting module to calculate Grad-CAM and get the backgrounds. This module has a very small computational cost, as it only needs to apply the gradient backward once at the last layer of the network. Compared to the global gradient backward for updating the whole model, this cost can be ignored. Also, we do not need too much space to save the extracted backgrounds, since we set the size of the context bank equal to the mini-batch size. Moreover, we calculate the time cost per epoch of CIFAR100-LT using a single RTX3090. Following are the results.\n\n| Loss          | CE     | BCL    |\n| ------------- | ------ | ------ |\n| Single-Branch | 2.04 s | 4.76 s |\n| Dual-Branch   | 2.38 s | 4.85 s |\n| Ours          | 2.98 s | 5.10 s |\n\n**Concern #3 (Weakness #3 & Questions #5&7): More long-tailed benchmarks or baselines.**\n\n**Response:** Thanks for your suggestion. We have combined our method with LA and reported the results in Section C.7 in the supplementary material. Following are the experimental results.\n\n|            | Overall | Many | Medium | Few  |\n| ---------- | ------- | ---- | ------ | ---- |\n| Ours       | 45.8    | 64.3 | 49.7   | 18.2 |\n| Ours w/ LA | 47.2    | 62.4 | 48.8   | 26.1 |\n\nMoreover, our proposed Context-Shift Augmentation (abbreviated as CSA) can be integrated with the contrastive-learning-based method BCL to further improve the generalization. Following are the experimental results.\n\n| CIFAR100-LT         | ir=100 | ir=50 | ir=10 |\n| ------------------- | ------ | ----- | ----- |\n| BCL                 | 51.9   | 56.6  | 64.9  |\n| BCL w/ proposed CSA | 52.6   | 57.1  | 65.8  |\n\nWe conduct experiments on Places-LT and report the results below. The results show that our method outperforms cRT.\n\n|      | Overall | Many | Medium | Few  |\n| ---- | ------- | ---- | ------ | ---- |\n| cRT  | 36.7    | 42.0 | 37.6   | 24.9 |\n| Ours | 37.2    | 42.3 | 38.7   | 22.4 |\n\n**Concern #4 (Weakness #4 & Question #6): More ablation studies or further analysis.**\n\n**Response:** We have done ablation studies and further analysis. The results are reported in the supplementary material due to the page limit. Please refer to Section C for more details.\n\n**Concern #5 (Question #3): Is this re-sampling strategy optimal at the case of irrelevant contexts?**\n\n**Response:** There might be some misunderstanding. It is not that using re-sampling can solve irrelevant contexts, but that directly using re-sampling can cause overfitting to the irrelevant contexts. So we propose a context-shift augmentation method to alleviate this problem.\n\n**Concern #6 (Question #4): The Grad-CAM is not reliable at the early stages of the training. Multiple stages (warmup) are required to ensure the training stability.**\n\n**Response:** We have multiple strategies to ensure training stability. First, we have a warmup stage of 10 epochs for all experiments. The implementation details can be referred from Section B in the supplementary material. Second, we use a threshold $\\delta$ to filter only the well-learned training samples to ensure the quality of extracted backgrounds, as is mentioned in Section 3.1. The validation experiment of $\\delta$ can be found in Section C.1 in the supplementary material.\n\n**Concern #7 (Question #8): Some related data augmentation baselines are missing in this paper.**\n\n**Response:** We have already compared and discussed some data augmentation baselines such as FSA [5] in Sections 3.3&4.3. The accuracy on IMAGENET-LT is 42.7 (ours) vs. 35.2 (FSA). Besides, we will add the other given literature [6-9] in the next version. Thanks for your notation.\n\n[1] Invariant Risk Minimization.\n\n[2] Learning Not to Learn: Training Deep Neural Networks with Biased Data.\n\n[3] REPAIR: Removing Representation Bias by Dataset Resampling.\n\n[4] Learning from Failure: Training Debiased Classifier from Biased Classifier.\n\n[5] Chu, P., Bian, X., Liu, S., & Ling, H. (2020). Feature space augmentation for long-tailed data. ECCV 2020.\n\n[6] Li, S., Gong, K., Liu, C. H., Wang, Y., Qiao, F., & Cheng, X. (2021). Metasaug: Meta semantic augmentation for long-tailed visual recognition. CVPR 2021.\n\n[7] Zang, Y., Huang, C., & Loy, C. C. (2021). Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. ICCV 2021.\n\n[8] Zhou, Z., Yao, J., Wang, Y. F., Han, B., & Zhang, Y. (2022). Contrastive learning with boosted memorization. ICML 2022.\n\n[9] Ahn, S., Ko, J., & Yun, S. Y. (2023). CUDA: Curriculum of Data Augmentation for Long-tailed Recognition. ICLR 2023."}, {"review_id": "o1uECl0ico", "reviewer": "Reviewer_J7TL", "summary": "The authors suggest looking closer at the resampling strategy for long-tail recognition. The motivation comes from the observation that in recent works resampling brings rather negative than positive effects. Based on MNIST experiments, the authors demonstrate that context (background) imposes spurious correlations during training that prevents generalization on tail classes. To deal with such correlations, the authors suggest a context shift augmentation module to generate diverse training images for the tail classes. Experimental results show improvement over listed by the authors baselines.", "questions": "Q1: Would it be possible to conduct a similar experiment on more complex data with a more in-depth analysis of the correlations? It would significantly help to improve the paper.\n\nQ2: Is there any adaptation/analysis of the spatial structure of the images, e.g. if the object is not in the center of the image or there are multiple objects of interest? \n\nQ3: in Eq. (7), the mask M relates to image x_double_hat but is unrelated to x_tilde. In this case not only background is affected by the augmentation. I didn’t find the discussion in this regard. \n\nQ4: Could the authors discuss more in detail the difference to [28] and provide quantitative confirmation of the improvements based purely on the proposed novelty?", "limitations": "The authors mention that their method does not achieve comparable performance to papers presented at CVPR/ICCV21. No further discussion is added. \nOther limitation in regard medical images and autonomous driving is rather general than specific to the method.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "S1: The authors suggest an interesting direction of analysis, namely why re-sampling fails to help generalization of the tail classes. The study is performed on the toy dataset with the aim to demonstrate that spurious correlations change drastically the predictions.\n\nS2: The overall structure and the storyline are nice; it is easy to follow the method section", "weaknesses": "W1: The synthetic experiment does not support the statements of the authors. The claim that spurious correlations hinder tail class accuracy is not presented in Fig.4 (a) on CMNIST-LT. Namely, starting from class 3, all the accuracies are jumping around independently of the number of samples in the class and the sampling strategy. “Overfit to the irrelevant context from the over-sampled tail data” is also not observed. Moreover, the generated dataset is not visualized. Further, the statement on L162-L165 sounds reasonable; however, I find the provided arguments not convincing and rather unrelated. \n\nW2: the authors claim to improve generalization on the tail classes; however, based on Table 3 the improvement comes mostly from “many” and “med” classes. \n\nW3: the tables overall are misleading. The authors simply do not include multiple baselines that outperform their method such as \n- Long-Tail Learning via Logit Adjustment  \n- Parametric Contrastive Learning   \n- Balanced Contrastive Learning for Long-Tailed Visual Recognition  \n\nW4: The authors briefly discuss differences to [28], but it looks like the novelty is rather limited wrt this method. \n\nW5: the presentation of the paper can be improved. Some sentences are hard to parse and connect to the previous context without reading further, e.g. L10, L44, L51-52. The caption of Fig.1 does not help to understand Fig.1, related text in the intro also is not helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful reviews, and for commenting that the overall structure and the storyline are 'nice'. We will address the concerns below.\n\n**Concern #1 (Weakness #1 & Question #1): The synthetic experiment does not support the statements. “Overfit to the irrelevant context from the over-sampled tail data” is also not observed. Moreover, the generated dataset is not visualized.**\n\n**Response:** Thank you for the concerns. There might be some misunderstandings.\n\n- The overfitting problem occurs on the tail classes, but will affect the performance of **all classes, not only the tail classes**. The reason is that re-sampling the tail-class samples will lead to overfitting to the irrelevant contexts, thus inevitably leading to misclassification from the head to the tail classes. Therefore, both the head and the tail classes in Fig 4(a) have a performance decrease, and **the main decrease comes from the head classes**. This phenomenon is also shown in Figure 3 in the main paper, where the head-class feature is affected by CB-RS more obviously. Moreover, we visualize the representation of CIFAR100-LT in Figure (r1) in the Rebuttal PDF file, which shows that CB-RS learns worse representation than CE on all classes.\n- “Overfit to the irrelevant context from the over-sampled tail data” is hard to be observed. However, the injected color is the single factor that lead to opposite results on MNIST-LT and CMNIST-LT, thus demonstrating the effects of the irrelevant contexts. Moreover, the training and test accuracy can also demonstrate the overfitting of CB-RS, as is shown below.\n\n  |       | training acc | test acc |\n  | ----- | ------------ | -------- |\n  | CE    | 60.75        | 39.07    |\n  | CB-RS | 69.40        | 31.40    |\n\n- Moreover, We give a brief illustration of the generated dataset in Figure (r5) in the Rebuttal PDF file.\n\n**Concern #2 (Weakness #2): Based on Table 3 the improvement comes mostly from “many” and “med” classes.**\n\n**Response:** Our empirical study shows that re-sampling can leads to the overfitting of irrelevant features, thus affecting the performance of head classes (See Figure 3 and explanation in Section 2.2.2). Therefore, solving this problem can alleviate the negative impact on head classes, and improve the corresponding performance. There is a trade-off between head- and tail-class performance, and our method achieves a higher overall performance, which demonstrates the generalization abilities among all classes.\n\n**Concern #3 (Weakness #3): Include more baselines.**\n\n**Response:** Thanks for your suggestion. We have combined our method with LA and reported the results in Section C.7 in the supplementary material. Following are the experimental results.\n\n|            | Overall | Many | Medium | Few  |\n| ---------- | ------- | ---- | ------ | ---- |\n| Ours       | 45.8    | 64.3 | 49.7   | 18.2 |\n| Ours w/ LA | 47.2    | 62.4 | 48.8   | 26.1 |\n\nMoreover, our proposed Context-Shift Augmentation (abbreviated as CSA) can be integrated with Contrastive-Learning based method BCL to further improve the generalization. Following are the experimental results.\n\n| CIFAR100-LT         | ir=100 | ir=50 | ir=10 |\n| ------------------- | ------ | ----- | ----- |\n| BCL                 | 51.9   | 56.6  | 64.9  |\n| BCL w/ proposed CSA | 52.6   | 57.1  | 65.8  |\n\n**Concern #4 (Weakness #4 & Question #4): The novelty is rather limited wrt FSA.**\n\n**Response:** Our work is the first that conduct systematical empirical studies to investigate the reasons behind the success/failure of re-sampling. Moreover, we propose a context-shift augmentation method. Compared to FSA, our method directly applies data augmentation to the original image, which is closer to the well-known mixup [1] method.\n\n**Concern #5 (Weakness #5): The presentation of the paper can be improved.**\n\n**Response:** Thank you for the suggestion. We will improve the presentation in the next version.\n\n**Concern #6 (Question #2): Is there any adaptation/analysis of the spatial structure of the images, e.g. if the object is not in the center of the image or there are multiple objects of interest?**\n\n**Response:** According to the paper [2], the bounding boxes of objects mainly appear in the center. Moreover, no matter whether the object is in the center of the image, we can detect the location of the semantically related parts by using Grad-CAM. And our proposed method will not change the semantical information of the augmented images regardless of the spatial structure of the images.\n\n**Concern #7 (Question #3): In Eq. (7), the mask M relates to image x_double_hat but is unrelated to x_tilde.**\n\n**Response:** The original mixup [1] method also randomly mixes the unrelated two images. The other mixup-based methods, such as CutMix [3], Remix [4], and CMO [5], also mix unrelated images. Therefore, we follow these works and mix unrelated images in our method.\n\n[1] Mixup: Beyond empirical risk minimization.\n\n[2] ImageNet: A large-scale hierarchical image database.\n\n[3] Cutmix: Regularization strategy to train strong classifiers with localizable features\n\n[4] Remix: Rebalanced mixup. \n\n[5] The majority can help the minority: Context-rich minority oversampling for long-tailed classification."}]

Author rebuttal:
Dear Reviewers,

We greatly appreciate all of you for your thoughtful comments and valuable suggestions. These are very helpful for improving our paper. We have carefully referred to the concerns and written the response. In addition to the text responses, **we also report some figure results in the PDF file**. We hope the responses would meet your requirements.

Best Regards,

Authors

Meta-review: This paper focuses on long-tailed learning, which has drawn great attention in recent development of deep learning due to the non-negligible challenges on tail classes during learning. The re-sampling method that should make an effect to improve the training usually fails in a lot of practices. The authors dive into the phenomenon and finds some new insights towards this problem, and then design a new method to alleviate the negative impact of re-sampling on some datasets, which shows the final promise compared with the recent SOTAs.

Specifically, the authors rethinks an interesting problem about the re-sampling method, which are under-estimated or mistaken in previous studies. That is, re-sampling mainly works in the classifier layer through a two-state decoupling way. How to make the re-sampling works in a single-stage framework is an unexplored yet important challenge. The observation and interpretation are insightful, as two different types of datasets contracts on the final performance for re-sampling. It means actually re-sampling can help to extract the discriminative representation but just sensitive to the irrelevant contexts. The authors design a context-shift augmentation method along with re-sampling to alleviate the negative impact. They construct a context memory bank to boost the augmentation of samples in diversity and avoid overfitting the spurious correlation from backgrounds. The experimental results demonstrate the promise of the proposed method. The whole paper is well written, in which the motivation parts are well justified with a range of experiments to show the intuition, making the whole parts clearly understood.

Overall, the clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (poster)