Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning
Authors: Binwu Wang
Pengkun Wang
Wei Xu
Xu Wang
Yudong Zhang
Kun Wang
Yang Wang
Venue: ICLR 2024 poster (2024)
Keywords: Long-tailed learning
data augmentation
imbalance
fairness
Primary area: general machine learning (i.e., none of the above)
TL;DR: We find traditional augmentation strategies in long-tailed learning lead to an extrinsic augmentation-wise imbalance, so we propose a Dynamic Optional Data Augmentation to overcoming this intertwined imbalance.

Abstract: Real-world tasks are universally associated with training samples that exhibit a long-tailed class distribution, and traditional deep learning models are not suitable for fitting this distribution, thus resulting in a biased trained model. To surmount this dilemma, massive deep long-tailed learning studies have been proposed to achieve inter-class fairness models by designing sophisticated sampling strategies or improving existing model structures and loss functions. Habitually, these studies tend to apply data augmentation strategies to improve the generalization performance of their models. However, this augmentation strategy applied to balanced distributions may not be the best option for long-tailed distributions. For a profound understanding of data augmentation, we first theoretically analyze the gains of traditional augmentation strategies in long-tailed learning, and observe that augmentation methods cause the long-tailed distribution to be imbalanced again, resulting in an intertwined imbalance: inherent data-wise imbalance and extrinsic augmentation-wise imbalance, i.e., two 'birds' co-exist in long-tailed learning. Motivated by this observation, we propose an adaptive Dynamic Optional Data Augmentation (DODA) to address this intertwined imbalance, i.e., one 'stone' simultaneously 'kills' two 'birds', which allows each class to choose appropriate augmentation methods by maintaining a corresponding augmentation probability distribution for each class during training. Extensive experiments across mainstream long-tailed recognition benchmarks (e.g., CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018) prove the effectiveness and flexibility of the DODA in overcoming the intertwined imbalance.

Published as a conference paper at ICLR 2024
KILL TWo BIRDS WITH ONE STONE: RETHINKING DATA AUGMENTATION FOR DEEP LONG- TAILED LEARNING
Binwu Wang", Pengkun Wang?.a, Wei Xu², Xu Wang".,, Yudong Zhang",
Kun Wang?, Yang Wang
1 Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China, Hefei, China
University of Science and Technology of China (USTC), Hefei, China
Suuhoou Institute for Advanced Research, USTC, Suzhou, China wbol 995@mail wsw.ustccc.. edu. en, pengkuněustc edu. en weixu1223@maiz usto. edu. cn, wx309fustc.eduu edu. cn (zyd2020, wk52052 @ma ustc..
uut.suucee. eec.c.edd.ccn angyan@ustc.edu.co on
ABSTRACT
world tasks are universally associated with training samples that exhibit a long-tailed class distribution, and traditional deep learning models are not suitable for fitting this distribution, thus resulting in a biased trained model. To surmount this dilemma, massive deep long-tailed learning studies have been proposed achieve inter-class fairness models by designing sophisticated sampling strate- gies improving existing model structures and loss functions. Habitually, these studies tend to apply data augmentation strategies to improve the generalization performance of their models. However, this augmentation strategy applied balanced distributions may not be the best option for long-taile long-tailed distributions. For a profound understanding data augmentation, we first theoretically analyze the gains of traditional augmentation strategies in long-tailed learning, and observe that augmentation methods cause the long-tailed distribution to be imbalanced again, resulting in intertwined imbalance: inherent data-wise imbalance and extrinsic augmentation-we learning. Motivated by this observation, propose adaptive Dynamic Op- tional Data Augmentation (DODA) to address this intertwined imbalance, i.e., one 'stone' simultaneously "kills" two 'birds', which allows each class choose appropriate augmentation methods by maintaining corresponding augmentation probability distribution for each class during training. Extensive experiments across mainstream long-tailed recognition benchmarks (c.g.. CIFAR- 100-LT, ImageNet- ITT, and iNaturalist 2018 8) prove the effectiveness and flexibility of the DODA overcoming the intertwined imbalance.
imbalance, i.c., two 'birds' co-exist in long tailed
INTRODUCTION
With the maturity of deep learning LeCun et al. (015)), massive deep models show extraordinary performance large-scale curated datasets (e.g., ImageNet Russakovsky et al. (2015) and CIFAR- Cao et al. (2019)). However, balanced artificial datasets do not conform to the data distribution (e.g., class imbalance) of real-world applications. Once facing imbalanced datasets, the performance of the deep models trained by the common practice of empirical risk minimization Vapnik (1911) will decrease significantly. e.g.. the model can be easily biased towards majority classes.
Recently, massive deep long-tailed lcarning studies have been proposed to surmount the class imbalance problem. The most intuitive and mainstream paradigm is class re-balancing, which balances the training sample numbers or weights of different classes during model training by re- sampling Kang et al. (2020): Ren et al. (2020p: Wang et al.) (2020); Dia et al. (2023) or cost-sensitive
Proff Yang Wang and Pengkun Wang the corresponding authors.

---

Published as a conference paper at ICLR 2024
learning Lin et al. (2017) Cui et al. (2019) Tan et al. 2020);; Guo al. (2022) Guo & Li (022))
Although the average performance is improved. class rebalancing methods cannot essentially handle the issue of lacking information, particularly on tail classes due to limited data amount Zhang et al. (2021b). To break through this restraint, some studies seek to transfer the knowledge from head classes enhance model training on tail classes Yin et al. (019)) Kim et al. (2020))
Another flexible line of research is to directly apply data augmentation methods to enhance the quantity and quality of training samples from the perspectives of data and representation. For example, FASA Zang et al. 202T) proposed to generate class-wise representations based on a Gaussian prior to augment the under-represented tail classes. Remix CCou et al. (2020) and VideoLT Zhang et al.) (2021a) introduced a re-balanced mixup method to particularly enhance tail classes. However, simply using existing inss-independeet augmentation strategies for improving long-tailed learning is unfavorable, since they may further increase imbalance considering head classes have more samples and would be augmented more Zhang et al. (202Tb). Considering this, as pioneer, FSR Wang et al.) (2023) first proposed an adaptive augmentation to rebalance the potential temporal feature space from the data perspective. CUDA Ahn et al. (2023) further proposed simple algorithm to find the proper class-wise augmentation strength through curriculum learning Hacohen & Weinshall (211)) Despite extensive recent studies, an unresolved problem is whether it is optimal to utilize the same kind of augmentation strategy on all classes for long-tailed learning.
With this question, we first theoreti- cally analyze the gains of traditional augmentation strategies in long-tailed learning. Inspired by Balestriero et al.
Sacrificed
Number of training samples
(2022). as shown in Figure D we ex- plain that data augmentation (DA) is
Aug-
not always beneficial to loig-tailed learning, i.e,, DA will potentially sac- rifice certain classes (especially tail classes) while improving the model performance. This means that two 'birds' co-exist in long-tailed learning:
Sacrificed
Aug.
inherent data-wise -wie imbalance and ex- trinsic augmentation-win wise imbalance. The former is the inherent property of real-world data distributions, while the latter is the side effect caused by DA when improving model general- ization. We also conduct extensive experiments on CIFAR- 100-00 with imbalance ratios (IR) of 50 and 100 to confirm the explanation. From the results shown in Figure we can observe that DA caused significant reduction in performance for some classes (e.g., most tail classes), which runs counter to the purpose of long-tailed tailed learning.
Sorted class index
Figure 1: Motivation of DODA. In deep long-tailed learning. traditional data augmentation can significantly improve the average performance of the model, but it will potentially sacrifice certain classes (i.e., red box), especially tail classes.
To this end, we propose an adaptive 'stone" called Dynamic Optional Data Augmentation (DODA) to 'kill' this intertwined imbalance, which allows each class to choose appropriate augmentation methods during training. Specifically, to avoid the sacrifice caused by class-independent DA, we maintain a "preference list' for each class (i.e., a probability distribution of DAs being selected) during training, which is the basis for each class to choose applied augmentation methods. Then, to make this 'list' more precise, we adjust the corresponding probability distribution according the positive sample size of each class. In this way, this 'list' will be dynamically corrected during training, thus each class will choose the most beneficial augmentation method avoid being sacrificed, thereby reducing the overall sacrifice probability. We integrate DODA with various long-tailed learning methods and prove that DODA can significantly improve the model performance and have high flexibility. Furthermore, we conduct several experiments to compare DODA with existing DAs for long-tailed learning and show that DODA achieves state-of-the-arr performance.
Our contributions in this paper are summarized as follows:
2

---

Published as a conference paper at ICLR 2024
New problem and insight: for the first time, theoretically analyze the gains of traditional augmentation strategies in long-tailed learning: DA will potentially sacrifice certain classes (especially tail classes) while improving the model performance, thus we need to discreetly handle the inherent data-wise imbalance and extrinsic augmentation-
imbalance.
New advisable augmentation: to 'kill' these two 'birds', we propose an adaptive 'stone' called Dynamic Optional Data Augmentation (DODA) to allow each class choose appropriate augmentation methods during training.
Compelling empirical results: DODA achieves the state-of-the-art performance across main- stream long-tailed benchmarks including CIFAR-IO0-LT, ImageNet-LT, and iNaturalist 2018.
RETHINKING DA: IS DA ALWAYS BENEFICIAL. TO LONG-TAILED LEARNING?
Applying DA in long-tailed learning has been empirically proven to significantly improve the average accuracy of the deep model. This overall improvement encouraging, but also prompts us to think about how DA brings gains the problem. In this section, we will analyze this further.
2.1 MOTIVATION: DA IN LONG-TAILED LEARNING 'HYPOCRITICAL'
Preliminary. Class-independent DA is the mainstream strategy in long-tailed learning, which is simple but effective (i.e., average improvement). However, recent work has found that DA can result in unfair model complexity control across different classes, leading to the deep model that achieve an overall accuracy improvement but perform poorly on some classes. This phenomenon reflects the fact that DA actually produces hypocritical improvement. In a
setting, 2022) explains this from the perspective of level-set. Formally, given a training (xx)) our goal is to learn a deep model fo to approximate the ideal model f closely as possible. To improve the generalization performance of the model, DA strategy (-)) will be habitually applied to æ, which is a combination of one or more augmentation methods. During this approximation, the model that we intuitively consider to be ideal may already be biased.
Balestriero et al.
dataset D = [xx; y/ 11
Theorem 1. Augmented samples produced by O(-) do not respect the level-set of s When we approximate the ideal model f by minimizing the training loss (i.e., 0 training error), the latter tends to zero, while the former is greater than zero due to the augmented samples deviate from the level set of ft. In this case, the trained model fo biased compared the ideal model f"
Σ Elly- - (f (DDD)D
>0
Elly fo(O(z))11 = 0 bias (2
Σ
(I)
Furthermore, we rethink the problem from the perspective of long-tailed learning. For each class in the long-tailed distribution dataset, such as class c, we also hope that the trained model fo can successfully predict the labels of all samples of this class. Therefore, fo also has the above bias in the class dimension. This explanation is formalized by the following theorem.
Theorem 2. Under the long-tailed distribution, minimizing the training loss (i.e., 0 training error) is equivalent to minimizing the training loss for each class. For class e and augmented samples {(O(z), y)ly c. (O(x), y) E D}, the ideal trained model can minimize the training loss of class c, but when ((x)) deviates from the level-set of J", DA will cause irreducible class-wise bias in So-
- (O OO
Ellly fo = = class-wise bias (2.))EDly=e
Σ
Theorem 2 states that for long-tailed distributions, DA cannot guarantee that it is label-preserving on all classes, which leads to some classes being sacrificed (i.e., seriously deviating from the level-set) after DA, resulting in irreducible class-wise bias. The detailed proof is in Appendix A. Based this theorem, we observed the class-wise bias caused by various DAs under two imbalance rate settings (IR (50, 100)) of LLFAR-100-L3. including simple augmentation (Cutout), flexible augmentation
3

---

Published as a conference paper at ICLR 2024
CE + Cutout
Cb + CUDA
CE CUDA Cutout
IR-50
IR-
AccCCE
Acc@CE+Aug.
Figure 2: Impact of class-independent long-tailed DAs on classification accuracy on 1IFAR-I00-LT.
(CUDA), and hybrid augmentation (Cutout with CUDA). From the experimental results shown in F we analyze the sacrifice rate (SR) under each setting and make the following findings: (1) All three types of DA can improve the average model performance (i.e., average accuracy); (2) The improvement in average performance inevitably sacrifices some classes, whether they are head or tail classes; (3) The phenomenon of sacrificing classes is especially evident on tail classes.
The experimental results are consistent with the statement of Theooem 2 which proves that the tradi- tional class iniseeindeeendent DA is "hypocritical" and will potentially sacrifice some classes (especially tail classes) when improving the average performance of the long-tailes learning model, which runs counter to the purpose of long-tailed learning.
2.2 DA FAVORS LONG-TAILED LEARNING THROUGH 'BULLYING'
class-indes indeeededden DA is hypocritical, and it shows its effectiveness by pleasing the 'strong', while tail classes are more likely to be regarded as the bullied 'weak'. To explain this phenomenon, we analyze the data distribution shift of different classes before and after DA from the perspective of feature space. In high-dimensional space, samples of the same class usually closer to each other and form approximate clusters due to their inherent similarities. When applying the same data augmentation, the features in high- nigh-dimensional space undergo shifts that are consistent across different classes. To intuitivell understand the impact of data augmentation on features of different classes, we approximate the high-dimensional space as regular hyperspace and use the variation in the distribution span to represent the diversity improvement and high-dimensional space expansion caused by data augmentation. Intuitively, we approximate the feature space of classes to two-dimensional space to illustrate this. And we also provide a theoretical explanation in high- high-dimensional feature space in Appendix A.
As mentioned in Section 2.1
Definition 1 (Distribution Span). Approximating the distribution of each class in the training set as circle a two-dimensional feature space, the distribution center of class e defined as (%,, W.) and the distribution radius is R.,. The distribution spun S. can be expressed as follows:
S. (% x=)²- (Y Y R.
For intuitive understanding, we assume that the data distribution of head class с₁ and tail class ca are S.,, and S.,, respectively. We uniformly apply the same DA O(-) to the whole training set. Furthermore, the distribution span after DA can be defined as follows:
S., + (X X. )2 (Y Yo, =Re, A.A)
S. = (X
(Y
(Re, + A2,)2
(5)
Here, Ao, and A., represent the increase in distribution radius within each class after DA. For the same augmentation method, A., A..- Based on the increase in distribution radius, we analyze the sensitivity of each class to this augmentation.
Theorem 3.
The augmented data distribution is a combination of the original data distribution (base space) and the expanded data distribution (marginal space). The augmentation sensitivity &b can be defined as the ratio of the marginal space to the base space. Under the same DA, tail classes have higher uugmentation sensitivity, indicating that tail classes are more sensitive to the marginal space.
Rea
Res
R., 2
they - -ee. -20g
+ Acp.
TR.n Rei
Rey.
2IR
4

---

Published as a conference paper at ICLR 2024
The main idea of the proof. provided in Appendix A, is to demonstrate that tail classes more likely to have label non- preservation compared to head classes after DA, resulting in a greater deviation between the level-set learned by fo and the level-set of f tail classes.
In summary, DA in long-tailed learning is hypocritical and bullying. as it potentially sacrifices certain classes, especially tail classes, while boosting the average performance. Therefore, not only need to deal with the inherent data-wise imbalance caused by the traditional sample distribution but also pay attention the side effects of DA, the extrinsic augmentation-wics imbalance. It is frustrating that the existing 'stones' cannot kill both 'birds', so we need to consider how to achieve a simple and effective DA to address this intertwined imbalance.
DYNAMIC OPTIONAL DATA AUGMENTATION: AN ADVISABLE STRATEGY
In this section, based on the aforementioned theoretical analysis, propose an advisable 'stone' called Dynamic Optional Data Augmentation (DODA) to address the intertwined imbalance. The core philosophy of DODA is to allow each class to choose appropriate augmentation methods during training by maintaining a 'preference list' for each class. The detailed algorithm overview DODA is shown in Figure
3.1 CLASS-WIS PREFERENCE LIST CONSTRUCTION
Firstly, we assume the exis- tence of K optional DAs and define the index of each DA as 21, ..., K}. These DAs are task-specific, such as Gaus- sian blur, rotation, and hori- zontal flip. Each DA (²(,):
o (
Training
Validation
Result
X1
X1
(x)
Sampling DA
List Maintenance
Update the
Fd + Rd has its prede- fined augmentation function and strength. For traditional strategies, pre-determine the selected DAs and use them for all classes during train- ing. However, this class- independent DA may sacrifice certain classes. Therefore, maintain an augmentation preference list Q. Ve E {I,.... C} for each class during training. This list records the optional DAs for each class and the probability of euch DA being selected.
Poto
Preference List after epoch
Figure 3: Overview of DODA. DODA allows each class to choose appropriate augmentation methods during training by maintaining 'preference list' for each class.
Definition 2 (Probability of Each DA Being Selected). The augmentation preference list Qe for class records the selection hierarchy 2e VE (1,..., K) of the K optional DAs. Based on the selection hierarchy, we define the probability of each DA O²()) being selected as follows:
10%
Pot(-) Ejee. 151'
where Qc =Q2, De.... 2K!
Based on this preference list, we perform DA on the original dataset and define the augmented dataset as D = { y: [[r,, E D). To preserve the knowledge of the original dataset D, we define the augmentation probability as Paug < 1, so each sample has a probability of Pang of being augmented. When performing DA, randomly decide whether to augment the current sample with Paug. Therefore, the augmented sample can be represented as follows:
O (x:), with prob. Paug Dt
f
(8)
otherwise
3.2 CLASS-WISE PREFERENCE LIST MAINTENANCE
On the augmented dataset D, we apply tailed learning algorithm F to learn the desired deep model fo. ie.. F(fo. D). It is worth mentioning that the choice of is flexible, for example, we test various long-tailed learning algorithms in subsequent experiments.
5

---

Published as a conference paper at ICLR 2024
During the early stages of training, the augmentation preference list set for one class may be inaccurate, meaning that there is still a high probability to select DAs that are detrimental to this class. Therefore, continually updating this list during training is necessary. The core philosophy of updating is to up-weight the positive DAs and down-weight the negative DAs. After each training epoch, we count the number of correctly predicted samples in each class, i.e., the positive sample size for each class. Formally, for class c, its positive sample size is defined as follows:
PiPOS
Σ (ED
Definition 3 (Augmentation Dominance). For the dataset DD and class DA Ohi is said to dominate DA Oka on class if and only V
V where V 2/2
-(er.))Dll 1 (10(0h (x.))=c)
and VPDe
(e 1 (1())
Theorem 4. The level-set bias 8(Q, P) can be defined the degree of distributional deviation between the level-sets Q and P. Suppose the level-set of the model fe trained on the original dataset P. and the level-sets of the models fo. and foa learned using DA O and DA Ohe respectively, are Q and Q²² If the augmentation dominance of O is higher than that of O then the bias of Qhi from P is smaller than that of Q*2 from P, i.e. 8(Q"1, < 8(Q"2,)) P).
Theorem states that more dominant DA tends to avoid severe distribution bias. The detailed proof is in Appendix A. Therefore, to explore optional DA, record the positive sample size temp, from the previous epoch for class e to facilitate observation of whether the DA used in this epoch brings benefits to the class.
Algorithm 1: DODA
Input: Dataset D = (ara, yr) algorithm F. training epoch E. number of optional ang. K, aug. probability paug- Output: trained model fo- Initialize: Weight of each DA for each class e 1 V. E {1,.... KJ, Ve E (1.....))
for S E'do
for c S do
Randomly select a DA (-) for class e according to the weight distribution Q..
Based on the positive sample sizes from the current epoch and the previous epoch, we up- date the augmentation preference list for each class. For class с and DA 0% used in the cur- rent epoch, if the positive sample size from the current epoch is greater than that from the previ- ous epoch, we consider this DA beneficial for class and thus we up- weight this DA that has a positive impact. Conversely, if this DA has a negative impact on class c, down-weight it.
Pot P. 17T
where Q. 10 e @%
end
Generate D = ((1,,) (.))(s D) where
(o)))))
with prob. Pang otherwise.
=
Run LTI. algorithm using D, i... F(fe, D). for c S C do Compute positive sample
of
By dynamically maintaining the augmentation preference lists, DODA achieves adaptive class- dependent augmentation, which is not 'hypo- critical' and not 'bullying', allowing each class (especially the tail classes) to choose appropri- ate augmentation methods and avoid being po- tentially sacrificed during training. To better illustrate the execution process of DODA, we provide detailed execution flow in Algorithm I
Poo of
ELE (f((())==))
if PoR *f
temp, then @ - Q +1
Up weight the positive
elif Po temp, then e + e
else // Down-weight the negative DA
if o > 1 then o + o
else Q temp,
1
The code available https://qithut
end
com ccm/pon httm/ponkkun/CCooo for-DODA
end
4 EXPERIMENTS
In this section, we conduct empirical evaluations on multiple mainstream datasets to demonstrate the superiority of the proposed DODA in long-tailed learning.
4.1 EXPERIMENTAL SETTINGS
Datasets. To ensure a fair comparison, we conducted experiments on three mainstream long-tailed image recognition benchmarks including (IFAR-100-L33 Cao et al. (2019), ImageNet-I.TLiu et al.

---

Published as a conference paper at ICLR 2024
Table Accuracy (%) on CIFAR-100-LT dataset (Imbalance ratio= 10, 50, 100}) wtih -fate-of-the- art methods. Blod indicates the best performance while underline indicates the second best. (+) and (-) indicate the the relative gain. We report the average results of three random trials.
Mathod
IR-10
IR-50
IR-100
Head
Medium
Tail
Hlead
Medium
Tril
All
Head
Medium
Tuil
CE 2016
40.3
56.5 (+0.0)
63.9
36.2
15.2
43.8
(+Q.0)
65.5
38.1 (+0.0
CETO
2022
62.0
45.0
(+.77)
68.6
27.5
18.7
45.0
70.1
(+.77
(9.5(43300)
18.4
13.7
70.7
(2.00
CMO
2023
43.0
(8.2(42.2)
67.6
39.2
17.6
71.1
(4.55
(6.5(+330)
41.2
19.7
CE-DRW/
1019
58.2(+0:0)
19.0
(-0.0)
(4000
CF-DRW-
(4.5()3.5)
DRW -DODA
63.6
56.2
(+.3
47.4
38.9
52.5 (+7.5)
60.2
(+6.))
1DAM-DRW
62.7
(7.50+000)
41:2
47.2 (-0.0)
60.8
(402)
LDAM
45.2
57.9 (+0.4)
46.2
26.4
(+3.6)
47.1 (+3.)
LDAM DODA
46.3
27.5
(+3.3)
uS|Ren R0A0
(+0.0)
41.3
(+0.0)
CU=CC
(1.5(+3.4)
18.4
37.3
(+4.8)
DODA
56.8
(1.8
51.2
41.5
$4.0
(46.1)
63.1
31.2
48.7 (45.9)
RIDE experts (Wnn
(1.1
47-7
31.8
(+0.0)
65.3
48.5
60:1 (-1.0)
47.0
33.4
53.1
(40.9)
67.9
(0.5)
RIDE exgenty Aln
2003
47.2
5939 (-1.2)
46.1
29.3
(-0.1)
68.7
(9.6(+2.))
(3 exparte) DXDA
65.7
49.9
60.8 (40.3)
46.5
33.6
(40.4)
68.4
51.1
27.8
50.2 (42.7)
51.8
(400))
43.1
34.3
(+0.0)
63.1
(400))
RCT Ponrer
65.3
56.6
(4.6(43..)
47.4
79.4
62.7 (43.6)
64.7
49.7
70.1
(448))
BCI.
65.6
62.7 (43.8)
64.0
48.0
40.6
53.6
(44.5)
66.0
(1.0(468)
(2019), and iNaturalist 2018 Van Horn et al. (2018) CIFAR-100-LT and ImageNet-LL are long- tailed versions artificially truncated from the original balanced datasets, while iNaturalist 2018 is a real-world, naturally long-tailed dataset. CIFAR-100-LT has three imbalance ratio settings {10, 50, 100), where the imbalance ratio is defined as Wmar//umn. For each dataset, we employ the officially provided version. The detailed descriptions for datasets in Appendix C.
Evaluation Metrics, Model performance is mainly measured by the overall Top-1 accuracy (All). Following Ahn al. (2023), we also statistically measure the accuracy on three disjoint subsets of the long-tailed datasets: head classes (Head), medium classes (Medium), and tail classes (Tail). Accuracy reported as a percentage.
Comparison Baselines. We select a variety of long-tailed recognition methods as baselines, which are based on different theoretical ideas, including cross-entropy loss (CE) He et al. (2016), class re- e--balnncing methods: CE-DRW Cao et al. (2019) LWS Kang et al. (2020).
et al. (202))
LDAM-DRW Cao et al. (2019). Balanced Softmax (BS) Ren al. (2020), information augmentation methods: CMr et al. (2022), CUDA Ahn et al. (023)) and module improvement methods: RIDE with three experts Wang et al. (022)) BCL.Zhu et al. 2002). Thanks to the high flexibility of DODA, we integrate it with the aforementioned baseline to observe the gains DODA brings to existing methods. The detailed discussions and descriptions for baselines are in Appendix B and Appendix C.
Implementation. We use Pytorch Paszke et al. (011)) to implement all neural networks and train the model 8 NVIDIA Tesla V100 GPUs For CIFAR- dataset, we follow the general experimental setup from [Coo et al. (2019) and utilize ResNet-32 He et al. (2016) as a backbone network. The networks are trained for 200 epochs by the SGD optimizer with an initial learning rate of 10-1, a momentum of 0.9, and a weight decay of 2 x 10-1, We use a random strategy (up-weight and down-weight operations aren't 't active) for the first 50 epochs to avoid cold-boot issues and then switch to the proposed strategy for the remaining epochs. For ImageNet-LT and iNaturalist 2018 datasets, we use ResNet-50
backbone network. We train the network for 100 epochs using an initial learning rate of 0, and decay the learning rate at epochs 60 and 80 by 0.1. For all experiments, we set the hyperparameter values Paug as 0.5.
4.2 BENCHMARK RESULTS
CIFAR-100-LT. Table I displays the overall classification accuracies on CIFAR-100-LT dataset.
It can be observed that DODA achieves comprehensive and stable improvements over the original long-tailed learning baselines: HE et al. (2016), CE-DRW et al. (2019), LDAM- MDW Cao
et al. (2019), BS Ren et al. (2020) RIDE with three experts Wang al. (202T) BCL Zhu al.
7

---

Published as a conference paper at ICLR 2024
Table 2: Accuracy (%) on ImageNet--.T and iNaturalist 2018 datasets wtih state-of-the-art methods.
ImageNet-LT
iNaturalist 2018
Method
Head
Medium
Tail
All
Head
Medium
Tail
All
Ht 2016
64.0
33.8
588
(1.60+0.0)
73.9
63.5
55.5
(=0.0)
CR + CUDA Atn ét al. 2023
67.1
47.1
13.4
(+5.6)
74.6
65.0
57.2
62,5 (+1.5)
DODA
47.5
13.9
48.1 (+6.5)
66.0
58.4
(3.6
CE-DRW Cao
2019
61.7
47.3
28.8
50.1 (+0.0)
68.2
67.3
66.4
67.0 (+0.0)
CE-DRW CODA Amii oral.
2023
61.7
48.4
30.5
51.1 (+1.0)
68.8
67.9
66.5
67.4 (+0.4)
CE-DRW DODA
62.4
48.5
31.3
(42.1)
69.0
68.2
67.8
68.2 (+1.2)
LWS Kang al. 2020
57.1
45.2
29.3
47.7 (+0.0)
65.0
66.3
65.5
65.9 (+0.0)
cRT Kang
2020
58.8
44.0
26.1
47.3 (+0.0)
69.0
66.0
63.2
65.2 (-0.0)
+CODA eral.
2023
62.3
47.2
28.1
50.2 (+2.9)
68.2
67.9
66.4
67.3 (+2.1)
cRT
+DDAA
62.8
47.7
28.9
51.3 (+3.6)
69.2
68.3
67.6
68.5 (+3.3)
LDAM BWW.DDW//R
Bwt.-DWWCaoeeeee Cao al.
2019
60.4
46.9
30.7
49.8 (+0.0)
(-0.0)
I.DAM-DRW CUDA Ahn ct
2023
63.2
48.2
31.2
(+1.7)
68.0
67.5
66.8
67.3 (=1.2)
LDAM DRW DODA
63.7
48.6
31.9
52.4 (+2.6)
68.6
68.1
67.9
68.7 (+2.6)
BS Ren cl 12220
48.8
32.1
(+0.0)
66.7
67.4
67.5
67.3
BS CSSAA Am ot 2023
61.8
49.1
31.8
(+0.5)
67.6
68.2
68.3
68.2
DODA
61.9
49.5
32.4
(+1.0)
68.1
69.5
69.4
RIDE (3 experts Wang et al 2021
64.9
50.4
34.4
$3.6 (+0.D)
70.4
71.8
71.8
71.6 (+0.0)
RIDE CMO Palk
72.0
70.9 (-0.7)
RIDE (3 experis + CUDA Ann al. 2023
66.0
51.7
34.7
54.7 (+1.1)
70.6
72.6
72.7
(-1.4)
RIDE experts) DODA
(+2.2)
73.9
(-2.8)
BCI /Zu et al 2022
65.3
53.5
36.3
55.6 (+0.0)
69.4
72.4
71.8
(-0.0)
BCL
2023
53.9
36.6
56.3 (+0.7)
72.7
72.0
(+0.4)
BCL + DODA
66.9
54.1
37.4
56.9 (+1.3)
71.2
73.2
73.4
73.7 (-1.9)
(022). The outstanding performance tail classes also demonstrates that DODA alleviates severe class sacrifice issues. Moreover, compared to existing class-indeped independentt long-tailed augmentation baselines: CMO Park al. (2022) and CUDA Ahn al. (2023). DODA achieves superior perfor- mance, especially on tail classes. Essentially, this improvement is due to the special 'care' given to the sacrificed classes in class-independent augmentation.
ImageNet-LT and iNaturalist 2018. We also compared DODA with state-of-the art taigetailed recognition methods on large-scall datasets, and the experimental results are shown in Table 2 Applying DODA to the basic CE loss significantly improves the model performance and can be comparable to the performance of existing long-tailed learning methods. It is worth noting that ImageNet-LT and INaturalist 2018 have higher imbalance ratios (i.e., 256 and 500) than CIFAR-100- LT (i.c., 10, 50, and 100), so the results also demonstrate that DODA can improve model performance in scenarios with varying degrees of imbalance.
4.3 FURTHER ANALYSIS
In this section, we conduct a detailed analysis of the mechanism of DODA and discuss the following four issues. All the analysis experiments are conducted on CIFAR-100-LT (IR=100). More empirical results are reported in Appendix D.
+ DODA
SR=7%
IR=50
Does DODA make fewer classes be 'sacrificed"? In Section 2 we analyze and validate that class- independent DA "hypocritical', as it achieves the improvement of average performance by sacrificing cer- tain classes (especially tail classes), which runs counter to the purpose of long-tailed learning. The goal of DODA is to pursue inter-class fairness while alleviat- ing both inherent data-wise imbalance and extrinsic augmentation wise imbalance. By allowing each class to choose appropriate DAs, DODA can effectively alle-
5.0
IR=1000
Ac
AccCCF+DODA
Figure 4: Visualization of the accuracy of each class between CE and CE with CUDA. viate the problem of sacrificing 'weak' classes while achieving generalization performance. From the it can be found that compared with CUDA,
visualization of the accuracy of each class in Figure 4
8

---

Published as a conference paper at ICLR 2024
DODA reduces the sacrifice rate by 31% and 24%, respectively, indicating that DODA makes fewer classes be 'sacrificed'.
Why DODA can alleviate the long-tailed problem? Long-tailed learning aims to learn accurate and robust deep model that can achieve generalized performance on long-tailed distribution tasks. DODA addresses the common problem of unfair DA in existing long-tailed methods from the perspective of DA. Our method is orthogonal to existing methods. By maintaining preference list for each class, DODA provides each class with the right choose DA fairly, As shown in Figure [ we observe the trend of the selection hierarchies on two baselines (BS and CE). We randomly selected four classes (Index= = {0, 10, 40, 80)) and 10 common DAs. can be observed that, the epoch increases, the preferred DA for specific classes gradually becomes apparent, indicating that each class can choose positive DAs to avoid being sacrificed, thereby alleviating the long-tailed problem.
BS, Index=0
BS. Index=80
CE. Index=10
CE. Index=40
Augmentation
5
987654322
10
10
7.6
10
50
100
200
300
100
900
330
100
350
Epoch
Epoch
Epoch
Epoch
Figure 5: Trend of the selection hierarchies during training.
What are the trends in DAS favored by classes? As illustrated in Figure 6 we visualized the preferred aug- mentation methods for different classes, and it can be observed that certain augmentation methods are highly favored because some DAs tend to distort some decisive information in the data, while other DAs prefer to preserve label-related information.
Would other augmentation methods be better? We also compared DODA with existing DAs to demonstrate its superiority in long-tailed learning. Following the setting Ahn et al. 2023), we select six augmentation methods, including AutoAugment (AA) Cubuk et al. (2019), Fast AutoAugment (FAA) Lim et al. (211)). DADA Li et al. (2020a), RandAugment (RA) Cubuk al. (2220) and CUDA Ahn al. (2023) Except for CUDA. other methods require additional computational resources to search for suitable DAs for datasets. Although CUDA implements class-wise strength adjustment, it still struggles to avoid class sacrifice issues. As shown in Figure 7
Figure 6: Class and 'its' preferred DA on BS and CIFAR-100-LT (IR=100).
DODA outperforms other search-based and strength-based based DAs,
achieving fair augmentation while being computationally efficient.
CF
CE-DRW
LDAM DRW
BS
RIDE (3 expens)
43
40H
27
20
10
Asgmersation
Figure 7: DA analysis on various algorithms, CE, CE-DRW. LDAM-DRW, BS, and RIDE.
5 CONCLUSION
In this study, we first theoretically analyzed the gains of traditional DAs in long-tailed learning and then proposed 'stone' called Dynamic Optional Data Augmentation (DODA) to kill two 'birds": inherent data-wise imbalance and extrinsic augmentation- wise imbalance. DODA allows each class to choose appropriate DAs by maintaining a corresponding DA probability distribution for each class. Extensive experiments across loigetailed benchmarks verify the effectiveness of the DODA.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This paper partially supported by the National Natural Science Foundation of China 62022772 No. 1o22922790, the Project of Stable Support for Youth Team in Basic Research Field, CAS (No. YSBR-00)))) Academic Leaders Cultivation Program, USTC.
REFERENCES
Sumyeong Ahn, Jongwoo Ko, and Se-Yo Younoung Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. In The Eleventh International Conference on Learning Representations, 2023.
Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmenta- tion are class dependent. In Advances Neural Information Processing Systems, 2022.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with fabel-distrbbution-aaarr margin loss. Advances in neural information processing systems, 32, 2019.
Hsin-Ping Chou, Shih-Chieh Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebal- anced mixup. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 95-110. Springer, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tail long-tailed data. In Computer Vision EsconnEECV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 694-710. Springer, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. blass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning, pp. 8082-8094. PMLR, 2022.
Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. Robust sumi-supervised learning when not all classes have labels. volume 35, pp. 3305-3317, 2022.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning, pp. 2535-2544. PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning to segment the tail. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14054, 2020.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3375-5384 2016.
Lin-Han Jia, Lan Lhe Guo. Zhi Zhou, Jie-Jing Shao, Yuke Xiang, and Yu-Feng Li. Bidirectional adaptation for robust supervised learning with inconsistent data distributions. In /CML 2023, 2023.
10

---

Published as a conference paper at ICLR 2024
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yan- nis Kalantidis. Decoupling representation and classifier for long- tailed recognition. In Eighth International Conference Learning Representations (ICLR), 2020.
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partitioning reservoir sampling. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 411-428. Springer, 2020a.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13896-13905, 2020b.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553))436-444 2015.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 580-595. Springer, 2020a.
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang. Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991-11000, 2020b.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 864-873, 2016.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6887-6896, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems Workshops, 2017.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252 2015.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11662-11671, 2020.

---

Published as a conference paper at ICLR 2024
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1524, 2020.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-clasd learning for image classifi- cation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
5486-5494, 2018.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard. Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2018
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991.
Pengkun Wang, Xu Wang, Binwu Wang, Yudong Zhang, Lei Bai, and Yang Wang. Long-tailed time series classification via feature space rebalancing. In Proc. of DASFAA, 2023.
Tao Wang. Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang. Steven Hoi, and Jiashi Feng. The devil is in classification: A simple framework for long-tail instance segmentation. In Computer Aision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 728-744 Springer, 2020.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-awarr experts. In International Conference on Learning Representa- tions, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5704-57133 2019.
Salah Zaiem, Titouan Parcollet, and Slim Essid. Automatic data augmentation selection and parametrization in contrastive self-supervised speech representation learning. In Interspeech 2022. pp. 669-673 ISCA, 2022.
Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3457-3466, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.00911 2017a.
Xiao Zhang, Zhiyuan Fang, Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017b.
Xing Zhang, Zuxuan Wu, Zejia Weng. Huazhu Fu, Jingjing Chen, Yu-Gang Jiang, and Larry S Davis. Videolt: large-scale long-tailed video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7960-7969, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep lang-tailed learning: survey. arXiv preprint 1 2021b.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for aest-agnostic long-tailed recognition. In Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2024
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brance network with cumulative learning for long-tailed tailed visual recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9719-9728, 2020.
Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Wei Gong. Class-conditional Sharpness-aware minimization for deep long-tailed recognition. In Proceedings of the TEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1499-3509, 2023.
Jianggang Zhu, Zheng Wang. Jingjing Chen, Yi-Ping Phoebe Chen, and Yu- Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6908-69) 17, 2022.
13

---

Published as a conference paper at ICLR 2024
Appendix
Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning
The content of the Appendix is summarized follows:
1) in Sec. A we state the proofs of Theorem 2 (Sec. 2.1). Theorem 3 (Sec. 2.2), and Theorem
(Sec. 3.2).
2) in Sec. B summarize existing long-tailes learning (LTL) and data augmentation (DA)
methods and explicitly illustrate the novelty of DODA.
3) in Sec. C
we demonstrate the details of datasets and baselines we use experiments of
DODA.
4) in Sec.
we illustrate more detailed empirical results and analyses of DODA.
A DETAILED PROOFS
A.1 PROOF OF THEOREM 2
Proof. In Sec. 2.1, Theorem states that when approximate the ideal model f* by minimizing the training loss (i.e., training error), the latter tends to zero, while the former is greater than zero due to the augmented samples deviate from the level set of f*. Therefore, the trained model fe will be biased compared to the ideal model f*
Σ (R.N)ED
Ellyy f (0(x)) OA
Eilyy fo(O (x.
Σ
bias
(10)
From the perspective of the dataset, the reason for the bias that the semantic information of some samples does not match their original labels after DA, which means that DA cannot guarantee that it preeeergrs.......... We reconsider this problem from the perspective of long-tailed learning. First of all, for the whole dataset D, minimizing the training loss essentially means that the trained model fo should achieve training error on each class, i.e.,
Σ Elly- (o(O = (Fa)) (K.))D
Σ
-1lv~~((O0))) / =0 VCEC (1))
Similarly, for class and augmented samples -,) or approximately achieve the minimization of the training loss on class c, DA inevitably makes some samples of class e mismatch with their original labels, i.e., the augmented samples deviate from the
DJ. when we achieve
level set of the ideal model f". Therefore, when we use the augmented samples of class as inputs to f*, f cannot predict the labels completely correctly.
Σ - (x,g)ED
>
Σ (x.y)
Eilly - f O VCE (12)
Although we achieve a seemingly optimal (ideal) training model fo. its fitting process class has actually deviated from the ideal optimization process. Therefore, the deviation between the trained model fe and the ideal model f on class is inevitable, i.e. fo has class-wise bias.
Σ
T [ilyy (0 A
Σ
Ellly (z))) 0 class-wise bias
ED
(x.))DD2==c
(13)
A.2 PROOF OF THEOREM 3
Proof. According Definition 1, the distribution of each class in the training can be approximate as circle in a two-dimensional feature space, and the distribution center of class e can be defined as (%e, Ye) and the distribution radius R., So, the distribution span S. can be expressed as follows:
S (X xx) (W W.)² R.²
(14)
14

---

(Re, + Dc, RR.. Published as a conference paper at ICLR 2024
For the data distribution S.z and Se, of head class e₁ and tail class ct, the new distribution span after using uniform DA O()) can be defined as Se, + (Rc, + Ao,) and Sc, + (R. + Ac,)² ), and Ac, and A. represent the increase in distribution radius within each class after DA. For the same augmentation method, 4I, = A.c
Here, define the original data distributions of head class ea and tail class as the base spaces Be, 2 and c.... 2 and define the expanded data distributions of head class cp and tail class as the marginal spaces (Ie,, + A R.: and (Rec. + Ae,) Reea" 2
Then, the augmentation sensitivity of head class op and tail class or can be defined as Ve, and dle.
(Ren + Ac)) )2 Rea
Ven
(15)
V.
(16)
I.,"" 2
Therefore, we can measure the augmentation sensitivity difference between head class ey and tail class Cz, i.e.,
Ber + Ar,
)2
Rey
(Re, + A.,.) )2 Rey,
the - V/en
R
B.y 2
Re, + 2BR Ac, + Act
Re,
R.n 2Rc Dcs Dcn 2 Ren
2
BRa...
2R, Ag, + A
2B,, Ac, + -Dcn. 2
Rey
Rec."
2R. Res "a., + -oo 2 D 21R R.., Dc, - R. ADch"
(17)
Rey 2RRen
2TR Rey (TR, Ds.
Re, Aen)
Th. 2A TR. PAnn
+
R., "Ren. 2
Re, "Re. 2
Rec R.
Rey 2
Rey
2
-2Aea
+ Den
2.
Roy Ro,
Rey
2 2IR,.
The above derivation indicating that tail classes are more sensitive to the marginal space.
For high-dimensional feature space,
Proof. Assuming that the dimension of high-dimennionnl features is nı, distribution center of class e is defined as (X), x2 Xe.... XC) and the distribution radius is Re The distribution span S, can be ...
expressed as
(18) We assume that the data distribution of head class G₁. and tail class et S., and S., and the distribution span after DA So, and S So the augmentation sensitivity v of class can be expressed as follows:
(
x3
x R
(3.+
/1/BBB
T(1+n/2)
T(14-//2
V.
(19)
7n/2B.
[(1+n/2)
and further deduce:
7/22
Re,RRe + ARR, )" (Re AR )r
Ve the
(20)
['11 + n/2)
Re, "R.." "
This indicates that this theoretical explanation is equally applicable to higher-dimensional spaces.
15

---

Published as a conference paper at ICLR 2024
A.3 PROOF OF THEOREM 4
Proof. We want to show that for a more dominant DA Oki, the bias of Ohi from P is smaller than that of kh from P, where Q and Q*= are the level-sets of the models fo and f learned using DA Oh and DA Okk,, respectively, and P is the level-set of the model fo trained on the original dataset.
Herr, we use Chebyshev's inequality to bound the probability that a random variable deviates from its expected value by certain amount. Let X be random variable that represents the deviation of fe (x) from for sample (a, y) in the original dataset. Let Y be a random variable that represents the deviation of f (O* (x)) from y for a sample (a, y) in the augmented dataset using DA O Then
we have:
P([X E(X) t) S Var(X)/1²
(21)
P(JY - E(Y) > t) S Var(Y)/1²
(22)
The level-set bias 8(Q*, P) can be defined as the degree of distributional deviation between the level-sets Q* and P. Intuitively, this can be measured by the difference between E and (
or the difference between Var(Y) and (Xr(X). We assume that E since fo is trained to minimize the training loss on the original dataset. Then we have:
8( P) = |E(Y) + [Var(Y) Var(X))
(23)
Now, suppose that Ohi dominates 0h2 on class c, i.c. V pos This means that fOh (x)) is
more likely to be equal to y than (e(Oaaaz)) (x)) for samples (æ, y) in class Therefore, we have:
E(Yly = c, 0 E(Y c, Okz)
(24)
Var(Yly c. okh Var(Y c, Okz)
(25)
By taking the weighted average over all classes, we obtain,
B(Y10h)>< E(Y)Ok)
(26)
V < <ar(V1
(27)
Hence, we conclude that:
8(@ P) 6(Q P)
(28)
This completes the proof.
B RELATED WORK
B. LONG-TAILED LEARNING (LTL)
Real-world training datasets typically exhibit a long-tailed class distribution, where a small fraction of classes have massive samples and the rest classes are associated with only few samples. Un- fortunately, the deep models trained by the common practice of empirical risk minimization cannot handle this distribution, resulting in a significant decrease in model performance Zhang al. (2021b). Recently, missive novel longt-tailes learning methods have been proposed to learn a more generalized model from imbalanced training datasets, which can be divide into three main categories: class re-balancing Kang et al. (2020); Ren et al. (2020); Wang et al. (2020); Lin et al. (2017); Cui et al.
(2019): : Tan et al. [2020), module improvement /hang et al. (201 7b); Ouyang et al. (2016): Tang et al.
(200 Kang et al. (2020); Zhou et al. (2020); Zhang et al. (2022). and information augmentation Chu
ct al. (2020); Kim et al. (202bb); Hu et al. (2020); Zang et al. [202] Park et al. (0222 Ahn et al.
(2023).
Class re- re-balancing is the most typical strategy, which balances inter-class sample numbers or weights by re-sampling cost-sensitive learning. On the one hand, traditional re-sampling methods, e.g., random sver-sampling (ROS) and random samer-sa (RUS), achieve re-balancing by repeating the samples from tail classes and discarding the samples from head classes, but they tend to overfit.
16

---

Published as a conference paper at ICLR 2024
to tail classes when datasets extremely unbalanced. To this end, recent studies propose class- balanced re-sampling strategies, e.g., bi-level blass-balanced sampling Wang et al. (2020) and meta learning based sampling Ren al. (2020). Besides from the perspective of classes, scheme-oriented sampling strategies try to re- balance classes by designing some specific learning schemes, such as quintuplet sampling |Huang et al. (20T6) and replay based sampling Kim et al. (2020a). On the other hand, some studies, called cost-sensitive learning, re-balance classes by adjusting the loss values of different classes. For example, CB Cui et al. (2019) proposed a effective number to approximate
the expected sample number of each class, and Focal loss Lin et al. (2017) used the prediction probabilities to inversely re-weight classes.
In addition to class re- re-balancing, researchers also explored enhancing model performance by improv- ing network modules. intuitive method is decoupled training. which decouples the learning proce- dure into representation learning and classifier training. As a pioneering work, Decoupling Kang et al. (2020) proposed a two-stage training scheme and showed some refreshing observations. KCL Kang al. [202 I and FRS Wang et al. (2023) believed that a balanced feature space is beneficial to LTL, so they designed contrastive learning based losses to learn a more class-balanced and class-discrmminative feature space. Furthermore,
classic theory, ensemble learning is also applied to LTL by designing and combining multiple expert networks. For instance, BBN Zhou et al. (2020) proposed to use two network branches to handle L.TI.. Following BBN, BAGS Li et al. (2020b) explored multi-head scheme. Not restricted to a balanced test set, SADE et al. (2022) explored the multi-expert scheme to handle test intributtionagnosiiii LTL.
Although the overall performance is improved, these methods cannot essentially handle the issue of lacking information, particularly on tail classes due to limited data amount. Orthogonally, some information augmentation studies seek to introduce additional information into model training, such as FTL Yin et al. (2019) and M2m Kim et al. (2020b) transferred the knowledge from head classes to enhance model training on tail classes considering the inter-class knowledge imbalance. To solve information restrictions in essence, another line of research to apply representation augmentation or data augmentation to L.TT.. For example, CMO Park et al. (2022) augmented diversified minority samples by leveraging the rich context of the majority classes as background images. Considering fairness, FSR Wang et al. (023) and CUDA Ahn et al. (202)) advocate to find appropriate augmentation strength for each class. However, although these methods enrich the overall information to certain extent and improve model performance, they ignored the sacrifice of some classes behind this improvement. For this reason, we jointly pay attention to the inherent data- wise imbalance and extrinsic augmentation-wiss imbalance, thereby minimizing the sacrifice.
B.2 DATA AUGMENTATION
DA has been applied in many fields because it can effectively alleviate overfitting and improve model generalization performance. DA is simple in design, and various DAs can be achieved through image manipulation, e.g., filp, crop, and rotate Robbins & Monro (195T). Recently, mixup based DA methods are proposed to improve model robustness by fusing two images and their labels Zhang ctt al. (2017a): Tokozume et al. (2018). Considering the diversity of DA, some studies try to combine them randomly or order, such as AutoAAugment Cubuk et al. (2019) Fast AutoAugment Lim et al. (2019), , DADA Li et al. (2020a), and RandAugment Cubuk et al. (2020). In addition, researchers are improving DAs to make them suitable for LTL, however, they ignore that DA is class dnassiindeeendett, and thus may cause a mismatch between augmented data and actual labels |abels|Park et al. (2022);
Lim
) Wang let al. (2023); Ahn al. (2023). Therefore, it is necessary to design class-dependent long-tailed DA to allow each class to choose an appropriate augmentation method.
C DATASET AND BASELINE DETAILS
C.I DATASET
CIFAR-100-LT Cao et al. (2019): is long-tailed version of artificially truncated from the original balanced dataset CIFAR-100, which includes 100 different categories, 50,000 training images and 10,000 test images. The 100 categories in CIFAR-100 form 20 superelasses, each with 5 classes. CIFAR- has three imbalance ratio settings 10, 50, 100, where the imbalance ratio is defined as the ratio of the sample sizes of the most frequent and least frequent classes, i.e., p N Nmaz/ Nmin-
17

---

Published as a conference paper at ICLR 2024
Table 3: Statistics of the long-tailed datasets.
Dataset
of Classes
# of Training set
# of Test set
Imbalance ratio
CIFAR-100-LT 100
50,000
10,000
(10, 50, 100)
ImageNet-LT
1,000
115,846
50,000
256
iNaturalist 2018
8,142
437,513
24,426
500
ImageNet-L.T Liu et al. (2019): is long-tailed version of artificially truncated from the original balanced dataset ImageNet, which includes 1,000 different categories, 115,846 training images and 50,000 test images. The most frequent or least frequent class has 1,280 or 5 images, the imbalance ratio p = 256.
iNaturalist 2018 Van Horn et al. (2018): is real-world, naturally long-tailed dataset, which includes 8,142 different categories, 437,513 training images and 24,426 test images. Each image has one ground truth label. The iNat dataset is highly imbalanced with dramatically different number of images per category and the imbalance ratio p is 500.
C.2 AUGMENTATION
we incorporated ten commonly used DA methods in our experiments. and descriptions are shown in Table 4 The specific code implementation can be found in //au//dddappy.
Table 4: Description of DAs utilized DODA.
DA
Parameter
Description
Flip
0/1
Flip top and bottom
Mirror
0/1
Flip left and right
EdgeEnhance
0/1
Increasing the contrast of the pixels around the targeted edges
Detail
0/1
Utilize convolutional kernel [[0,-1, 0J. [-1, 10,-1], [0,-1, 0]]
Smooth
0/1
Utilize convolutional kernel [[1, I, 1J. [1, 5, 11. [1, 1, I]]
AutoContrast
0/1
Remove a specific percent the lightest and darkest pixels
Equalize
0/1
Apply non-linear mapping to make uniform distribution
Invert
0/1
Negate the image
GaussianBlur
[0, 2]
Blurring an image using Gaussian function Rotate the image
Rotate
[0, 30]
C.3 BASELINES
To ensure a fair comparison, we select large number of aangttilled learning methods as baselines in our experiments, and integrate DODA with these baselines to evaluate the effectiveness and flexibility of DODA. In addition, we also select the cate-of-theeart data augmentation methods as comparison baselines to prove the superiority of DODA long-tailde learning.
Long-tailed methods:
CE He et al. (2016: cross-entropy loss based model, which is one of the most classic methods in the field of deep long-tailed learning.
CF-DRW Cao et al. (2019: is a two-stage fine-tuning strategy based on cross-entropy loss.
LWS Kang et al. (2000): is a two-stage training strategy, which keeps both the representations and classifier weights fixed and only leam the scaling factors.
18

---

Published as a conference paper at ICLR 2024
cRT Kang et al. (2020): is two-stage training strategy, which keeps the representations fixed and randomly re- re-miitiaiize and optimize the classifier weights using class- buiss-balanced sampling.
LDAM-DRW Cao al. (2019): extends the existing soft margin loss by enforcing class- dependent margins based on label frequencies and further introduces a deferred re-balancing
optimization schedule.
BS Ren et al. (2020): proposes to use the label frequencies to adjust mode predictions during training, that the bias of class imbalance can be alleviated the prior knowledge.
RIDE (3 experts) Wang ct al. 2021 introduces knowledge distillation multi-cxpert framework reduce the parameters by learning a student network with fewer experts.
2C et al.
2022 proposes balanced contrastive learning loss and learns stronger feature representations through dual-branch framework.
CMO Park et al.
[2022): focuses on utilizing the rich context of majority samples to improve
the diversity minority samples and mixes minority and majority images by using CutMix to enhance balancing and robustness simultancously.
CUDA Ahn et al. (2023): :
a simple and efficient curriculum, which is designed to find the appropriate per-class strength of data augmentation.
Data augmentation methods:
AutoAugment Cubuk et (209); describes simple procedure automatically search for improved data augmentation policies by designing a search space where policy consists of many sub-policies, one of which is randomly chosen for each image in each mini-batch.
Fast AutoAuggent Lim et al. (2019): finds effective augmentation policies via a more efficient search strategy based on density matching.
DADA et al, (2000a): relaxes the discrete DA policy selection to differentiable optimization problem via Gumbel-Softmax and introduces an unbiased gradient estimator to learn an efficient and accurate DA policy.
Randentment Cubuk et al. 20200;: proposes a simplified search space that vastly reduces the computational expense of automated augmentation, and permits the removal of separate proxy task.
D MORE EMPIRICAL RESULTS
D.1 PARAMETER SENSITIVITY ANALYSIS OF Pang
BS
BCL
Accuracy (%6
Accuracy (5))
50
Accuracy
45
40
45
40
114
05
Aug prob.
Paug
Aug prob. Pang
Aug prob.
Pang
Figure 8: Parameter sensitivity analysis of augmentation probability Pang on CE, BS, and BCL.
To preserve the knowledge of the original dataset, we define the augmentation probability Paug: For further analyze the impact of Pangs we conduct sensitivity analysis of hyperparameter Paug- As shown Figure g we test different hyperparameter settings on three baselines, and the experimental results showed that too small augmentation probability cannot sufficiently improve the model's generalization, while too large augmentation probability cannot retain the knowledge in the original dataset, resulting in decrease in model performance.
19

---

Published as a conference paper at ICLR 2024
50
Accuracy (%)
48
46
44
42
40
CE
BS Method
BCL
Figure 9: Parameter sensitivity analysis of number of DAs be selected L.
D.2 PARAMETER SENSITIVITY ANALYSIS OF
In previous analyses, select the optimal DA for each class. However, find that in some baselines, multiple DAs can be beneficial. Therefore, we further conduct a parameter sensitivity analysis the number of DAs be selected As shown Figure 9 we three hyperparameter settings (t = 1, 3) on three baselines. It can be observed that on CE and BCL, the model tends to select the optimal DA, while on BS, it tends select both the optimal and suboptimal DAs. This phenomenon is consistent with the trend of selection hierarchies during training mentioned in the main text.
D.3 PARAMETER SENSITIVITY ANALYSIS OF NUM. OF DAS
CE
BS
Accuracy (%)
48.5
44.5
48.0
44.0
43.5
47.5
10
Num. DAs
Num. of DAs
Figure 10: Impact of different numbers of DAs LIFAR-100-LE (IR=100).
As shown in Figure 10 we gradually reduced the number of DAs based on the degree preference. The results indicate that reducing the number of augmentations leads to loss of diversity. However, when the 'neglected' augmentations are removed, the model performance does not significantly degrade.
D.4 NETWORK ARCHITECTURE ANALYSIS
As shown in Figure II following Ahn et al. (023),, we also utilize ResNet-10 al, (2019) and ResNeXt-50 Xie et al. (0117) as our backbone network on ImageNet-LT. conduct comparative experiments on three baselines (e.g., CE, BS, and BCL), and the results show that matter what kind of backbone is used, DODA can always bring stable improvement to long-tailen learning algorithms.
D.5 TRAINING TIME ANALYSIS
In DODA's augmentation pipeline, we require additional computations to update and maintain the augmentation preference list for each class. Therefore, compared to the original baselines, using
20

---

Published as a conference paper at ICLR 2024
ResNet-10 w/o DODA
ResNcXt-50 DODA
ResNet-10 10 w/ DODA
ResNeXt-50 DODA
50
Accuracy (%)
45
35
CE
LDAM Method
BS
Figure 11: Network architecture analysis.
DODA incurs additional training time. As shown Table 5 using DODA inevitably brings varying degrees of computational cost, but these costs are acceptable. For example, BS w/o DODA achieves better model performance and avoids serious sacrifices with only x 0.09 additional cost.
Table 5: Training time (min.) analysis on various algorithms.
Method
CE
BS
BCL
w/o DODA
60
68
94
68
74
102
w/ DODA
(x 1.13)
(x 1.09)
(x 1.09)
D.6 MORE ANALYSIS ON SACRIFICE RATES
SR on Different Long-tailed Baselines: We provided the sacrifice rates of different data augmenta- tions on CE in Figure 2 indicating that DA can lead to sacrifice problems for the original baseline. Similarly, long-tailed learning baselines also face this issue. Based your comments, conducted further experiments on cRT and LIFAR-100-LT dataset (IR 100). The results in Table show that CUDA improved accuracy while sacrificing performance for certain classes, while DODA mitigated this sacrifice issue by preserving performance across classes.
have
Table 6: Accuracy (%) on CIFAR-100-LT dataset (IR 100) wiih cRT. SR (%) indicates the sacrifice rate.
Method
Head
Medium
Tail
All
SR
CE
65.6
36.2
8.2
38.1
CE CUDA
70.7
41.4
9.3
42,0
29
CE DODA
74.8
43.8
10.0
44.5
5
cRT Kang al. 2020)
64.4
49.1
25.8
47.5
cRT CUDA
63.2
50.9
26.6
47.9
22
cRT DODA
64.4
51.2
27.5
48.7
In general, just like focusing tail classes when improving the average accuracy, when applying DAs learning, focusing on vulnerable classes that are easy to be sacrificed is also in line with the purpose of long-tailed leamiin.
21

---

Published as a conference paper at ICLR 2024
SR Different DA Baselines: We also tested different class-independent techniques (e.g., Au- toAugment, CutOut) to demonstrate the superiority of our method. The specific experimental results shown in Table g
AutoAugment improves the average accuracy on cRT and takes effect on each shot. However, further analyze the sacrifice problem caused by DAs, and find that despite achieving good performance, AutoAugment still cannot avoid the sacrifice problem, which means,
The performance improvement of Augment is hypocritical, for example, in the tail classes, the model achieves performance gains on classes, while performing badly others (i.e., pleasing the 'strong' and bullying the 'weak'). This sacrifice goes against the purpose of long-tailed learning despite the average performance improvement of the model on the tail classes.
Both class- inss--dddeenddnt techniques lead to the sacrifice problem of sacrifice. From the sacrifice rate of different shots, it can be found that compared with the head classes, more classes in the tail classes sacrificed, indicating that the tail classes are more likely be regarded as the bullied 'weak' mentioned above.
Table 7: Accuracy (%) on CIFAR-I00-LT dataset (IR = 100) wtih cRT. (.) indicates the sacrifice rate of different shots.
Method
Head
Medium
Tail
All
cRT Kang et al. (020
64.4(-)
49.1(-)
25.8(-)
47.5(-)
cRT + AutoAugment
64.8(5)
49.9(6)
25.9(13)
47.9(24)
cRT + CutOut
61.3(12)
44.5(15)
21.7(23)
43.6(50)
+ DODA
64.4(2)
51.2(1)
27.5(3)
48.7(6)
on Different Epochs: Here, we analyzed the changes in the sacrifice rate. The results shown in Table 2 show that the sacrifice problem caused by previous DAs cannot be eliminated during training, while DODA significantly improves this.
Table 8: Sacrifice rate (%)
Various Epochs.
Epoch
100
200
400
CE RandAugment
328
315
309
CE + DODA
73
59
55
D.7 MORE COMPARISONS WITH MODIFIED TWO-STAGE MODEL
Here, we compare DODA with CC-SAM Zhou et al. 2023 which is a two-stage model improvement method that trains the model in a decoupled manner and introduces classcoonitional sharpness-awe aware minimization in the first stage. We have improved the existing open-source implementation and incorporated DODA's augmentation strategy. The quantitative experimental results are shown in Table
Table 9: Accuracy (%) on CIFAR-100-LT dataset (Imbalance ratio 100) wiih CC-SAM. SR (%) indicates the sacrifice rate.
Method
Head
Medium
Tail
All
SR
CC-SAM Zhou et al. (2023)
67.6
51.2
30.5
50.7 (+ 0.0)
CC-SAM CUDA Ahn et al. 2023
67.5
52.0
30.7
51.0 (+ 0.3)
31
CC-SAM + DODA
68.4
53.7
33.6
52.8 (+ 3.1)
6
22

---

Published as a conference paper at ICLR 2024
D.8 MORE COMPARISONS WITH AUTO DA OTHER FIELDS
In this section, we compare DODA with Auto DA selection algorithms in other fields. Here we choose the most advanced method Zaiem
Zaie et al. (2022) in the speech field as a comparison. However, directly applying the complete method from it in long-tailed learning does not lead to fair comparisons. So we partially implemented the augmentation strategies proposed in Zaiem al. (2022). Firstly, since Zaiem et al. (2022) relies on a carefully designed pretext task, we replaced it with contrastive learning using cropping and augmentation, where pretext labels for each augmented view of a sample
corresponding to the ID of the sample it originated from. Then, we replaced the downstream task
related to speech with a long-tailed classification task. The experimental results of this modified implementation are shown in Table 10
can be observed that using the automatic augmentation strategy from Zaiem et al. (2022) results in limited performance improvement, while our method outperforms it significantly. The reasons for this are as follows: (1) Zaiem et al. (2022) relies on a carefully designed pretext task, so the improvement it brings may come from diversified data augmentation. (2) Zaiem et al. (2022) lacks the necessary focus on the tail classes, while our method pays more attention to inter-class fairness, resulting in better performance.
Table 10: Accuracy (%) CIFAR-100-LT dataset (Imbalance ratio 100) wtih Zaiem et al. (2022).
Method
Head
Medium
Tail
All
CE
65.6
36.2
8.2
38.1 (+ 0.0)
CE + Zaiem et al. 2022
68.9
38.7
8.4
40.2 (+ 2.1)
+ DODA
74.8
43.8
10.0
44.5 (+ 6.4)
D.9 EXPLORATION OF COMBINATIONS WITH SOTA LONG-TAILED DA
From the macro perspective of long-tailed learning, both DODA and CUDA belong dynamic DA. However, at the methodology level, the two are different. A simple comparison is as Table [1
Table 11: Comparison at the methodology level.
Method
Adaptive Strength
Adaptive Function
Inter-class Fairness Cold-boot Issues
CUDA DODA
It can be seen that ensure fairness between classes while improving accuracy, have made some methodology-leww improvements. More interestingly, we find that CUDA and DODA are orthogonal, and find the optimal DA function and strength at the same time. The exploratory results are as follows:
Table 12: Exploratory results (%) on CIFAR-100-LT dataset (IR = 100).
Method
Head
Medium
Tail
All
CE DODA
74.8
43.8
10.0
44.5
CE DODA + CUDA
74.7
44.1
10.2
44.6
Although the performance gain is limited, continuing to explore this compositionality is beneficial for long-tailed learning.
23

---

Published as a conference paper at ICLR 2024
D.10
MORE TRENDS OF THE SELECTION HIERARCHIES ON DIFFERENT INDEXS
BS. Index=0
BS, Index=5
BS, Index=10
BS. Index=20
40
90
25
Augmentation
Augmentation
30
Apgmentation
50
2
20
4
5 4
15
20
40
8 6
10
8 7
10
20
250
500
2000
50
100
250
300
250
100
150
250
300
330
130
500
200
300
380
Epoch
Epoch
Epoch
Epoch
BS, Index=40
BS, Index=60
BS, Index=80
Augmentation
Augmentation
Augmentation
8 7 5 5
100
2011
130
300
350
400
100
200
250
300
350
400
100
150
200
250
300
350
400
Epoch
Epoch
Epoch
CE Index=0
CE. Index=5
CE. Index=10
CE Index=20
Augmentation
2
Augmentation
Augmentation
40
40
30
30
20
5
20
20
10
100
2003
350
50
100
200
150
350
109
100
1.50
155
300
400
50
1.0
200
150
300.
350
Epoch
Epoch
Epoch
Epoch
CE, Index=40
Index=60
CE. Index=80
30
20
Augmentation
Augmentation
Augmentation
15
30
3 6 5
20
10
200
500
330
50
100
200
250
300
350
400
100
150
250
300
390
500
Epoch
Epoch
Epoch
Figure 12: Trends the selection hierarchies on different indexs.
24


Reviews:
[{"review_id": "MZr7B1fsWt", "reviewer": "Reviewer_PqHc", "summary": "This paper makes a theoretical analysis of data augmentation strategies in long-tailed learning and points out that there may be a variety of potential interweaving imbalances in long-tailed learning, such as class and augmentation. The author further proposes a dynamic optional augmentation strategy to alleviate the above imbalance problem from a new perspective. The proposed method is tested on different main-stream long-tailed datasets, including dealing with imbalance problems and augmentation problems. The comparison with existing methods shows the superiority of the proposed method.", "questions": "1. It would be beneficial if the author could provide a brief explanation or discussion on the scalability of the proposed method, which could promote its application in other open scenarios.\n2. Regarding Section 4.3, the authors mention that DODA can avoid more classes being sacrificed. However, it may not be intuitive to find this based on Figure 4. Therefore, it would be helpful if the author could provide a more detailed explanation of the model's effect, which would allow others to better understand the improvements brought about by this method.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation proposed in this paper on how to choose suitable data augmentation for different classes is practical, which innovates the traditional way of long-tailed learning.\n2. The theoretical analysis is sufficient, and the author has analyzed the potential risks of data augmentation in long-tailed learning. The explanation that data augmentation may bring hypocritical performance improvements is interesting.\n3. This paper conducted a large number of comparative experiments with methods of different tendencies on mainstream benchmarks, including analysis at different levels. The results can demonstrate the effectiveness of this method.\n4. This paper is well written, and the dynamically optional method proposed is easy to understand. In particular, the visualization of the training process allows me to have a more intuitive understanding.", "weaknesses": "1. My main question or weakness is that although the author has conducted analysis and verification in the traditional long-tailed learning field, the long-tailed distribution may not only exist in the visual field, and some other fields (e.g., graph cls.) seem to have the same problem. This paper lacks a discussion on the scalability of the proposed method.\n2. Section 4.3 mentions that 'DODA can avoid more classes being sacrificed', but it seems that this cannot be intuitively found based on Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DHoyMCGqA5", "reviewer": "Reviewer_rRZN", "summary": "This paper addresses the problem of imbalance in existing data augmentation techniques in long tailed learning. This paper proposes to improve this by introducing a dynamic augmentation strategy - DODA (Dynamic Optional Data Augmentation) to reinforce well-performing augmentation strategies and punish the inefficient ones during training. The authors claim that this method kills two ‘birds’ - inherent data imbalance and extrinsic augmentation wise imbalance with one ‘stone’.", "questions": "For contrastive methods such as BCL, how is DODA being used to generate multiple views of the same image for different classes?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The work addresses the problem of long tailed learning from a data augmentation perspective and shows potential for huge gains by simply using augmentations efficiently without much computational overhead.\n\nThe paper has a simple writing style and is easy to read and follow.", "weaknesses": "**Proposed Method**\n\na) The proposed method for DODA is exclusively used where augmentations  are solely for one single class and do not encourage inter class interaction. Augmentation methods such as mixup and its improved version Remix (Chou et al. 2020) have proven to be efficient in data-imbalance settings. However, comparison of the proposed method with Remix + LTL methods is lacking.\nThe algorithm optimizes the augmentation strategy on 10 predefined augmentation functions with fixed strengths. The authors should discuss the optimality of the selection of these functions and provide ablation studies for the same.\n\nb) The algorithm optimizes the augmentation strategy on 10 predefined augmentation functions with fixed strengths. The authors should discuss the optimality of the selection of these functions and provide ablation studies for the same.\n\n**Missing Baselines **\n\nThe effectiveness of DODA on state-of-the-art long-tailed methods PaCo [R1], and NCL[R2] is not demonstrated in the paper. It would be great if the results are provided in the rebuttal.\n\n[R1] Cui, Jiequan et. al. “Parametric Contrastive Learning”, Proceedings of the IEEE/CVF international conference on computer vision, 2021. \n\n[R2] Li, Jun et. al. “Nested Collaborative Learning for Long-Tailed Visual Recognition”. Proceedings of the IEEE/CVF international conference on computer vision, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BB77HCOZLl", "reviewer": "Reviewer_ZxxQ", "summary": "Data augmentation (DA) is one of the strategies employed to address the dataset imbalance issue in long-tailed classification. This paper proposes the DODA algorithm based on the premise that in situations handling long-tailed datasets, the effectiveness of DA may vary for each class. In a nutshell, the proposed algorithm includes setting up a set of DAs and adjusting the preferences for each augmentation individually for each class during the training process. The paper offers experimental findings for CIFAR-LT, ImageNet-LT, and iNaturalist-2018.", "questions": "1. Could you provide some valuable insights regarding the decline in performance that we see in the RIDE column of Table 1?\n2. UniformAugment (LingChen et al., 2020) is a suitable baseline for comparison because of its straightforward application of a set of augmentations at random. It would be nice to see the UniformAugment baseline results.\n3. It seems that the proposed methodology could be effective even in scenarios handling balanced datasets. Have you experimentally verified this by any chance?\n4. ’its’ in the caption of Figure 6; quotation mark issues in LaTeX :)\n\n---\nLingChen et al., 2020, UniformAugment: A Search-free Probabilistic Data Augmentation Approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The provided problem regarding data augmentation in long-tailed classification has a well-established and clear motivation.\n2. Comprehensive experimental outcomes are presented, encompassing a range of existing training approaches and data augmentation techniques for addressing long-tailed classification.", "weaknesses": "1. The straightforward yet most important baseline is missing. The central idea presented in this paper is that there are both positive and negative DA effects specific to each class when working with long-tailed datasets, and it is beneficial to adjust them during the training process through the proposed DODA algorithm. Consequently, keeping the per-class weight distribution $\\mathfrak{Q}_c$ fixed as a uniform distribution in the proposed DODA algorithm would serve as an essential baseline, which highlights the efficacy of the proposed adjusting approach. One can interpret the performance improvement described in the paper as simply due to the incorporation of extra data augmentation.\n\n2. While the main tables present average values, the standard deviations are missing; it would be better to provide complete results, including standard deviations, in the appendices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "diH82dSOSU", "reviewer": "Reviewer_rn58", "summary": "This study focus on data augmentation for long-tailed learning. First, the authors theoretically analyzes that the traditional data augmentation techniques cause  the long-tailed distribution to be imbalanced again. Specifically, the traditional data augmentation techniques cause inherent data-wise imbalance and extrinsic augmentation-wise imbalance, which are called as two birds in this paper. To kill the two birds, the authors propose an data augmentation technique termed DODA which make each class to choose appropriate augmentation methods. Experimental results on 4 benchmark datasets verify effectiveness of the DODA.", "questions": "I have no question.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Code is submitted.\n\nMany studies are reviewed in the Appendix.\n\nThe paper is easy to read.\n\nTheoretical ground for the proposed algorithm.\n\nExtensive experiments are conducted with 4 benchmark datasets.\n\nDetailed analyses for the superiority of the proposed algorithm are provided.\n\nThe proposed algorithm can be orthogonally used with other long tailed learning algorithm such as LDAM and RIDE.", "weaknesses": "I cant find weakness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)