CUDA: Curriculum of Data Augmentation for Long-tailed Recognition
Authors: Sumyeong Ahn
Jongwoo Ko
Se-Young Yun
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Long-tailed recognition
class imbalance

Abstract: Class imbalance problems frequently occur in real-world tasks, and conventional deep learning algorithms are well known for performance degradation on imbalanced training datasets. To mitigate this problem, many approaches have aimed to balance among given classes by re-weighting or re-sampling training samples. These re-balancing methods increase the impact of minority classes and reduce the influence of majority classes on the output of models. However, the extracted representations may be of poor quality owing to the limited number of minority samples. To handle this restriction, several methods have been developed that increase the representations of minority samples by leveraging the features of the majority samples. Despite extensive recent studies, no deep analysis has been conducted on determination of classes to be augmented and strength of augmentation has been conducted. In this study, we first investigate the correlation between the degree of augmentation and class-wise performance, and find that the proper degree of augmentation must be allocated for each class to mitigate class imbalance problems. Motivated by this finding, we propose a simple and efficient novel curriculum, which is designed to find the appropriate per-class strength of data augmentation, called CUDA: CUrriculum of Data Augmentation for long-tailed recognition. CUDA can simply be integrated into existing long-tailed recognition methods. We present the results of experiments showing that CUDA effectively achieves better generalization performance compared to the state-of-the-art method on various imbalanced datasets such as CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018. 


Published as a conference paper at ICLR 2023
CUDA: CURRICULUM OF DATA AUGMENTATION FOR LONG-TAILED RECOGNITION
Sumyeong Ahn Jongwoo Ko", Se- Se-Young Yun
KAIST Al
Scoul, Korea
(sumyeongahn, jongwoo ko, @usseyoung
ABSTRACT
Class imbalance problems frequently occur in real-world tasks, and conventional deep learning algorithms are well known for performance degradation on imbal- anced training datasets. To mitigate this problem, many approaches have aimed to balance among given classes by re-weighting or re-sampling training samples. These ce-balancing methods increase the impact of minority classes and reduce the influence of majority classes on the output of models. However, the extracted repre- sentations may be of poor quality owing the limited number of minority samples. To handle this restriction, several methods have been developed that increase the representations of minority samples by leveraging the features of the majority samples. Despite extensive recent studies, no deep analysis has been conducted on determination of classes be augmented and strength of augmentation has been conducted. In this study, we first investigate the correlation between the degree augmentation and class-wise performance, and find that the proper degree augmentation must be allocated for each class to mitigate class imbalance problems. Motivated by this finding, we propose a simple and efficient novel curriculum, which is designed to find the appropriate per-class strength of data augmentation, called CUDA: CUrriculum of Data Augmentation for long-tailed tailed recognition. CUDA can simply be integrated into existing long-tailed recognition methods. We present the results experiments showing that CUDA effectively achieves better general- ization performance compared the state-of-the-at art method on various imbalanced datasets such as CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018
INTRODUCTION
Deep neural networks (DNNs) have significantly improved over the past few decades on a wide range of tasks (He et al. 2017 Redmon Farhadi 2017 Qi et al. 2017). This effective performance made possible by come from well-organized datasets such as MNIST (LeCCun et al. 1998). CIFAR- 10/100 (Krizhevsky et al. 2009), and ImageNet (Russakovsky et al. 2015). However, as Van Horn et al.) (2018) indicated, gathering such balanced datasets is notoriously difficult in real-world applications. In addition, the models perform poorly when trained on an improperly organized dataset, e.g., in cases with class imbalance, because minority samples can be ignored due to their small portion.
The simplest solution to the class imbalance problem is to prevent the model from ignoring minority classes. To improve generalization performance, many studies have aimed emphasize minority classes or reduce the influence of the majority samples. Reweighting (Cao et al.) 2019 Menon et al.. 2021) resampling (Buda al. 2018 Van Hulse al. 2007) two representative methods that have been frequently applied to achieve this goal. (i) Reweighting techniques increase the weight of the training loss of the samples in the minority classes. (ii) Resampling techniques reconstruct
class-balanced training dataset by upsampling minority classes or downsampling majority classes.
Although these elaborate rebalancing approaches have been adopted in some applications. limited information on minority classes due to fewer samples remains problematic. To address this issue, some works have attempted to spawn minority samples by leveraging the information of the minority
"Two authors contribute equally
Code available at Link

---

Published as a conference paper at ICLR 2023
Major
Minor
Strong Augment
Weak Augment
Strong Augment
Major Minor
Aug. Strength
Aug. Strength
Balanced
Augmentation Strength minor, Major,
Strenggtt
Imbalanced (IR 100)
Malor acc.
Augmentation strength index
minor acc.
Ja 1008 DE2
Better "minor (Malor) average accuracy from strong augmentation MMjorr formin
Figure 1: Motivation of CUDA. If one half of the classes (e.g., class indices in 0-49) are strongly augmented, the averaged accuracy of the other half of the classes (i.e., in 50-99) increases. The heatmaps in the first and the second rows present the accuracy for class indices in 0-49 and 50-99, respectively. Also, each point in the plots means the accuracy under corresponding augmentation strength. This phenomenon is also observed in the imbalanced case in which the top first half (major; the first row) have more samples than the second half classes (minor; the second row). Setup and further analysis are described in Appendix A
samples themselves. For example, Chawla et al. (2002); Ando & Huang (2017) proposed method to generate interpolated minority samples. Recently (Kim et al. 2020 Chu et al. 20200 Park et al. 2022)
suggested enriching the information of minority classes by transferring information gathered from majority classes to the minority ones. For example, Kim et al. (2020) generated balanced training dataset by creating adversarial examples from the majority class to consider them minority.
Although many approaches have been proposed to utilize data augmentation methods to generate various information about minority samples, relatively few works have considered the influence the degree of augmentation of different classes on class imbalance problems. In particular, few detailed observations have been conducted to which classes should be augmented and how intensively.
To this end, we first consider that controlling the strength of class-wise augmentation can provide another dimension to mitigate the class imbalance problem. In this paper, we use the number of augmentation operations and their magnitude to control the extent of the augmentation, which we refer to herein as its strength, e.g., strength parameter of means that two randomly sampled operations with a pre-defined magnitude index of are used.
Our key finding is that class-wise augmentation improves performance in the non-augmented classes while that for the augmented classes may not be significantly improved, and in some cases, perfor- mances may even decrease. As described in Figure regardless of whether a given dataset is class imbalanced, conventional class imbalance methods show similar trends: when only the major classes are strongly augmented (e.g., strength 4), the performance of majority classes decreases, whereas that for the minority classes have better results. To explain this finding, augmented classes get diversified feature representation, preventing the growth of the norm of a linear classifier for corresponding classes. As result, the softmax outputs of the strongly augmented classes are reduced, and thus the accuracy of those classes decreases. It is described in Apppendix A This find the proper augmentation strength for each class improve the performance
further find that strongly
result motivates for other classes while maintaining its own performance.
We propose simple algorithm called CUrriculum of Data Augmentation (CUDA) to find the proper class-wise augmentation strength for long-tailed recognition. Based on our motivation, we have to increase the augmentation strength of majorities for the performance of minorities when the model successfully predicts the majorities. On the other hand, we have to lower the strength of majorities when the model makes wrong predictions about majorities. The proposed method consists of two modules, which compute a level-of- of-ea-if- score for each class and leverage the score determine the augmentation. Therefore, CUDA increases and decreases the augmentation
Contribution.
2

---

Published as a conference paper at ICLR 2023
strength of the class that successfully and wrongly predicted by the trained model. To the best of our knowledge, this work is the first to suggest a class-wise augmentation method to find a proper augmentation strength for class imbalance problem.
We empirically examine performance of CUDA on symthetically imbalanced datasets such as CIFAR- 100-LT (Cao et al., 2019), ImageNet-LT (Liu et al. 2019), and a real-world benchmark, iNaturalist 2018 (Van Horn et al.] 2018). With the high compatibility of CUDA, we apply our framework to various long-tailed recognition methods and achieve better performance compared to the existing long-tailed recognition methods. Furthermore, we conduct an extensive exploratory analysis to obtain a better understanding of CUDA. The results of these analyses verify that CUDA exhibits two effects that mitigate class imbalance, including its balanced classifier and improved feature extractor.
RELATED WORKS
Long-tailed Recognition (LTR). The datasets with class imbalances can lead DNNs to learn biases toward training data, and their performance may decrease significantly on the balanced test data. To improve the robustness of such models to imbalance. LTR methods have been evolving in two main directions: (1) reweighting (Cui et al.; 2019 Cao et al. 2019 Park et al., 202T) methods that reweight the loss for each class by a factor inversely proportional to the number of data points, and (2) resampling methods (Kubat al.) 1997; Chawla et al. 2002, Ando & Huang 2017) that balance the number of training samples for each class in the training set. However, studies along these lines commonly sacrifice performance on majority classes to enhance that on minority classes, because the overfitting problem occurs with limited information on minority classes as result of increasing the weight of a small number of minority samples.
Several methods have recently been developed to alleviate the overfitting issues in various categories:
(1) two-stage training (Cao et al.] 2019; Kang et al. 2020; Liu et al., 2019), (2) ensemble meth-
ods (Zhou et al. 2020a Xiang et al. | [220 Wang et al. 2021 Cai al et al. (021). and (3) contrastive
learning approach (Kang et al. | Cui et al. 2021 Zhu et al. 2022 Li et al. 2022a b). To
re-balance the classifier layers after achieving good representation on the imbalanced training dataset in an early phase, Cao et al. (2019) proposed deferred resampling (DRS) and reweighting (DRW) approaches. KKan et al. (2020) decoupled the learning procedure into representation learning and training linear classifier, achieved higher performance than previous balancing methods. |Wang et al. (2021) and et al. (2021) suggested efficient ensemble methods using multiple experts with a routing module and a shared architecture for experts to capture various representations. Liu et al. (2022) found that self-supervised representations are more robust class imbalance than supervised representations, and some works have developed supervised contrastive learning methods (Khosla
ct al.
2020) for imbalanced datasets (Cui al., 202 Zhu al.. 2022 Li et al. 2022b).
Another line of research has considered augmentation methods in terms of both input and feature spaces (Kim et al., 20200 Chu al. 2020; Li et al., 202T). Recently, (Pak et al. (2022) mixed minority and majority images by using CutMix with different sampling strategies to enhance balancing and robustness simultaneously. These methods commonly focus on utilizing the rich context of majority samples to improve the diversity of minority samples. Zhou et al. (2022) proposed an augmentation-basse contrastive learning method which boosts memorization of each samples for long- tailed learning. Moreover, these augmentation-bann based methods relatively in easy to apply orthogonally with other LTR methods.
Data Augmentation (DA). DA has been studied to mitigate overfitting which may occur due to a lack of data samples. Some works have been proposed erase random parts of images enhance the generalization performance of neural networks (DeVries & Taylor 2017; Zhong et al. 2020
Kumar Singh & Jae Lee| [217] Choe & Shim 2019). Recently, variants of Mix (Zhang et al. 2018
have been proposed; this method combines two images with specific weights (Tokozume et al. 2018
Guo et al. 2019 Takahashi et al., 22188] Deeesss & Taylor 2017 Verma et al., 2019). By aggregating two approaches, CutMix (Yun et al. 20199) was proposed to erase and replace a small rectangular part of an image into another image. In another line of research, methods have been proposed to automatically configure augmentation operations (Cubuk et al. 2019; Lim et al., 2019 Li et al. 2020b) Hataya et al. 2020 Gudovskiy et al.] (021). In addition, Cubuk et al. (2020) randomly selected augmentation operations using the given hyperparameters of the number of sampling augmentation and their magnitudes. Recently, class-wise or per sample auto-augmentation methods have also been proposed (Cheung & Yeung 2021 Rommel al. 2022
3

---

Published as a conference paper at ICLR 2023
Data Augmantation Strength paramatart
Train with CUrriculum of Data Augmantation
Measuring Lot.
Sampling (A)) Cat(AS
Update L
(E) ky: S-iitShitt
Model
12-1
(2 Ny Brightees
Dihenatee
Ky: S-Sti
Figure 2: Algorithm overview. CUDA is composed of two main parts: (1) strength-based augmentation and (2) Sevel-of-Learinng (LoL) score. To control the difficulty of augmented images, strength-basee augmentation utilizes two values, the number of augmentations and their magnitudes. We use the strength-based augmentation module to score the Lol.. Based on the measured Lol. score, CUDA generates adequately augmented images for LTR algorithms.
3 CURRICULUM OF DATA AUGMENTATION FOR LONG-TAILED RECOGNITION
The core philosophy of CUDA is to "generate augmented sample that becomes the most difficult sample without losing its original information." " In this section, we describe design CUDA in terms of two parts: (I) method to generate the augmented samples based the given strength parameter, and (2) a method to measure a Level-of-Learning (LoL) score for each class.
3.1 PROBLEM FORMULATION OF LONG-TAILED RECOGNITION
Suppose that the training dataset D {1 y/ ) Ji=1 and their corresponding labels E {l...., C}. D. C D is set of class c, i.e. D. {(I, v)ly =
composed of images with size d, x4 Rd.
c, (xx, y) E D). Without loss of generality, we assume [D, 2 [D2] Σ > (Del. where (D] denotes the cardinality of the set D. We denote the Nmmx = [Di and N |Del. LTR algorithms, AITR D), mainly focus on training the model S' with parameter 0 when the class distribution of training dataset Pumin (y) and test dataset Piest (y) are not identical. More precisely, Prain (y) is highly
imbalanced while P Pres((y) balanced, i.e., uniform distribution.
3.2 CURRICULUM OF DATA AUGMENTATION
In this section, describe proposed DA with strength parameter, and the methods used to measured the LoL score. Then, we integrate the two methods in a single framework to propose CUDA.
DA with strength parameter. Let us assume that there exist pre-defined K augmentation opera- tions. We utilize visual augmentation operations which is indexed as h E {1, ... K), e.g., Gaussian blur, Rotation, Horizontal flip. Each augmentation operation OmT(())) Rd - I has its own pre- defined augmentation magnitude function m& where the strength parameter & E (0. ..., S}. These operations are described in detail along with each magnitude functions in in/Appendix D
Given an augmentation strength parameter and an input image x, we model sequence of augmen- tation operations ( s) as follows:
Ox: 5) Ome,, (+)
mly (5) (2), k; ~ CatCKK
Vi = (1.....8), s),
where, Cat(-) and U(-)) denote categorical and discrete uniform distributions, respectively. The sequential augmentation operation O(z; 8) samples operations from the categorical distribution when the probability of seeing the operations follows uniform distribution. As depicted on the left side Figure suppose that the random sampled augmentations k1, k2, and ky are brightness. X-shift, and Y-shift, respectively. Then, Oxx; 3) outputs an image in which bright is raised by Manigh (3) and moved by sin (3) the x-axis and shifted by my the y-axis.
4

---

Published as a conference paper at ICLR 2023
Algorithm CUrriculum Data Augmentation
Algorithm Via.: Update Lol. score
Input: LTR algorithm Armlf, P), training dataset
Input: De, L, Jo T.
D == [(#; de) (i) j-m train epochs E, aug. probability Paug threshold y, number of sample coefficient T.
Output: updated L
Initialize: check =
Output: trained model Initialize: L. 4 (},
for 1 S. L
1 Varnal (Dr, Jo. 4/
for
do Update = (De 10: T.
Sample D. C D2 st. D. = 7(--1)
Ve Generate Druma = {(T, Villi 9h) D) where
1/ A-g.
Compute Excry A(((0)/)))))
if v S + then
10(0, Lig)
with prob. Pang otherwise.
check break end end
3
Run LTR algorithm using DEUDA, ie., ALTO (Ja, DEUDA)-
if check = 1 then J. + L+1
end
else L L-1
Level-of- (LoL). To control the strength of augmentation properly, we check whether the model correctly predict augmented versions without losing the original information. To enable this, we define the LoI. for each class e at epoch e, i.e, Lg which is adaptively updated as the training continues as follows:
L = (Do Le- [o.n,T))
where
[L-- +1
if Veorea (Do, fo, T) TT(+1) VI e (0, (0, -E-11
Vial. (D L fo,7,,
I. 1 otherwise
Here, E [0, 1] is threshold hyperparameter, T is coefficient of the number of samples used to updating LoL. Veorrest is a function which outputs the number of correctly predicted examples by the model Jo among / + randomly augmented samples with strength I. Veorrea is defined as:
(Connaa(ee. fe. = E.ED; ((((O((:)))=)) where D'd C De.
Note that D' is randomly sampled subset of D. with replacement and its size is T(I + 1).
The key philosophy of this criterion is two fold. (1) If samples in the class are trained sufficiently with an augmentation strength of Le, the model ready to learn a more difficult version with augmentation strength of L <- L + 1. In contrast, if the model predicts incorrectly, it should re-learn the easier sample with an augmentation strength of I + I 1. (2) As the strength parameter increases, the number of candidates for the sequential augmentation operation O(:; L) increases exponentially. For example, the amount of increment N 1) when L is increases to 1. To control the LoL in large sequential augmentation operation space, we take more random samples to check as the strength parameter gets bigger. In our experiments, linearly increasing the number of samples evaluate corresponding the strength with small additional computation time was sufficient. VoL is described in Figure 2) and Algorithm 2
Curriculum of DA. By combining two components, including DA with strength parameter and LoL, our CUDA provides class-wise adaptive augmentation to enhance the performance of the others without losing its own information. As shown in Figure 2 and Algorithm [ measure the LoL score Le for all classes in the training dataset to determine the augmentation strength for every epoch. Based on L., generate the augmented version (z:; L.) for augmented samples. Additionally, we randomly use the original sample instead of the augmented
D. and train the model with the
sample with probability Paug that the trained models do not forget the original information. In
our experiments, this operation improved performance robustly on wide range of Paug values, The results are provided in Section
Advantage of CUDA design. Our proposed approach mainly has three advantages. (1) CUDA adaptively finds proper augmentation strengths for each class without need for validation set, (2) Following the spirits of existing curriculum learning methods (Hacohen Weinshall, 2019 Zhou et al. 2020b, Wu et al.) 202I), CUDA enables modeling by first presenting easier examples earlier during training to improve generalization. This encourages the model to learn difficult samples (i.e., within high augmentation strength) better. (3) Moreover, owing to the universality of data augmentation, CUDA is easily compatible with other LTR algorithms, such as (Cao et al. 2019 Ren
et al. 2020 Wang et al. 2021)
5

---

Published as a conference paper at ICLR 2023
Table 1: Validation accuracy on CIFAR-100-LT dataset. i are from Park al. (022)) and f. * are from the original papers (Kim et al. 2020 Zhu et al. 2022). Other results are from our implementation. We format the first and second best results as bold and underline. We report the average results of three random trials.
Algorithm
Imbulance Ratio (IR)
Statistics (JR 100)
100
30
IU
Many
Med
Few
CE
38. 7
13.420a
56.6.9.8
06.250.
37 3...
8.200.
CF: + CMO
Park
2022
42.0
47.0+0.5
60.0=a.4
29.4
41.220.6
11.3g07
CE + CUDA
12.7 12.7404
47.220.4
09.6...4
71.620s
42.3
9.4
CMO CUDA
13.5
18.
60.0...
70.0
43.4.0.
12.7
DEE (Cau el
2019
41.4
45.5100
57.820.
62.820.
41.7207
16.1
CE-DRW + Remix Chou et al. 2020
15.8
49.5
59.2
CE-DRW CUDA
47.746
32.4
64.3
49.2a0.
26.7
L.DAM-DEWW (Cao et al. 2019
42.540
47.420.5
57.6g0.1
02.820.5
42.320.6
1300g0.7
1.DAM +
(Kim 2020
43.5
LDAM A.DA--DW-- + CUDA
47.6
51.140.4
58.4-01
67.3.0.
50.4
21.4
BS (Ren et al. 2020
13.3
16.9
58.3...4
61.6go8
12.3gn.
23.010
BS CUDS
17.7 17.7
32.120.4
61.720.
63.3ga.4
48.4gn
28. 28.7
RIDE (3 experts) Wang et al. 2021
48.6
51.4
59.8
RIDE (3 experts)
49.7g02 2
52.7
60.2ga.
67.7g06
51.5gos
26.7206
RIDE CMO Park et al. 2022
50.0
53.0
60.2
RIDE+ CMO
49
33.0
58.92g.
67.3gg.
51.3go6
28. 28.1
RIDE (3 experts) + CUDA
50.710.2
33.719.
60.2.m.1
69.2
62.81m.2
27.31os
BCT. 7hu et al. 2022
$1.0
54.9
64.4
67.2
63.1
32.9
CCDA
52.3.o.3
56.2.0.
64.6g..1
66.4g0.
54.2.0.6
33.920a
4 EXPERIMENTS
In this section, we present empirical evaluation, the results of which demonstrate the superior perfor- mance of our proposed algorithm for class imbalance. We first describe the long-tailed classification benchmarks and implementations in detail (Section|4.3 441). Then, we describe the experimental results on several synthetic (CIFAR- 100-LT, ImageNet-LT)) and real-world (iNaturalist 2018) long-tailed benchmark datasets in Section 4.2 Moreover, we conduct additional experiments to obtain a better understanding of CUDA, and this analysis is provided in Section
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate CUDA the most commonly used long-tailed image classification tasks: CIFAR-100-LT (Cao et al. 2019). ImageNet-LT (Liu et al. 2019). and iNaturalist 2018 (Van Horn et al. 2018). CIFAR- 100-LT and ImageNet-LT are provided with imbalanced classes by synthet- ically sampling the training samples. CIFAR- is examined with various imbalance ratios {100, 50, 10), where an imbalance ratio is defined Nmax/ fmax/Nmin iNaturalist 2018 is large-scale real-world dataset includes natural long-tailed imbalance. We utilize the officially provided datasets.
Baselines. We compare CUDA with previous long-tailed learning algorithms, including cross-entropy loss (CE), two-stage approaches: CE-DRW (Cao et al., 2219) and cRT (Kang et al.) 2020), balanced loss approaches: LDAM-DRW (Cao et al. 2019) and Balanced Softmax (BS: et al. 2020), the ensemble method: RIDE with three experts Wang et al. (02TT. resampling algorithms: Remix (Chou 2020) and CMO (Park et al. 2022). and contrastive learning-basee approach: BCL (Zhu et al. 2022 We integrate CUDA with CE. CE-DRW LDAM-DRW, BS, RIDE. and BCL algorithms. For longer epochs, we compare CUDA with PaCo (Cui et al. 2021). BCL, and NCL (Li et al.) 2022a). by combining CUDA with BCL and NCL. For a fair comparison of the computational cost, we train the network with the official one-stage implementation of RIDE (i.e., without distillation and routing).
eet al.
Implementation. For CIFAR- 100-LT dataset, almost all implementations follow the general setting from Cao et al. (2019), whereas cRT (Kang et al.) 2020). BCL, NCL and RIDE follow the settings used in their original implementation. Following Cao et al. (019), we use ResNet-32 (He et al. 2016) as backbone network for CIFAR-100-L2 The network is trained on SGD with a momentum of 0.9 and a weight decay of 2 x 10-4 4, The initial learning rate is 0.1 and a linear learning rate warm-up is used in the first 5 epochs to reach the initial learning rate. During training over 200 epochs, the learning rate is decayed at the 160th and 180th epochs by 0.01. For the ImageNet-LT and iNaturalist, the ResNet-50 is used as a backbone network and is trained for 100 epochs. The learning rate is decayed at the 60th and 80th epochs by 0.1. As with CIFAR, for cRT, RIDE, and BCL, we follow the original experimental settings of the official released code. For the hyperparameter values of

---

Published as a conference paper at ICLR 2023
Table 2: Validation accuracy on ImageNet-LT and iNaturalist 2018 datasets, i indicates reported results from the Park et al. (2022) and $ indicates those from the original paper Kang et al. 2020 means we train the network with the official code in an one-stage RIDE.
ImageNer-Li
INaturalist
2018
Algorithm
Many
Med
All
Many
Mid
Few
CE
64.0
33.5
41.6
73.9
63.5
56.5
61.0
CE CB+ CB+CUDA
47.0a0a
13.5
17.3 47.3
74.6
61.9
57.2
625g.2
CK-DRW CC 2019
61.720
47.1a0
60.1g6.1
68.210a
07.310
67
61.8en
48.3
30.3402
68.840.
68.3
67.0en1
LWS
Kung ml. 2020
57.1
45.2
20.3
47.7
65.0
66.3
66.5
63.9
Kang
cKT
CCDD
62.3.n1
47:2
28.4
50.2.6.
68.2
67.810.
66.4
67.3gn1
LDAM DRW
Can al. 2019
60.4
48.9
30.7
49.8
66.1
LDAM-DRW
63.1
68.0.0
11.
61.4.6.
67.8
07.6
16.7
67.2,00
BS [Ren et
2020
6111207
48.5g02
31.8404
50.9g6
65.3g0
67.329
67.2209
88 FCDDS
61.9201
40.2.00
32.310.1
51.66.01
07.6101
08.310
08.31.
68/2.1
espertsl (Wame
6L8an1
50.8004
30.0.0.3
53.0g0a
70.4g0
118
1L.7
710aa1
RIDE CMO (Park al 2022
65.6
50.6
34.8
64.0
68.0
70.6
72.0
70.9
RIDE (3 experts CODA
65.92m
51.7
30.9
54.7ea
70.740
12.3
72.7=02
72.4
BCL 22n 2022
65.3nn
36.3.oo
55.6-02
60.5
72.4
71.7a0a
718821
68.5 in)
53.9
36.6gmm
3,.1
20.9
22.8.m1
72.0
22.
Table 3: Comparison for CIFAR-LT-100 per- formance on ResNet-32 with 400 epochs.
Table 4: Augmentation analysis on CIFAR-100-LT with IR 100.
AA (Cubuk et al. (019), FAA (Lim et al. 2019), DADA eLi al. 2020b), and RA (Cubuk et al. 2020) with = I, 2 policies are used. C, S, I represent CIFAR, SVHN, and ImageNet policy.
Algorithm
Imbolance Ratio
1000
30
Vanilla
AA
FAA
DADA
RA
CUDA
PaCo
52.0
50.0
35.7
40.7
40.8
41.2
41.4
45.5
44.8
45.7
NCL
54.2
58.2
LDAM-DRW
42.5
37.0
44.7
44.9
46.6
45.8
45.9
46.5
44.0
47.2
BCL CUDA
53.5
57.4
BS
45.3
47.0
46.1
45.5
46.5
45.0
45.0
46.9
46.2
47.7
NCL CUDA
RIDE expens)
49.7
40.5
47.3
45.5
49.8
50.6
60.4
50.5
47.9
50.7
CUDA, we apply a Paug of 0.5 and T of 10 for all experiments. For y, we set the values as 0.6 for CIFAR- 100-R-10-LT 1T0-LT and 0.4 for ImageNet-LT and iNaturalist 2018. The detailed implementation for baselines are in Appendix B
4.2 EXPERIMENTAL RESULTS
In this section, we report the performances of the methods compared on the CIFAR-100-LT, ImageNet- LT, and iNaturalist 2018. We include four different categories of accuracy: all, many. medm) and few. Each represents the average accuracy of all samples, classes containing more than 100 samples, 20 to 100 samples, and under 20 samples, respectively.
CIFAR-100-LTT. In Table 1l we report the performance when CUDA is applied to the various algorithms: CE, CE-DRW (Cao et al. 2019), LDAM-DRW (Cao et al. 2019), BS (Ren et al. 2020 RIDE WWang et al. 2)21) with 3 experts, RIDE+CMO (Park al. 2022) and BCL (Zhu et al. 2022
Compared to the cases without CUDA, balanced validation performance is increased when we apply the proposed approach.
Recently, some works (Cui et al. 2021 Alshammari et al. 2022 Zhu et al. 2022 Li et al. 2022a)
have shown impressive performances with diverse augmentation strategies and longer training epochs. For a fair comparison with these methods, we examine CUDA using the same experimental setups from PaCo (Cui al. 202T 400 epochs with batch size of 64). Table using CUDA can enhance LTR performance compared to the other baselines. In particular, CUDA with NCL obtains the best performance over 400 epochs. As noted by Li et al. (2022a), the NCL algorithm utilizes six times as much memory compared to the vanilla architecture with three experts. Hereinafter in large-scale benchmarks, we focus on the cases with similar network size.
that augmented images
ImageNet-LT and iNaturalist 2018 To evaluate the performance of CUDA on larger datasets, we conduct experiments on ImageNet-LT (Liu et al. 2019) and iNaturalist 2018 (Van Horn et al.) 2018). Table 2 summarizes the performance of various LTR methods and the performance gain when integrated with CUDA. Our proposed method consistently improves performance regardless of the LTR method and target dataset by simply adding class- wise data augmentation without complicated methodological modification. Additionally, evaluate the performance gain of CUDA other
7

---

Published as a conference paper at ICLR 2023
DMC
hyal
000 180
000
(a) CE
(b) CE-DRW
(c) LDAM-DRR DRW
(d) BS
(c) RIDE
Figure 3: Analysis of how CUDA improves long-tailed recognition performance, classifier weight norm (top row) and feature alignment gain (bottom row) of the CIFAR- 100-LT validation set. Notably that weight norm and feature alignment represent class wise weight magnitude of classifier and ability of feature extractor, respectively. The detailed analysis described in Section 4.3
Locch
Loce
(a) CE
(b) CE-DRW
(c) LDAM-DRW
(d) BS
(c) RIDE
Figure 4: Evolution of LoL score on various algorithms, CE, CE-DRW, LDAM-DRW, BS, and RIDE.
architectures, we experiment with CUDA ImageNet-LTT with ResNet-10 ResNeXt-50 (Xie et al. 2017), as reported in Appendix C
(Liu et al. 2019) and
4.3 ANALYSIS
We design our analyses to answer the following questions. (1) How does CUDA perform? (2) Does CUDA perform better than other augmentation methods? (3) How does LoL score change over training epochs when combined with various LTR methods? (4) Which part of CUDA is important improved performance? These analyses provide additional explanations to understand CUDA. All experiments with imbalance ratio of 100.
are conducted on
How does CUDA mitigate the class imbalance problem? To deeply understand CUDA, we observe two types of metries: (1) variance of weight L1-Norm of linear classifier between each class (2) feature alignment gain for each class (i.e., cosine similarity with and without CUDA) on validation dataset. The classifier weight norm is usually used to measure how balanced the model consider the input from a wise perspective (Kang et al.) 2020: Alshammari et al., (022). Feature alignment, especially feature cosine similarity amongst samples belonging the same class, is a measure of the extent to which the extracted features are aligned (Oh et al. 202T). As shown in Figure 3] CUDA has two forces for alleviating imbalance. For all cases, CUDA reduces the variance of the weight norm (i.e., balance the weight norm), and thus the trained model consider the minority classes in a balanced manner. Note that because L.AM-DRW and RIDE utilize a cosine classifier (i.e., utilizing 1.2 normalized linear weight), their standard deviation scale is quite different from those other methods. Because LDAM-DRW, BS, and RIDE include balancing logic in their loss function, they exhibit lower variance reduction compared to the CE and CE-DRW. Second, shown in the bottom row in Figure 3, CUDA obtains feature alignment gains for almost all classes. This shows that CUDA facilitates network learn to extract meaningful features.
Compared with other augmentations. To verify the impact of CUDA, we examine the other augmentation methods as follows. We compare five augmentation methods, including AutoAugment (AA. Cubuk (AA.CCubuk et al. 2019). Fast AutoAugment (FAA, Lim al. 2019). DADA (Li et al. 2020b), Randeuggent (RA, Cubuk et al. 2020), and the proposed method CUDA. Because AA, FAA, and DADA provide their policies searched by using CIFAR, SVHN (for AA), and ImageNet, we leverage their results. Furthermore, RA suggests using their parameter (n, m) = (1,2) for CIFAR, and we follow their guidelines. As shown in Table 4 even though the automated augmentation methods use additional computation resources to search, CUDA outperforms the other pre-searched augmentations. This shows that CUDA is computationally efficient.
Dynamics of LoL score. We evaluate how LoL scores vary with algorithms: CE, CE-DRW, LDAM- DRW, BS, and RIDE. Note that we set lower class index (i.e., 0) as the most common class (i.e. the
8

---

Published as a conference paper at ICLR 2023
51.
51.0
00.50
Baok Ont
CURA Oir
10.5
10.5
20.4
20.0
DA2A
50.2
50.0
05
20
00
86
RCE
Aug Pwo
Nkm feet
Ascept FOR
Algurtin
(a) Aug prob. Pang-
(b) Num Test T.
(c) Accept rate 7
(d) Curriculum.
Figure 5: Additional analysis of CUDA, (a) sensitivity of augmentation probability Paug. (b) sensitivity analysis of number of sample coefficient T,, (c) sensitivity of acceptance threshold y, and (d) impact of curriculum. The dotted lines in (a), (b) and (c) represents the performance of CE.
number of samples is 500), while an index of 100 represents the rarest class (ie., with five samples). As described in Figure 4, training progressed, the LoL score of all algorithms increase. After learning rate decay (i.e., 160 epoch) all algorithms are able to learn to classify minority classes more easily than before. In particular, except for BS, the majority classes of most algorithms show steep increment. The reason that BS exhibit a similar increasing speed for majority and minority classes is that it includes module balance the impact of majority and minority samples. Furthermore, found that CE-DRW and BS have similar end average accuracy in the case of applying CUDA but different LoL. score dynamics. We can conclude that LoL. score on one category of classes has high correlation with the performance of opposite classes from the observation that CE-DRW has higher and lower performance gain for many and few, respectively, than BS.
Parameter sensitivity. For further analysis, we conduct a sensitivity analysis of hyperparameters in CUDA. More precisely, we study three kinds of parameters, including augmentation probability Paug (Figure Sa), number of tests T (Figure 5b), and LoL update threshold y (Figure Sc). We examine CUDA case with RIDE and the remainder of the hyperparameters
each hyperparameter sensitivity are fixed to the default values in Section 1 All results show that the performance gains of CUDA decreases if the parameters are adjusted make the augmentation too strong weak. For example, the augmentation strength of all classes steeply increases when y becomes small. The strength cannot increase when 7 becomes large, and thus it cannot improve the performance of the model. Moreover, as shown in Figure 5b) the performance of CUDA increases as T increases. However, larger T spends computational overhead, we set T as 10 and obtained cost-effective performance gain.
Impact of curriculum. In addition to studying the impact of CUDA, we examine its performance component-wisst In particular, we test the case where class-wise augmentation strength is searched based on the hyperparameter optimization algorithm. We check five cases overall: baseline algorithm, hyperparameter optimization (HO), re- scarched DADA for CIFAR- 100-L- CUDA without curriculum, (i.e., re- te-trrinnin utilizing the final augmentation strength of CUDA), and CUDA. We provide detailed description for each method in Appendixx E As described in Figure 5d CUDA finds better augmenta- tion strengths compare to the hyperparameter search case. This means that CUDA exhibits not only lower searching time but also obtains better augmentation strength. Moreover, by comparing the performance of with or without curriculum, the curriculum also can provide additional advance to the model achieve better generalization. Additionally, as Figure 4 lower augmentation strength the beginning of training is more effective than static higher augmentation strength. These results are consistent with the results of previous studies on curriculum learning methods (Zhou et al., (0226))
CONCLUSION
In this study, we proposed CUDA to address the class imbalance problem. The proposed approach is also compatible with existing methods. To design a proper augmentation for LTR, we first studied the impact of augmentation strength for LTR. We found that the strength of augmentation for a specific type of class (e.g., major class) could affect the performance of the other type (e.g., minor class). From this finding, we designed CUDA to adaptively find an appropriate augmentation strength without any further searching phase by measuring the LoL score for each epoch and determining the augmentation accordingly. To verify the superior performance of proposed approach, we examined each performance with various methods and obtained the best performance among the methods compared, including synthetically generated and real-world benchmarks. Furthermore, from our analyses, we validated that our CUDA enhanced balance and feature extraction ability, which can consistently improve performance for majority and minority classes.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) 200-009----007,5 Artificial Intelli- gence Graduate School Program (KAIST), 10%) and the Institute of Information & communications Technology Planning Evaluation (IIT)) grant funded by Korea government (MSIT) (No. 2022- 0-00871, Development of AI Autonomy and Knowledge Enhancement for Al Agent Collaboration, 90%)
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897-6907, 2022.
Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 770-785. Springer, 2017.
James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning, pp. 113-123. PMLR, 2013.
Mateusz Buda, Atsuto Maki, and Macicj A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, (06:249-259, 2018.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribttion-awaa margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Tsz-Him Cheung and Dit- Yan Yeung. Adaaug: Learning class-an and instance-adaptle data augmenta- tion policies. In International Conference Learning Representations, 2021.
Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2219-2228, 2019.
Hsin-Ping Chou, Shih-Chich Chang, Jia- Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference Computer Vision, pp. 95-110. Springer, 2020.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling, Feature space augmentation for long-tailed data. European Conference on Computer Vision, pp. 694-710 Springer, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu. and Jiyy Jia. Parametric contrastive learning. In Proceedings of the IEFE/CVF international conference computer vision, pp. 715-724, 2021.
10

---

Published as a conference paper at ICLR 2023
Yin Cui, Menglin Jia, Tsung- Lin, Yang Song, and Serge Belongie. baass-bllaced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2019.
Terrance DrViies and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Denis Gudovskiy, Luca Rigazio, Shun Ishizaka, Kazuki Kozuka, and Sotaro Tsukizawa. Autodo: Ro- bust autoaugment for biased data with label noise via scalable probabilistic implicit differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16601-16610, 2021.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 371--22-37 2019.
Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In International Conference on Machine Learning, pp. 2535-2544. PMLR, 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. In European Conference on Computer Vision, pp. 1-16. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URL httpps::///peeeeee.e.e net/ forum?
[d=rigRTcVER
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021. URL
httpp:://openreviee..eeete htt.://wwww......ooooooooooooaa.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural 2020.
Information Processing Systems, 33:18661-188
Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13995, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Ieml, volume 97, pp. 179. Citeseer, 1997.
Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek- Forcing a network to be meticulous for suaely-s object and action localization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3524-3533, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233 1998.
Jun Li, Zichang Tan, Jun Wan, Zhen Lei, and Guodong Guo. Nested collaborative learning for Long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022a.

---

Published as a conference paper at ICLR 2023
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar, A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230-246. 2020a.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang. Feng Qiao, and Xinjing Cheng. Metasaug:
Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 5212-5221, 2021.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6918-6928, 6928, 2022b.
Yonggang Li, Guosheng Hu, Yongtao Wang. Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In European Conference on Computer Vision, pp. 580-595. Springer, 2020b.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05111 2018.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2022. URL hittp:://ppereet http:/ewwweeeeooeeeeeeeeeeooeeee (ttp:/oruumm114272200000
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537 2537-2546, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021. httpstt::: wow.eeveeeeee.eeeeeeeeeeee net/ (ttp://oorommnnn///ooooeele
Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. (BOIL): Towards representation change for few-shot learning. In International Conference Learning Representations, 2021.
URL htttps://pperreiieee htwp:oppwrreiieeeeeeeeeeeeeeeeee htt/://oommmiimmmmoommm
Seulki Park, Jongin Lim, Younghan Jcon, and Jin Young Choi. mallence-balaneee loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 735-744, 2021.
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the JEEF/VV Conference on Computer Vision and Pattern Recognition, pp. 6887-6896, 2022.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271 2017.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed
visual recognition. Advances in neural information processing systems, 33:4175-4188 2020.
Cédric Rommel, Thomas Moreau, Joseph Paillard, and Alexandre Gramfort. CADDA: Class-wise au-
tomatic differentiable data augmentation for EEG signals. In International Conference on Learning Representations, 2022. URL
net/ www.moorrmiioooo 6IYp 3-0--35---00 xJ
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115())221 1--22,, 2015.
12

---

Published as a conference paper at ICLR 2023
Saptarshi Sinha, Hiroki Ohashi, and Katsuyuki Nakamura. Class-diffieuly based methods for long-tailed visual recognition. International Journal of Computer Vision, 130(10):2517-2531 2022.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Ricap:
Random image cropping and patching data augmentation for deep emns. In Asian conference on machine learning. pp. 786-798. PMLR, 2018.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifica- tion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
5486-5494, 2018.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 8769-8778, 2018.
Jason Van Hulse, Taghi M Khoshgoftaar, and Amri Napolitano. Experimental perspectives on learning from imbalanced data. In Proceedings of the 24th international conference on Machine learning. pp. 935-942, 2007.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference Machine Learning, pp. 1438-6447. PMLR, 2019.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by rout- ing diverse distribution-awware experts. In International Conference on Learning Representations, 2021. URL
net wtt.:eeeoooorummdddr133r...
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? International
Conference Learning Representations, 2021. htt/:tttps///ppeeeeee.eee.e
Id=tW40EInpnii
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018. URL https
//ppenreview.. wwweerrvieewnet/eeeeee www.//ett/////eeeeeee 1d=
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmenta- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Boyan Zhou Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 9719-9728, 2020a.
Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hardness. Advances in Neural Information Processing Systems, 33:8602-8613, 2020b.
Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pp.
PMLR, 2022.
13

---

Published as a conference paper at ICLR 2023
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6908-6917, 2022.
14

---

Published as a conference paper at ICLR 2023
Appendix
CUDA: Curriculum of Data Augmentation for Long-tailed Recognition
Owing to the page limitation of the main manuscript, we provide detailed information in this supplementary follows. (I) In Appendix A, summarize the experimental setup of Figure and further explain why augmentation (2) In/Appendix B] we describe in detail our experimental setting, including dataset configuration, data preprocessing, and training implementation. (3) In Appendix C we show ImageNet-I.T L.T performance on different size and architecture networks, training time analysis, and accuracy on the balanced dataset case. (4) In Appendix D] we present in detail the augmentation operations that CUDA utilizes.
one side causes performance degradation on the opposite side.
(5) In Appendix E we describe experimental setting of Figure 5d
A DETAIL FOR FIGURE
A.1 EXPERIMENTAL SETTINGS
Major and minor group decomposition. To check the impact of augmentation on majority and minority classes, split the training dataset into two clusters. The majority cluster is the top 50 classes by sorting through the number of samples for each class. The bottom 50 classes are in the minority cluster. For simplicity, we utilize class indices of 49 as the majority and 50 to as the minority, respectively. For the balanced case, we utilize 0 to 49 classes as cluster 1, and the others as cluster 2.
Controlling augmentation strength. We set the augmentation strength as the number augmenta- tion and its augmentation magnitude by following the augmentation rule of CUDA. For example, the samples in the majority classes with magnitude parameter 4 represents that they are augmented with randomly sampled 4 augmentations with their own pre-defined augmentation magnitude.
Training setting. For heatmaps in Figure we follow the training recipe of CIFAR-100-LT for CE case, e.g,, ResNet-32, learning rate of 0.1, and so on. Further details, hyperparameters, and datasets described ssccttiion 4) and Appendix B
Tran Coe-Sim
Train Coe-Smm
Test Test
Weght Norm
Wegght Norm
Without Augment
Parial
Augment
Without Augment
Partil Augment
All Augment
Figure 6: Analysis on Balanced CIFAR-100.
Figure 7: Analysis on CIFAR-100-LT (IR 100).
A.2 ANALYSIS
Analysis for Figure 1. To figure out the reason for the phenomena Figure we conduct further analysis as shown in Figure 6 and Figure 7 Our experimental setups are as follows:
Train the networks with three augmentation strategies, respectively (without, partial, and all), then measure the class-wise feature alignment and linear classifier weight norm for all networks. (Experiment 1)
15

---

Published as a conference paper at ICLR 2023
From trained network without augmentation in Experiment 1, freeze the feature extractor and train the linear classifier layer with augmenting partial classes. Then, we measure the class-wise Ll-nom for all linear classifiers. (Experiment 2)
From the Figure 6 and Figure 7
have three observations from Experiment 1:
1.
When conduct augmentation only for partial classes (0-49 classes), the feature alignment for augmented classes of the training dataset is degraded compared to the non-augmented classes. This is because the augmentation classes have more diversified training data than non-augmentation classes, which leads to more diversification in feature space. We observe the balance between alignment between classes in the cases of without augmentation and with all augmentation since all classes have similar diversity. (See the first rows in Figure 7
2. However, all three augmentation strategies have balanced class-wise feature alignment for the same test dataset. This tendency can be observed both balanced and imbalanced datasets. This result is consistent with Kang et al. (2020). Furthermore, the values for feature alignment are increased when we conduct augmentation partially or all, compared to without augmentation. This result shows that augmentation enhances the feature extraction ability, which is consistent with conventional studies. (See the second rows in Figure 6 7
3. When we conduct augmentation only for partial classes on a balanced dataset, the class-wise weight norm of the linear classifier is larger for con-augmentation classes. This result incurs performance improvement for non-augmentation classes and reduction for augmentation classes since this linear classifier has tendeney to classify non-augmented classes with larger weight values. However, observe that class-wise weight norms balanced in "without augmentation" and "all augmentation" cases. (See the third row in Figure 6)
4. We observe that the class-wise weight norm of the linear classifier is larger for majorities for all classes that have the same augmentation strength. These results are consistent with previous works (Kang et al., 2020; Alshammari et al. 2022). However, when we conduct augmentation only for majortiee,,
class-wise weight norm is more balanced. This phe- nomenon is similar to the balanced case in that partial augmentation incurs a reduction in the norm of the linear classifier for augmented classes. (See the third row in Figure
3.5
Type (Augmented)
AMks
Type (Not augmented)
3.0
Zut's
Aw
ANF
2.5
Altly
2.0
Lower fnature consistency
Higher feature consistency
50
100
150
200
Epoch
Figure 8: Concept of the impact of lower feature
Figure 9: The difference of linear classifier norm |
alignment on linear classifier.
along training epoch.
Our observations from Experiment are highly consistent in both balanced and imbalanced datasets. The results in FFiurre 1, Figure (aandFFiuur highhy motivate the design of CUDA. Moreover, our results for Experiment 2 can explain these observations as shown in Figure 8 and Figure
We observe that the presence feature alignment degradation from augmentation, the correspond- ing norm is relatively small, shown in Figure 8 This is because in the class that has lower feature alignment, the variation of the gradient for the linear classifier is larger than in the class with high feature alignment. As shown in Figure 9 from Experiment 2, we observe that || /ww|, the norm of blasswwise difference of between current and initialize linear classifier parameters Aw := w We- have smaller value in augmented classes than con-augmented classes. From experimental analysis in Figure 6 [ and [
we can conclude that augmentation breaks the consistency of feature alignment and it makes the weight norm of the linear classifier decreases.
16

---

Published as a conference paper at ICLR 2023
B IMPLEMENTATION DETAIL IN SECTION 4
B.1 DATASET DESCRIPTION
CIFAR-100-LT. -FARR-00-LL1 is a subset of CIFAR-100. Following Wang et al. (0221 Park et al. 20022 Zhu al. 2022), we use the same long-tailed version for fair comparison. The number of samples of leth class is determined as follows: (1) Compute the imbalanced factor NN /Nmin which reflects the degree of imbalance in the data. (2) [Dx between [D1 [ Nmax and [P1o0l = min follows an exponential decay (i.e. [D&] [D1] (Nmux/ (Nmm))mm) ) The imbalance factors used in the experiment are set to 100, 50, and 10.
ImageNet-LLT. ImageNet-LT (Liu et al. 2019) is modified version of the large-scale real-world dataset (Russakovsky ct al. 2015). Subsampling is conducted by following the Pareto distribution with power value 0.6. It consists of 115.8K images of 1, 000 classes in total. The most common or rare class has 1, 280 or 5 images, respectively.
iNaturalist 2018. iNaturalist (Van Horn et al. (0018) a large-scale real-world dataset which consists of 437.5K images from 8, 142 classes. has long-tailed property by nature, with an extremely class imbalanced. In addition to long-tailed recognition, this dataset is also used for evaluating the fine- grained classification task.
B.2 DATA PREPROCESSING
For data preprocessing, we follow the default settings of Cao et al. (2019). For CIFAR-100-LY each side of the image padded with 4 pixels, and a 32 32 crop is randomly selected from the padded image or its horizontal flip. For ImageNet-LT and iNaturalist 2018, after resizing each image by setting the shorter side to 256 pixels, 224 x 224 crop randomly sampled from an image or its horizontal flip.
For BCL and NCL, which use AutoAugment (Cubuk et al. 2019) or RandAugment (Cubuk et al. 2020) as default data augmentation, papers (Zhu et al.) 2022] Li et al.] 2022a). Then, we finally conduct CUDA after all default aug- mentation operations, and then normalize the image with following mean and standard deviation values sequentially: CIFAR-100-LT 194, 0.4822, 0,4465), (0.2023, 0. 1994, 0.2010)). ImageNet- LT (0.485, 0.456, 0.406), (0.229, 0.224, 0.225)), and iNaturalist 2019 (00.466, 0.471, 0.380), (0.195, 0.194, 0.192)).
apply them after random cropping by following their original
B.3 DETAILED IMPLEMENTATION
Because some official codes do not open their entire implementations, re-implement by following the rules. For re-implementation, we reproduce the code based on their partial code and the authors' responses.
RIDE. We follow the officially offered code Among various experimental configurations of official code (e.g,, one-stage RIDE, RIDE-EA, Distill-IDDE), for fair comparison (to leverage similar computation resources), we utilize one-stage training (i.e., one-stage RIDE) for all cases. We confirm that CMO (Park et al. 2022) also utilizes this setup for RIDE + CMO from the response of the authors.
CMO. We re-implement all CMO results from their official code in our work. However, the official code of CMO does not contain code for RIDE CMO. Therefore, we re-implement by injecting the CMO part for BS in the official code (weighted sampler and mixup part) into the RIDE code. Furthermore, for iNaturalist 2018, we train the model for 100 epochs for fair comparison with other methods (whereas the original RIDE CMO is trained for 200 epochs on iNaturalist 2018).
BCL. The officially released code of BCL only contains ImageNet-LT and iNaturalist 2018. Whereas the official code applies a cosine classifier for ImageNet-LT and iNturallsst 2018, we apply
"https///github... https://giuub..oo/maawaa http://ghhhb.comm///aaaaaa//oooo thtps://gituu..cmm// http:///hw...ooom///////////aaaa
17

---

Published as a conference paper at ICLR 2023
ordinary linear classifier for CIFAR- 100-RT from the author's response. All hyperparameters the same as the experiment settings of the original work (Zhu et al. 2022).
B.4 GUIDELINE FOR HYPER-PARAMETE PARAMETER TUNING
Although we did not tune the hyper-parameters extensively. However, we give a guideline to select the syper-parameters.
The number of samples for updating LoL (T). We can set this value according to the given computing resources (i.e., the largest 7' under computing resource constraint). This is because the performance improves T increases from obtaining definite LoL score by testing many samples.
The acceptance threshold (7). Our strategy for tuning gamma to select the largest value in which at least one of LoL. scores among all classes increases within 20 epochs. This is because for large-scale datasets, the network fail infer even the easier-to-learn majority classes, Here the detailed tuning strategy for
We initially set y as 0.6.
We decrease the threshold 7y by 0.1 points whenever it fails to raise any of LoL score for the first 20 training epochs.
We condeut this search on CE with CIFAR-100-LT with IR 100 and using the same y value of the other algorithms with remaining IR settings. Also, we conduct this search rule on ImageNet-LTT with CE and use the same value to the other large-scale dataset, i.e., iNaturalist 2018 with remaining algorithms.
The augmentation probability (Pang). While we did not tune this hyper-parameter, offer the guideline how tune this value based on Figure 5a shown in Figure Sa the shape of graph between Paug and performance is concave. Thanks to concavity, optimal value for this hyper-parae paper-parammter. Note that the reason for the concavity because the decision of Paug value has trade-off between preserving the information of the original image and exploring diversified images.
think that is easy to find the
Further sensitivity analysis ImageNet-LT In Section we apply different values of y in CIFAR-100-LT (0.6) and large-scale datasets (0.4; ImageNet-LT and iNaturalist 2018). addition we further conduct the sensitivity analysis for ? on the ImageNet-LT to verify CUDA works well robustly with different values of y on large-scale datasets. As shown in Table 5 our proposed method CUDA is also robust to hyper-paun parameter selection for not only the small datasets such as CIFAR-100-LT but also large-scale datasets.
Figure 5]
Table 5: Sensitivity analysis of y with BS on ImageNet-LT dataset
7
0.3
0.4
0.5
0.6
Acc. (%)
51.42
51.59
51.38
51.24
C
FURTHER ANALYSES
Training Time Analysis. CUDA requires additional computation for computing LoL score. We measure the additional training time for adding CUDA on various algorithms. As shown in Figure II when utilizing CUDA additional training time spent. However, the additional operation for searching the Lol. score does not require a large value. For example, BS with CUDA spends x 1.29 time to obtain adequate augmentation strength.
Network Architecture Analysis. We also present our ResNet- 10 (Liu et al. 2019 and ResNeXt- 50 (Xie et al. (2017) experiments the ImageNet-LT dataset in Figure 10 respectively. These results show that CUDA consistently improves performance regardless of network sizes and corresponding LTR methods.
18

---

Published as a conference paper at ICLR 2023
40
CUDA
CUDA CUDA
Training time (min.)
CUDA
Accuracy (%)
w/o CUDA
Acouracy (%)
30
w/ CUDA
40
20-
10
CE CD LD BS ResNet-10
CE CD LD BS ResNeXtt-50
CE
LDAM
BS
RIDE
BCL
Algorithm
Figure 10: Network architecture.
Figure 11: Training time.
What if CUDA is ran on the balanced dataset. We examine that if CUDA is applied to the balanced case, i.., imbalance ratio 1. As described in the Table CUDA obtains 1.9% accuracy gain, which is lower than the other auto augmentation methods. However, other autoaugmentation methods spend more computation time searching a good augmentation than CUDA. Furthermore, as described in Figure 4 CUDA has higher performance than the others when the class imbalance dataset is given.
Table 6: Balanced case. mark represents the reported value in (Li et al. 2020b)
Angmentation
Scurching time (Overhead)
CE
68.5
AutoAng
70.7
5, 0.000 hours!
RandAug
69.4
FAA
70.7
3.5 GPU hours!
GPU bourst
CUDA
70.4
0.07 GPU hours
D AUGMENTATION PRESET
D.1 DATA AUGMENTATION OPERATIONS USED IN CUDA.
There have been numerous data augmentation operations in vision tasks. We used totally 22 aug- mentations for CUDA with their own parameter set. Details of the operation set and parameters are described in Table 7 For augmentation magnitude parameter (8), we divide parameters into thirty values linearly. For example of, ShearX case, its and min values are 3 and 0, respectively. Therefore, MSheau (s) (3 0)/30 thus MSheaux 0.01 = (3 0)/30 1.
D.2 FURTHER ANALYSIS ON AUGMENTATION PRESET
To get further intuition on the effect of number of predefined augmentation operations, we conduct several exploratory experiments.
Validity of our main finding (Figure I) under few predefined augmentation. The observation in Figure is caused by minorities becoming relatively easy to lear since majorities have become difficult. Therefore, the sample of majorities becomes difficult enough to learn, the same phe- nomenon as FFiiie Joccurs regardless of the number of augmentation presets. To verify that main finding is valid regardless of the number of predefined augmentations, we conduct the experimental with ten augmentation operations (Mirror, ShearX, Invert, Smooth, ResizeCrop. Color, Brightness, Sharpness, Rotate, AutoContrast), Table 8 describes the performance of (0,0), (0,4), (4,0), and (4,4) that each configuration denotes the augmentation strength of (majority; top 50 class, minor; bottom 50 class). Through the results, we verify that the finding in Figure is valid even in small number of predefined augmentation operations.
Effect of number of predefined augmentation. We further analyze the impact of predefined augmentation operations (K in Figure 2); we additionally experiment by replacing the augmentation preset in Appendix D with the following two augmentation presets: (1) 10 randomly sampled augmentations (Mirror, ShearX, Invert, Smooth, ResizeCrop, Color, Brightness, Sharpness, Rotate,
19

---

Published as a conference paper at ICLR 2023
(a) Raw
(b) Flip
(c) Mirror
(d) Edge Enhance
(e) Detail
(I) Smooth
(g) AutoContrast
(h) Equalize
(i) Invert
(j) Gaussian Blur
(k) Resize Crop
(l) Rotate
(m) Posterize
(n) Solarize
(o) SolarizeAdd
(p) Color
(q) Contrast
(r) Brightness
(8) Sharpness
(t) ShearX
(u) ShearY
(v) Translate
(w) Translate Y
Table Description of augmentation operations utilized in CUDA. We show the examples of each augmentation with maximum augmentation parameters.
Operation
Parameter
Description
Flip
On/Oif
Flip top and bottom
Mirror
On/Off
Flip left and right
Edge
Enhancement
On/Otf
Increasing the contrast of the pixels around the targeted edges
Detail
On/Oif
Uulize convolutional kernel [[0, 1, 0
10,
00, 1, 0]
Smooth
On/Off
Utilize convolutional kernel 1,
1, 5.
AutoContrast
On/Oif
Remove specific percent of the lightest and darkest pixels
Equalize
On/Off
apply non-linear mapping to make uniform distribution
Invert
On/Off
Negate the image
Gaussian Blur
[0.2]
Blurring an image using Gaussian function
Resize Crop
[11.3]]
Resizing and center random cropping
Rotate
[0.30]
Rotate the image
Posterize ze
[0.4]
Reduce the number bits each channel
Solarize.
[0,256]
Invert all pixel values above threshold
SolarizeAdd
[0.110]
Adding value and run solarize
Color
[0.1, 1.9]
Colorize gray scale values
Contrast
[0.1.1.9]
Distance between the colors
Brightness
10.1
Adjust image brightness
Sharpness
[0.1,1.9]
Adjust image sharp
Shear
[0,0.3]
Shearing X-axis
Shear
[0.0.3]
Shearing Y-axis
Translate X
[0, 100
Shifi X-axis
Translate
[0,100]
Shifting Y-xiis
AutoContrast) and (2) RandAugment Cubuk al. 2020 preset that consists of (AutoContrast,
20

---

Published as a conference paper at ICLR 2023
Table 8: The performance comparison between different augmentation strength on major (class indices 0-49) and minor (class indices 50-99) categories.
(major, minor)
Many
Med
Few
All
(major, minor)
Many
Med
Few
0,0
66.2
37.3
8.2
38.7
0,0
62.8
41.7
16.2
41.4
69.7
30.4
35.7
65.9
37.2
39.3
CE
CE-DRW
4,0
60.9
39.3
12.8
38.9
4,0
49.3
45.2
28.3
41.6
4,4
67.0
34.5
4.7
37.0
4,4
56.6
46.6
24.6
43,5
0,0
62.8
42.3
19.0
42.5
0,0
61.6
42.3
23.0
43.3
LDAM-DRW
0,4
70.1
34.3
6.4
38.5
BS
0,4
66.9
37.9
10.8
39.9
52.3
42.0
41.3
48.7
42.5
28.7
40.5
4,4
61.1
43.4
17.3
41.8
4,4
56.3
44.2
23.0
42.1
0.0
67.7
51.5
26.7
49.7
0,4
70.5
36.8
7.7
39.9
RIDE
56.6
44.5
27.2
43.6
62.3
44.5
43.9
Equalize. Invert, Rotate, Posterize, Solarize, SolarizeAdd, Color, Contrast, Brightness, Sharpness, ShearX, ShearY, CutoutAbs, TranslateXabs, Translate Yabs). Table 9 demonstrates that the accuracy slightly increases when the size of the augmentation preset increases. However, the gap between the RandAugment preset (14 operations) and our original preset (22 operations) small compared the gap between the vanilla (without CUDA case) and the RandAugment case. These results verify our belief that the impact of the number of predefined augmentations is small.
Table 9: The performance comparison of CUDA with different number (K) of predefined augmentation operations.
Category
CE
CE-DRW
LDAM-DRW
BS
RIDE
Many
66.2
62.8
62.8
61.6
67.7
Med
37.3
41.7
42.3
42.3
$1.5
Vanilla (w/o augmentation)
Few
8.2
16.2
19.0
23.0
26.7
All
38.7
41.4
42.5
43.3
49.7
Many
70.8
62.3
65.2
62.6
68.5
Med
40.4
49.0
49,2
46.9
$2.0
Random Selection (K=10)
Few
9.0
26.7
21.6
27.3
27.1
All
41.6
47.0
46.5
46.5
50.3
Many
70.3
63.5
65.4
62.9
68.5
Med
40.7
49.1
50.6
48.1
52.3
RandAugment (K=14)
Few
9.6
26.0
21.6
28.7
27.0
All
41.8
47.2
47.1
47.5
50.4
Many
71.6
64.3
67.3
63.3
69.2
Med
42.3
49.2
50.4
48.4
52.8
Ours (K=22)
Few
9.4
26.7
21.4
28.7
27.3
All
42.7
47.7
47.6
47.7
50.7
Effect randomly ordered data augmentation. Our proposed CUDA operates randomly sequential
of the selected augmentations based on the strength of DA. To study the impact of these randomly compare CUDA and CUDA with fixed order augmentations, For examples, when the operation indices (6, 3,,3,77 5) among 22 augmentations are samples,
ordered augmentations,
applied with (3, 5, 6). Table 10 shows small performance differences between the two methods. Thus, we believe that the effect of the augmentation order on the difficulty is negligible. This is because the effectiveness of CUDA is expected be sufficiently high even in a given order of augmentations since the goal is make it harder to learn, regardless of the ordered (determined or random) order.
Comparison with random augmentation. To verify that the success of CUDA is not simply from richer dataset made by DA, we compare our proposed method CUDA to randomly sampled augmentation for every iteration. Our comparison methods are Random 5 and Random 10, which denote the conduct of five and ten randomly sampled augmentations for every iteration. As shown in Table II] while Random 10 generates the most diversifying images, the network trained with this showed the worst performance, even lower than vanilla. Our CUDA achieves the best performance among all methods.
21

---

Published as a conference paper at ICLR 2023
Table 10: The performance comparison of CUDA with random order (Ours) and fixed order of augmentation operations. Note that the values in parentheses are differences between CUDA and CUDA with fixed augmentation order (Random order Fixed order).
Category
CE
CE-DRW
LDAM-DRW
BS
RIDE
Many
71.6
64.3
67.3
63.3
69.2
Med
42.3
49.2
50.4
48.4
52.8
Random order (Ours)
Few
9.4
26.7
21.4
28.7
27.3
42.7
47.7
47.6
47.7
50.7
Many
70.5 (-1.1)
62.8 (-1.5)
66.9 (-0.4)
62.5 (-0.8)
68.2 (-1.0)
Med
43.0 (+0.7)
50.2 (+1.0)
49.7 (-0.7)
48.3 (-0.1)
53.5 (+0.7)
Fixed order
Few
9.0
27 (40.3)
(40.5)
29.6 (+0.9)
26.9 (-0.4)
All
42.4 47.7 (40.0)
(-0.2)
47.7 (40.0)
50.5 (-0.2)
Table 11: The performance comparison between train network with randomly selected five and augmentation operations for every iteration and our proposed CUDA.
Category
CE
CE-DRW
LDAM-DRW
RIDE
Many
66.2
62.8
Med
37.3
41.7
42.3
42.3
51.5
Vanilla
Few
8.2
16.2
19
23
26.7
38.7
Many
68.9
56,9
64.2
59.5
64
Med
35
48.2
42.7
48.2
44.8
Rundomly selected augmentations
3.7
25.6
16.3
All
37.5
44.5
42.3
44.6
44,1
Many
61.7
51.2
57.9
54
57.7
Rundomly selected 10 augmentations
Med
25.7
43.2
34.7
41,2
38.3
1.3
20.6
12.7
All
31.0
39.2
36.2
38.4
38.6
Many
71.6
64.3
67.3
63.3
69.2
42.3
50.4
48.4
CUDA
Few
9.4
26.7
21.4
28.7
27.3
All
42.7
47.7
47.6
47.7
50.7
E EXPERIMENTAL SETTING OF FIGURE 5D
To further analyze the impact of curriculum, we compare CUDA with the performance of previous hyper-parameter search algorithms and auto-augmentation methods, especially DADA (Li et al. 2020b). We describe each setting in detail as follows.
Baseline, This is the case of training with standard data augmentation that consists of random cropping and probabilistic horizontal flip.
Hyper-parameter search. We utilize the strength score-based augmentation module in CUDA to verify the hyper-parameter search. In other words, samples in each class utilize K augmentation operations. Therefore, we search the class-wise augmentation on the search space N where N is the number of classes. We leverage the et al. 2018), for search KN space efficiently. Among various search modules, we utilize the Hyper- OptSearch module, which is the implementation of the Tree-structured Parzen Estimator (Bergstral et al. 2013).. Moreover, for fast search, we use the Asynchronous Successive Halving Algorithm (ASHA) (Li al. 2020a). We 1, 000 trials for each algorithms which spends almost 20 GPU hours (i.e., x80 overhead compare to CUDA).
searching open-source library, Ray (Liaw
Researched DADA operation on imbalanced CIFAR. Because the officially offered policies on CIFAR by Li et al. (2020b) are searched for balanced CIFAR dataset, we have to re-search the augmentation policy for the imbalanced dataset. We utilize the official code of DADA and replace the dataloader to re-search the operations. It spends 48 minutes for searching the augmentation policy ( (x8.6 than the overhead of CUDA), Despite this additional overhead. DADA outputs worse performance than CUDA (even CUDA without curriculum case). This is because (1) DADA does not consider class- wise augmentation and (2) does not consider the impact of class imbalance.
22

---

Published as a conference paper at ICLR 2023
CUDA without curriculum To verify the impact of curriculum itself, the following steps. (1) We conduct experiments with CUDA and get the strength of data augmentation for each class the final epoch. (2) We re-train the network from scratch by using the strength parameter obtained from (I).
F FURTHER ANALYSES
To get better understanding, we conduct several analyses for our proposed method, CUDA.
F.1 FURTHER ANALYSIS ON LoL SCORE
In this section, we conduct experimental ablation studies to understand the performance gain of our proposed method, CUDA.
Suitability of LoL score as metric for class-wise difficulty. The superiority of LoL score is to measure the difficulty metric based on the augmentation strength for each class, which is motivated by our main findings. To verify the suitability LoL. score as a metric for class-wise difficulty, we compared CUDA and the case where LoL score is replaced by the score in (inha et al. (2022). As same with our proposed method, we increase the strength parameter when the score in Sinha et al. (2022) is larger than the same threshold : 0.6. Table 12 summarizes the results that Lol. score showed performance improvement compared to the case of (inha et al. (2022). From the results, we can conclude that this improvement comes from the characteristic LoL score that directly related to augmentation strength.
Table 12: The performance comparison between the scores for determining strength parameter, Sinha
al. (2022) and LoL score (ours).
Category
CE
CE-DRW
LDAM-DRW
BS
RIDE
Many
68.4
59.7
62.0
59.7
67
Med
42.5
48.8
48.7
47.0
52.1
Sinha al. (2022
Few
11.6
27.3
25.4
32.0
26.7
All
42.3
46.1
46.4
46.9
49.6
Many
71.6
64.3
67.3
63.3
69,2
Med
42.3
49.2
50.4
48.4
52.8
LoL score
Few
9.4
26.7
21.4
28.7
27.3
All
42.7
47.7
47.6
47.7
50.7
Effect of random sampling for computing LoL score To implement the computation of LoL score efficiently, we randomly selected the instances for each class. The reason for using random sampling to compute Vocnea is that we want measure how much the model learns entire information for each class. To understand the effect of random sampling, we compare our random sampling method sampling instances with larger (or smaller) losses. Table 13 describes the comparison of performance between various sampling strategies. As shown in the results, if CUDA measures the degree of learning with only easy samples (the samples with small losses), CUDA increases the strength of augmentation too quickly and generates performance degradation. Therefore, it is better way to grasp the degree of learning for each class without prejudice through uniform random sampling. Furthermore, computing loss for all samples for sorting them at the beginning of each epoch requires 1..5 times of computation overhead than our method.
Numerical values of LoL. score dynamics. We provide the numerical values for Figure 4 that is, the average values (for every 20 epochs) of LoL score for the classes with indices 1-10 and the classes with indices 91-100. From the numerical values, we can easily understand the explanation which is discussed in Section
23

---

Published as a conference paper at ICLR 2023
Table 13: The performance comparison between the large loss sample selection, small loss sample selection, and random selection (ours).
Category
CE
CE-DDW
LDAM-DRW
RIDE
Many
67.0
61.6
59.9
67.9
Med
37.1
45,2
45.7
42.4
51.2
Larger Loss
Few
7.3
20.3
20.3
23.3
25.8
All
38.6
43.5
44.2
42.8
49.4
Many
53.4
54.5
51.2
Med
33.0
33,9
38.4
Smaller Loss
Few
24.2
32.9
33.7
35,4
38.2
All
41.6
44 .0
45.5
45.7
49.8
Many
71.6
64.3
67.3
63.3
69.2
Med
42.3
50.4
48.4
52.8
Random (Ours)
Few
9.4
26.7
21.4
28.7
27.3
All
42.7
47.7
47.6
47.7
50.7
Table 14: The averaged LoL score of top 10 classes (class indices with 1-10) and bottom classes (class indices with 91-100) for every 20 epochs.
Class Epoch
120
140
200
Top 10
0.5
0.8
1.7
2.7
3.2
3.2
3.4
3.7
CE
Bottom 10
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.1
0.0
0.1
Top 10
0.8 1.3
2.5
1.6
2.6
1.8
2.6
2,5
2.6
CE DRW
Bottom
Top 10
3.3
3.2
3.1
3.6
4.3
LDAM- DRW
Bottom 10
0.0
0.0
0.1
0.0
0.0
0.1
0.0
0.0
1.8
1.4
Top 10
1.1
1.1
1.6
2.4
2.1
2.5
2.5
2.8
3.9
4.0
BS
Bottom
0.7
Top Bottom 10
1.8
1.1
2.6
RIDE
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.7
0.8
F.2 ANALYSIS THE CASE OF WITHOUT CLASS-WISE
To examine the validity of class-wise augmentation of CUDA, we apply the CUDA with the same strength of DA for all classes. Instead of computing LoL score class-wisely, we computed only one LoL score for the entire dataset by uniformly random sampling instances in the training dataset regardless of class. 'Table 15 shows the significant performance degradation of CUDA without class- wise augmentation compared CUDA. This because, without class-wise augmentation, we cannot allocate the appropriate strength of augmentation to each class.
Table 15: The performance comparison between different augmentation strategy on CIFAR- 100--T with imbalance ratio 100. Note that values in parentheses are differences of CUDA w/o class-wise with vanilla (vanilla CUDA w/o class-wise) CUDA (CUDA- CUDA w/o class-wise).
Category
CE
CE.
LDAM DRW
BS
RIDE
Many
69.0
62.2
67.7
Med
38.7
47.3
47.6
44.7
52.2
CUDA w/o class-wise
Few
6.7
23,5
25.5
26.4
All
39.7
45.5
45.7
44.9
49,9
Many
(-2.8)
62.8 (+0.1)
(-2.4)
61.6
67.7 (40.0)
Vanilla
Med
(-134)
41.7 (-5.6)
42.3 (-5.3)
42.3
(40.7)
Few
(.2.5))5)
(-7.3)
19.0
23.0
26.7 (40.3)
(-3.2)
43.3 (-1.6)
(-0.2)
Many
71.6(42.6)
(+1.6)
67.3
(+1.)
69.2 (+1.5)
Med
42.3 (43.6)
49.2 (+1.9)
50.4 (42.8)
48.4 (+3.7)
$2.8 (+0.6)
CUDA (Ours)
Few
9.4.7)77
26.7 (+3.2)
21.4
28.7 (43.2)
27.3 (40.9)
All
(43.0)
47.7 (+2.2)
(+1.9)
47.7 (+2.8)
50.7 (40.8)
24


Reviews:
[{"review_id": "eI4nqooE-o", "reviewer": "Reviewer_URnx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a class-dependent augmentation strategy for tackling the class imbalance problem. The paper initially observes that by using a class-dependent augmentation where only the majority classes are augmented, the performance of the minority classes improves but that over the majority class regresses. Thus, the paper proposes to learn a curriculum for class-dependent augmentation strength, with the idea being that if the model is performing good over a certain class for an augmentation strength then the strength can be increases and reduced otherwise. They propose level-of-learning, which each class computes the augmentation strength to be used for that epoch. At each epoch, it is either incremented or decremented based on the model's performance over a subset of data for that class over all the augmentation strengths up to the current one.\n\nEmpirically the proposed method achieves a good boost on top of several different existing long-tail tackling approaches over different datasets. To show that their approach leads to a good balance between the majority and minorty classes, the authors plot the variance of weight L1-norm of linear classifier between each class and show that the variance is lower when using the proposed approach. The authors also compare against using a fixed augmentation strategy such as RandAugment and AUtoAugment and show that their method works better than using a fixed augmentation.", "summary_of_the_review": "I really like the extensive experimentation and the different ablations done in the paper to show the efficacy of the approach. I am thus leaning towards accepting the paper.", "strength_and_weaknesses": "Strengths - \n1. The paper is very well written and easy to flow along.\n2. The motivation in the paper about how doing augmentations for some classes affects the performance of non-augmented classes is really useful.\n3. Extensive experiments are conducted on top of existing baselines and over different datasets and the gains achieved show the usefulness of the approach.\n4. I really like the comparison against fixed augmentations such as AutoAugment and RandAugment. Such comparisons show the efficacy of the proposed class-dependent augmentation strategy.\n5. The comparison against doing a curriculum search using hyperparameter optimization algorithms is also neat and useful.\n\nWeakness - \n1. Since for computing the LoL the paper uses only a subset of the data from that class, I am curious to see the variance in the performance. However, none of the tables report the variance in the performance but only the mean result across 3 trials.\n\n2. The paragraph \"Dynamics of LoL score\" needs more emphasis. Specifically, it would be interesting to see some statistics of what are the LoL scores for the majority classes and the minority classes after the end of the training. While figure 4 has the plots, it is difficult to conclude something from there. Can authors include some statistics, such as the mean LoL score for the top k majority and minority classes respectively?\n\n3. How do the authors select which subset of data to use for LoL computation? Is it completely random? Can the authors also report what happens when the samples with the highest/lowest training loss and instead chosen?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ap216e0jy0", "reviewer": "Reviewer_Z6g5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an auxiliary data augmentation technique that can be used on top of several long-tailed recognition methods. The main insight of the paper is that class-wise augmentation actually improves the performance of non-augmented classes. This is an important as well as counter-intuitive observation. The authors devise a data augmentation strategy that relies on the `strength' of augmentation, which is calculated using level-of-learning.\n", "summary_of_the_review": "Overall, I feel the paper has a good amount of novelty and the observations made are crucial for the community. However, there are some concerns that need to be answered before going forward. Hence, at the moment I will keep my score as borderline reject. I would like to see other reviewers' scores and author rebuttals. ", "strength_and_weaknesses": "Pros:\n1. The main observation of the paper- \"class-wise augmentation actually improves the performance of non-augmented classes\" is very crucial and important.\n2. The data augmentation strategy devised by the authors also seems clever and logical. However, there are a few things that are concerning (see concerns).\n\nConcerns:\n1. The way DA with strength parameter is devised is concerning because it assumes that the sequence of augmentation is permutation invariant and preserves the identity of the original sample. This need not be the case with many augmentations.\n2. From Fig. 5c it seems that the method is sensitive to threshold / accept rate \\gamma, in experiments also, the authors choose 2 different values of \\gamma (0.4 and 0.6). What procedure do the authors follow to decide this value? Is there any strategy one can follow to come up with this value?\n3. In Table 2, the improvement with CUDA looks very incremental. \n4. For such close values it is suggested that the authors report the scores in the form of mean \\pm std. dev to give a more clear view of how close the scores actually are.\n5. The authors show that the variance of L1-norm of a linear classifier decreases due to CUDA. In general, any data-augmentation strategy should decrease the variance because of an increase in the data samples.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XISA4lss_I", "reviewer": "Reviewer_nPVy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with class imbalance from the data augmentation perspective. It is based on and motivated by an analysis on the relationship between data augmentation degree and performance per class. ", "summary_of_the_review": "A OKish work on class imbalanced learning with some limited evaluations and also likely limited scalability issue. \n\nSome clarity issues need to be addressed for the final judgement. ", "strength_and_weaknesses": "**Strengths**\n\n- Data augmentation is an interesting dimension to study class imbalance in general.\n\n- The proposed method is simple and reasonably effective in the sense that it exploits the empirical analysis.\n\n\n**Weaknesses**\n\n- The contribution of \"the first to suggest a class-wise augmentation method to find a proper augmentation strength for class imbalance problem\" is over claimed. As this is a special case of exiting dataset-specific data augmentation methods such as  (Cheung & Yeung, ICLR 2022).\n\n- In Introduction, the findings on the data augmentation and class performances are given without proper and concise explanation. This is fine only when they are intuitive and there is no need for explaining, but it seems not.\n\n- Figure 1 needs more interpretation in the caption. e.g. What is the metric for the grid charts\n\n- Method:\n> 1)  A few hyper-parameters are introduced, and how to tune them is not clear.\n> 2) What means by curriculum of DA? \n\n- Experiments: \n> 1) Given that this paper is about data augmentation, it is useful and necessary to compare with existing dataset-specific data augmentation search methods such as (Cheung & Yeung, ICLR 2022). This is missing now. \n> 2) The performance gain on larger datasets (ImageNet, iNaturelist 2018) in Table 2 is smaller. This seems that class-wise data augmentation will get less useful along the scale dimension. This is a key issue for experimental evaluation.\n> 3) Fig 5 (d): This shows that the most performance margin is from the use of curriculum. Given this, what is the performance for the case of w/o CUDA & w. Curriculum?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OIc_-xnVn5v", "reviewer": "Reviewer_QwML", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new data-augmentation strategy based on curriculum learning for long-tail problems. The key idea is to estimate the appropriate strength of data augmentation needed for each class during training. The proposed method was evaluated on widely used datasets and achieved favorable performance.", "summary_of_the_review": "I think the key idea of the paper is reasonable and the empirical evidence provided in the experiments are sufficient. Although the paper has some room for improvement, I think the paper has already enough contribution to the community.", "strength_and_weaknesses": "## Strength\n1. The idea of changing the strength of data augmentation for each class during the training depending on their performance is reasonable and interesting. The idea is so simple that it can be easily integrated with other methods for long-tailed learning as shown in the paper.\n1. The proposed method is evaluated on popular datasets for long-tailed learning and shows consistently good performance.\n1. The analysis provided in section 4.3 helps readers understand the characteristics of the proposed method.\n1. The paper is mostly well written and easy to follow.\n\n## Weakness\n1. In my view, the important contributions of the paper are two-fold: (1) the paper presented a new way to estimate the performance of each class during training, and (2) the paper presented the idea of changing the strength of augmentation according to the performance of each class during training. I think the paper becomes stronger if it could provide more in-depth analysis on which part of the two are essential.\n    1. For (1) [A1] proposed to use validation data to estimate the difficulty of each class. Possibly the authors can use this method in the LoL process instead of the proposed one. This clarifies if it is the proposed LoL process that makes the overall performance better or an alternative method can also be used in the LoL process. \n    1. For (2), even though the authors compare the proposed method with other augmentation methods, it is not clear if the overall performance gain comes from the proposed curriculum, or the selected set of data augmentation and its strength presented in Appendix D. The last paragraph of Section 4 “Impact of curriculum” may be the one for this analysis, but I could not judge if the proposed curriculum is actually effective since there is no detailed explanation on other methods. For example, what is the “vanilla algorithm”? Which “hyperparameter optimization” method is used? Please elaborate “the final augmentation strength of CUDA” more. What happens if the kinds of augmentation and its strength is randomly sampled from the sets presented in Appendix D in each step? I am afraid that the performance gain presented in Table 1-3 actually come from just a richer set of data augmentation.\n1. It is not clear what “the augmentation policies” in the 3rd line from the bottom of p8 means. Is it the augmentation policies of the existing methods or the proposed method? Even though the authors state that ”CUDA is computationally efficient” in the last line, I doubt that it is not necessarily the case because the proposed method needs to run multiple inferences to calculate $V_{Correct}$.\n\n[A1] Sinha+, Class-Difficulty Based Methods for Long-Tailed Visual Recognition, IJCV 2022\n\nMinor points.\n1. I needed to read several times and make some guesses to understand what each graph in Figure 1 shows. I think the 6 graphs shown in the top row represent the accuracies of majority classes (class index 0-49) while the 6 graphs in the bottom row show those of minority classes (class index 50-99). I think it is better to add clearer and more detailed explanation in the caption or the figure itself.\n1. Figure 3: It is necessary to clearly indicate in the caption that the bottom row shows feature alignment gain, not feature alignment.\n1. It is good that the dynamics of the LoL score is shown in Figure 4, and it shows reasonable results. I wonder if it has actually correlate with the accuracy. The paper may become stronger if the analysis on the correlation between the estimated LoL score and validation or test accuracy is added.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EzEKWx5113e", "reviewer": "Reviewer_N4yQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper found that augmentations in one class may be negative for itself but positive for other classes. Thus, they propose a novel data augmentation method, call CUDA for long-tailed recognition. This method can generate proper class-wise augmentation strength for long-tailed recognition. They first compute a level-of-learning score for each class and leverage the score to determine the augmentation. The authors conduct experiments on cifar-lt, imagenet-lt and inaturalist 2018. The comparisons with previous SOTA methods demonstrate the effectiveness of CUDA.", "summary_of_the_review": "The motivation, writing, and the results are good of this paper. However, there are some points which need further clarification.", "strength_and_weaknesses": "Strength:\n \n1. \"Our key finding is that class-wise augmentation improves performance in the non-augmented classes while that for the augmented classes may not be significantly improved, and in some cases, performances may even decrease.\" The motivation and the findings in this paper are really interesting. \n\n2. The results are good and comprehensive in the experiments. \n\n3. The paper is well-presented and easy to follow. \n\nWeakness:\n\n1. One of the key ablations is missing. What are the results of uniformly assigning all data augmentations to all classes? This should be a baseline and presented in the tables. That is, the conventional operation\n\n2. Will the order of different augmentations have an effect on the final results?\n\n3. What is the effect of preset augmentation number K?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%