[Re] CUDA: Curriculum of Data Augmentation for Long‐tailed Recognition
Authors: Barath Chandran.C
Venue: Accepted by TMLR ()

Abstract: In this reproducibility study, we present our results and experience during replicating the paper, titled CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition(Ahn et al., 2023).Traditional datasets used in image recognition, such as ImageNet, are often synthetically balanced, meaning each class has an equal number of samples. In practical scenarios, datasets frequently 
exhibit significant class imbalances, with certain classes having a disproportionately larger number of samples compared to others. This discrepancy poses a challenge for traditional image recognition models, as they tend to favor classes with larger sample sizes, leading to poor performance on minority classes. CUDA proposes a class-wise data augmentation technique which can be used over
 any existing model to improve the accuracy for LTR: Long Tailed Recognition. We successfully replicated all of the results pertaining to the long-tailed CIFAR-100-LT dataset and extended our analysis to provide deeper insights into how CUDA efficiently tackles class imbalance. The code and the readings are available in https://anonymous.4open.science/r/CUDA-org--C2FD/README.md

Published in Transactions on Machine Learning Research (06/2024)
[Re] CUDA: Curriculum of Data Augmentation for Long-
tailed Recognition
Barath Chandran.C
barath ocftece. wwwr.tc.c.......... in
Indian Institute of Technology
Roorkee
Reviewed on OpenReview: https: // openreview. net/ forum? id= Wm6d44I8St
Abstract
In this reproducibility study, we present our results and experience while replicating the paper titled CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition (Ahn ct al. 2023 Traditional datasets used in image recognition, such as ImageNet, are often syuthetically balanced, meaning each class has an equal number of samples. In practical sccnarios, datasets frequently exhibit significant class imbalances, with certain classes hav- ing disproporttonnaley larger number of samples than others. This discrepancy poses challenge for traditional image recognition models as they tend to favor classes with larger sample sizes, leading to poor performance in minority classes. CUDA proposes a class-wise data augmentation technique that can be used over any existing model to improve the accu- racy for L.TR: Long-Taile Recognition. We successfully replicated a substantial portion of the results pertaining the long-tailed CIFAR-100-LE dataset and extended our analysis to provide deeper insights into how CUDA efficiently tackles class imbalance.
1 Introduction
Long-tailed Recognition presents of the most formidable challenges in visual recognition. This problem revolves around training highly effective models from datasets characterized by large number of images distributed along long-tailed distribution. In datasets characterized by LTD, a notable skew exists in the distribution of samples across different classes, resulting in certain classes being vastly over-represented (Hoad classes) while others are significantly under-represented (Tail classes). To clucidate this phenomenon, consider disease screening lests: the head class comprises of numerous samples of non-patients, whereas the tail class comprises of fewer samples of patients. In such scenarios, deep learning models often perform well on the head classes but struggle with the tail classes due to their limited representation.
Solutions to Long-tailed recognition primarily involve three methods: (1) Resampling (Buda et al. 2018
upsampling minority classes and downsampling majority classes. (2) Reweighting (Cao et al. 2019): rebal-
ancing the loss to give more weight to minority classes. (3) Transfer learning (Kim et al. 2020 enriching the information minority classes by transferring information gathered from majority classes to minority classes. While numerous strategies have been suggested use data augmentation techniques for generating diverse representations of minority samples, minimal attention has been given to asscssing the impact of varying augmentation degrees across different classes on addressing class imbalance issues.
The original authors proposed that applying an algorithm to determine class wise augmentation strength potentially address the imbalance problem in long-tailed visual recognition tasks. This data augmentation technique called CUDA, when used along with existing Long-Tailed Recognition (LTR) models, increases the performance those models. The other key finding, as highlighted in the original paper, is that after training when we examine the class-wise augmentation strength, the majority classes have a stronger dogree of augmentation and the minority classes have weaker degree of augmentation. This finding is counter- intuitive, as one would typically anticipate that the minority class with fewer samples would undergo strong

---

Published in Transactions on Machine Learning Research (06/2024)
augmentation, while the majority class with more samples would undergo weaker angmentation. The original authors utilized two metrics, weight L1-norm, and feature alignment gain, to demonstrate the effectiveness of CUDA in mitigating the imbalance problem. However, the original authors did not elaborate on how these two metrics are linked to the feature representation space of imbalanced datasets or elucidate how CUDA influences the feature representation space.
The motivation behind this reproducibility study is threefold: (1) To validate the assertion made by the original authors that employing class-wise augmentation strength can enhance the performance of existing LTR models.
(2) To confirm the counter-intuitive observation from the original paper that employing stronger augmentation on majority classes and milder augmentation
minority classes yields superior model performance compared to the opposite strategy. (3) To delve deeper into how CUDA effectively addresses the imbalance problem by leveraging insights from prior research on feature representations in long-tailed datasets, a dimension not explored in the original paper.
2 Scope of Reproducibility
To address our threefold motivation behind this paper and to reproduce the results of the original paper, we perform the following experiments over the CIFAR-100-LT dataset:
1. For our first motivation, we examine the performance of CUDA across LTR models like CE (cross- entropy), CF-DRW (cross-entropy dynamic reweighting), LDAM-DRW (label-iistribttion-awar margin loss), BS (balanced softmax), and RIDE (Table11.
2. For second motivation, investigate the LOL (Learning Objective Level) score, representing the augmentation strength of each class after training with CUDA [Figure-2)
We examine how accuracy changes with the three prper-parametors augmentation probability, num- ber of test samples, and acceptance rale to reproduce the result that both excessive and insufficient augmentation adversely affect performance (Figure-5).
We examine the metrics variance weight Ll-norm and feature alignment gain to reproduce the result that CUDA leads to docrease weight L1-norm and a positive feature alignment gain Figure-3 and Figure-4).
5. We evaluate the contribution curriculum learning and class-wisc score on the performance of CUDA to reproduce the result that both curriculum learning and class-wise score are important to the performance of CUDA (Figure-6).
6. We compare the performance of CUDA with other angmentation methods to validate the result that CUDA outperforms all existing augmentation techniques Figuur--6.
7. We conduct the performance analysis across three different imbalance ratios (100, 50, 10) to examine how the performance of CUDA varies with dataset imbalance ratios (Tble-2).
We compare how CUDA performs with and without cutout augmentation to quantify its contribution Figuro-8..
9. For our third motivation, we compare the feature representation space of the vanilla and CUDA versions, examining metrics such as inter-class distance and intra-class distance ( Figure-7]
3 Methodology
3.1 Model descriptions
We use ResNet-32 our backbone model for all models: CE, CE-DRW, BS, LDAM-DRW, and RIDE. However, RIDE uses modified version of ResNet-32 implemented according to the original paper. We
2

---

Published in Transactions on Machine Learning Research (06/2024)
Training with Curtoul am Augmnetation
Data sugrentttiin strength parameter 1i
Undate Lok.
Sampling sugmorbriae ope
Marit
Aivblity
90..1
Cay
Corus 2010 12.t A2
Ohonii -Q- 1p
Figure Schematic CUDA. The left part depicts the strength-bases augmentation and the right part depicts how the Level-of- (oaningLLa))))) score updated in a given epoch
apply the data augmentation technique CUDA over these existing models. The core philosophy of CUDA
is to 'generate an augmented sample that becomes the most difficult sample without losing its original information'. CUDA utilizes two main parts to achieve this: (1) Strength-based augmentation and (2)
Using Level-of- Learning (LOL) score. In the following sections 3.1.1 and 3.1.2, we explain the methodology of CUDA outlined in the original paper.
3.1.1 Strength-based augmentation
We need to define a metric to quantify the degree of augmentation we are applying to an image. Let us assume that there exist K predefined augmentation operations indexed as & E {I....,K' for example, gaussian blur, rotation, horizontal flip, etc. Each augmentation operation Omza(s) RR - Rd has its own predefined augmentation magnitude function m²(s) with the strength parameter EE {O....,)) augmentation strength parameter s and an input image x, we model a sequence of augmentation operations (xx;s) s) as follows:
S). Given an
Oz s) O m.a (+)
(6) o
mL, O k1
(z), k~ Cat(,K,((())) =i=(1,.... (1,...,s)
The sequential augmentation operation O(2; 8) involves sampling operations from categorical distribution, where each operation is chosen from uniform distribution among K possible operations. In essence, out of the K possible operations, only operations are selected, each with a magnitude m(3). For example, let's consider s = 3. Suppose the selected augmentation operations k1,k2, and h3 correspond to brightness adjustment, x-shift, and y-shift, respectively. In this case, O(x; 3)outputs an image where the brightness is increased by (3)) shifted by m²=shift(3) on the x-axis, and shifted by my-shift(3) on the y-axis. As s increases, both the number of augmentation operations applied to each image and the magnitude of each of these operations increase consequently, the complexity of the augmentation process increases. Appendix A augmentation preset describes how (s()) the magnitude function of cach data augmentation opcration is defined.
3.1.2 Updating Level of Learning Score
To control the strength of augmentation properly, need to check whether the model correctly predict augmented versions without losing the original information. To enable this, we define the LOL for each class at epoch i.e., L which is adaptively updated as the training continues as follows:
Initialization: At the start of training, set L to zero for all classes. For clarity, lot's consider that Iwo epochs have already passed, and we begin the third epoch. Let's assume L the LOL score for a specific class after the second iteration is
Update Mechanism: The LOL value for each class updated using the function Viot, which takes inputs such as the images D. belonging to class e, the previous LOL, value Le-1, the model fo used
3

---

Published in Transactions on Machine Learning Research (06/2024)
for prediction, and hyperparameters y and T.
L = Viol (Des Le- 70,77
Sampling: Within Viot iterate over all values of I less than L7 2. For each (0, 1, 2 in this case), randomly sample T( + 1) samples from D. to form D'd ensuring D = T 1). Instead of utilizing all samples in set D. for prediction, we opt for a subset of images denoted as D' The size of this subset is determined by the first hyperparamcter, denoted as T, which represents the number of lest samples.
Calculation of V This involves summing the indicator function ((((=))))== for all samples D'c, where O(z; () represents the application of augmentation strength / to sample If the model correctly prodicts class (c=1 in this case), the function evaluates to 1. In essence, Veereect quantifies the number of correct predictions among the T( + 1) samples in D'
VCorrett fo. = Σ ((((0(()))=)) where D' C De
ZED
Adjustment: If the number of correct predictions exceeds the threshold y T([ + 1), where / takes values from 0 to 2, it signifies that the ratio of correct predictions to total samples surpassos the second hyperparameter score for the next epoch, denoted as L2, is ineremented by compared to the current value of L = =1 3). If the condition is not met, the value of the LOL score decremented by
representing the acceptance rate. In this scenario, the value of the LOL
(L) = -
-E-1 + L -1
if VCotent (De,,, TT(+1) (T((11 vE [0-1) otherwise
VLOL (Der 12-1 T)
Augmentation Sequence: By applying Via for all classes could then define sequence of aug- mentation operations O Lg.) with their strengths defined by the respective Ly where each is a class.
Augmentation Probability: Instead of completely replacing the training dataset D with augmented images, we apply a sequence of augmentation operations with a probability specified by a third hyperparameter, p, representing the augmentation probability. This results in the creation of a new dataset DCUDA.
LTR algorithm: Execute any Learning to Rank (LTR) algorithm utilizing the dataset DCUDA. comprising both augmented and original images.
3.2 Dataset and Hyper Parameters
The reproduction study is done on the dataset CIFAR-100-LT as mentioned in the previous works(Cas et al. 2019). The CIFAR-100 dataset (Canadian Institute for Advanced Research, 100 classes) is subset of the Tiny Images dataset and consists of 60000 32x32 color images. Three different datasets from the CIFAR-100 datasct with imbalance ratios 100, 50, and 10, where the imbalance ratio is defined as |D1| /iooo =Nmax/N D&l between Di Nmax and |Diool Nmin follows a linear decay. The hyperparameters- paappaaramettrs--annnnntttttoo
derived
augmentation probability, number of test samples, and acceptance rate are set according the values stated in the original paper: 0.6 for augmentation probability, 10 for the number of test samples, and 0.5 for the acceptance rate. The hyperparameter sensitivity analysis done in Section 4.1.5 for the three hyperparameters further substantiated the values used.
4

---

Published in Transactions on Machine Learning Research (06/2024)
3.3 Experimental setup and code
We have conducted all experiments for the CIFAR-100-LT dataset using the official repository, implemented in PyTorch. The code from the repository was reorganized into a Jupyter notebook to enhance portability and offer better control over the environment. Dependencies were not explicitly provided, and the versions of different libraries were determined through trial and error. Deprecated elements within the code were replaced with suitable, up-to-date alternatives. For training the model, parameters were set based on the specifications outlined in the paper. Any parameters not explicitly mentioned in the paper were assumed to use default values. The code for conducting component analysis on curriculum learning, class wise score measurement, and measuring the standard deviation in weight L1 norm were re-implemented based on the specifications provided in the paper. The code and the readings are available here
3.4 Computational requirements
We trained the model in Kaggle with NVIDIA Tesla P100 the GPU accelerator. The average training time of the model was approximately 40 minutes with a batch size of 128 for 200 epochs and the overall budget of the reproduction study was 250 GPU hours.
4 Results
4.1 Results reproducing original paper
Section 4.1.1 delves into the findings related to the first motivation of validating the assertion made by the original authors that employing class-wise augmentation strength can enhance the performance of existing LTR models. Section 4.1.2 delves into the findings related to the second motivation of confirming the counter-intuitive observation from the original paper that employing stronger augmentation on majority classes and milder augmentation on minority classes yields superior model performance compared to the opposite strategy. Section 4.1.3 and Section 4.1.4 provide additional insights into the original study weight I.1 norm and feature alignment, gain that also justify our third motivation of investigating feature representations in long-tailed datasets. The subsequent sections in 4.1 encompass the results obtained from our reproduction of various analytical studies conducted in the original paper, aiming to further validate and understand the efficacy and implications of employing CUDA.
4.1.1 Comparison of validation accuracy
We measure the validation accuracy of CUDA when used with CE (Cross-Entronn CE-DRW (Cross entropy Dynamic reweighting) (Cao et al. 2019). L.DA--DRW (label-distribttion--veee margin loss) (Cao et al. 2019), BS (balancod softmax) (Ren et al. 2020 and RIDE (Wang et al.) 2021h for the CI0AR-100-LT dataset following the general settings outlined in Cao et al. (2019). Specifically, we use ResNet-32 (He & Gareia 2009) as the backbone network. The network trained using stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 0.0002. The initial learning rate is set to 0.1, and lincar learning rate warm-up is applied during the first 5 epochs to reach the initial learning rate. The training process spans over 200 epochs, during which the learning rate is decayed at the 160th and 180th epochs by a factor of 0.01. The hyperparameters: acceptance rate, augmentation probability, and Number of test samples were 0.6, 0.5, and 10, respectively. The anticipated outcome was a consistent rise in validation accuracy across all five models and three imbalance ratios upon employing CUDA. In our reproduction of this study, we were able to reproduce the accuracy values with a maximum deviation of 2.4%. Further, we can observe from [able-1] consistent increase in accuracy across all models for the imbalance ratios 100 and 50 when paired with CUDA compared to the Vanilla edition. However, for the imbalance ratio 10, we see a minor improvement in accuracy for all models except when CUDA is paired with CE+CMO.
4.1.2 Dynamics of LoL Score
We plot the progression of LOL scores (the strength of augmentation) for various classes across five models: CE, CE-DRW, BS, L.DAM-DRW and RIDE. The anticipated outcome was stark difference between the
5

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm
IR=100
IR=50
IR=10
IR=100
many
med
few
CE
38.49
42.92
56.61
65.8
36.27
8.43
CE+CMO
42.08
46.93
60.08
68.87
41.23
11.7
CE+CUDA
42.27
46.91
59.79
70.47
41.93
9.6
CE+CMO
43.01
48.14
58.62
70.2
42.47
11.77
CE-DRW (Coo et al. 2019
40.86
46.22
57.83
62.33
41.2
15.23
CE-DRW + Remix Chou et al., 2020 t
45.8
49.5
59.2
CE-DRW + CUDA
47.52
50.57
58.18
66.43
50.27
22.03
LDAM-DRW CCao et al., 2019)
42.48
47.15
57.96
62.17
42.43
19.4
LDAM + M2m(Kim et al. 2020)
43.5
57.6
LDAM+CUDA
47.52
50.57
58.18
66.43
50.27
22.03
BS
42.24
45.99
58.29
59.63
41.33
22.8
BS+CUDA
47.47
52
61.52
63.03
48.8
27.83
RIDE Wang et al. f
48.6
51.4
59.8
RIDE
48.87
51.95
59.13
67.7
50.17
25.27
RIDE+CMO Park et al. 2
53
60.2
RMO
49.61
52.78
59.91
68.03
50.9
26.47
RIDE+CUDA
49.53
53.58
61.21
68.23
50.7
24.63
Table 1: Validation accuracy on CIFAR-100-I.T datasel. are from Park et al. 2022 and j.* are from the original papers (Kim et al. (2020); Zhu et al. (002)) .e We report the average results of three random trials on each scenario. The rows highlighted in green refer to the versions where CUDA is used. We can observe an increase accuracy when we compare similar models with and without CUDA. For example, vanilla CMO for imbalance ratio 100 has an accuracy of 42.08 whereas CE+ has an accuracy of 43.01.
Figure 2: Evolution of LOL score in the order CE, CE-DRW, LDAM-DRW, BS, and RIDE over the epochs. The x-axis represents the epochs, with the 200th epoch positioned on the rightmost side of cach graph. The y-axis will display the classes, arranged in descending order based on the number of samples in each class. The intensity of the color of the heatmap represents the strength of augmentation. training the majority classes(O-49) have stronger augmentation compared to the minority (asses(50-99),
evident that during

---

Published in Transactions on Machine Learning Research (06/2024)
Variance of Li-norm Among Classes
Varlance Among Classes
Vanilla
Vanilla CUDA
CUDA
1.0
2.0
100
200
100 epochs
200
epochs
Among Classes
Variance of L1 Li-nom Among Classes
Vanilla CUDA
3.5
Vanilla CUDA
Vettnce of cum
Vaatunce of eATE
18
100
200
100
200
epochs
egochs
Figure 3: Variation Of weight-LI norm in the order CE, CE-DRW, LDAM-DRW, BS over the epochs. The value of variance at epoch 200 represents the final value of variance of weight-LI norm after training CUDA (indicated by the blue line) exhibits lower variance of weight-I.1 Norm compared to the vanilla version (depicted by the red linc).
LOL scores of the majority class (0-49) and the minority class (50-99) with the LOL score of the majority class being greater. In our replication of this study, we observed that there isn't a steady decrease in augmentation strength as we move along the y-axis. However, there a noticeable pattern of higher average augmentation strength for the majority classes (0-49) in comparison to the minority classes (50-99). The heat maps presented in Figure-2 validate the assertion that 'Stronger augmentation on majority classes and weaker augmentation minority classes yield better performance".
4.1.3 Variance of Weight L1-norm
Image recognition can be considered as a coupling of two tasks: feature learning where we extract features from the images and embed them into a feature space, and classifier learning where we train a classifier over the Icarned features (Kng et al., 2020). Long-tailed data distribution tends to have a skewed represcntation space, where the distance between head and tail categories is much larger than the distance between two tail categories (Fu et al.) 2023). A naively trained model on long-tailed distributed data tends to have "artificially" large weights for the head classes. This yields wider classification boundary in feature space for the head classes, allowing the classifier to have much higher accuracy on head classes, but hurting the performance of the tail classes (Kang et al. 2020).
The variance of the classifier weight norm is usually used to mcasure how balanced the classifier considering the input from class-wist perspective (Kang et al. 2021). A lower variance in weight L1 norm indicates that the classifier assigns similar importance to all classes. We analyze how CUDA affects the standard deviation
7

---

Published in Transactions on Machine Learning Research (06/2024)
alignment gain
0.04
0.04
0.02
Diffrencce
0.00
-0.04
-0.04
40
60
80
100
Index
feature reature_alignmenn alignment_gen gain
feature alignmmen, gain
0.04
0.08
Deftrrnce
0.06
Difference
Mil A Ma
0.01
0.02
0.00
-0.01
100
Indox
Figure 4: Feature alignment gain in the order CE, CE-DRW, LDAM-DRW, BS. Feature alignment gain is the increase in the average feature cosine similarity between feature vectors belonging to the same class, The x-axis represents the classes and the y-axis represents the feature aligmment gain for that class. CUDA exhibits a positive feature alignment gain(indicated by the blue line) for most of the classes across all 4 models.
Best Accuncyy Prbb
Best Accuracy Numbet
Beal Accunncy in Accept
330
Nam
Figure 5: Hyperparameter Analysis of CUDA when paired with the model LDAM-DRW. The leftmost graph represents the sensitivity of accuracy with Augmentation probability. We see a concavity in performance when the value of Prob Aug is changed between 0.3 to 0.7. The middle graph represents the sensitivity of accuracy with the Number of lest samples. We see a slight change accuracy when the value of the Num Test is changed, The rightmost graph represents the sensitivity of accuracy with the Acceptance rate. We concavity in performance when the Acceptance rate value is changed between 0.1 to 0.9.

---

Published in Transactions on Machine Learning Research (06/2024)
of weight L1 norm across the 4 models over the epochs. The anticipated result was consistent reduction in the value of the variance of weight L1 norm using CUDA across all 4 models. In our reproduction of this study, we observe a significant decrease in the standard deviation of weight L1 norm when we use CUDA compared to the vanilla version from Figure-3 validating the results of the original study.
We can say any enhancements performance can be attributed to either improved feature representation or a more effective classifier. In this case, since the classifier remains the same for both vanilla and CUDA versions, the improvements are likely due to CUDA optimizing the feature representation space. The variance in weight norm decreases, implying a reduction in the distinction between different classification boundaries (between two head classos, or between two tail classes, or between a head and tail class). Specifically, we can infer that CUDA is able to make the feature representation more balanced in terms of inter-class distances [Song et al. 2015
4.1.4 Feature Alignment Gain
Contimuing from the discussion of an imbalanced feature space representation in subsection 4.1.3, another significant. impact of class imbalance on the feature representation space is the vulnerability of cluster containing instances of tail class (Huang et al., 2016). Due to their sparse nature these clusters, which are expected to contain instances of the same class, are more prone to invasion by imposter feature vectors from other classes present in the neighborhood.
Feature alignment gain serves as a metric to quantify the effectiveness of a feature extractor in embedding features into the feature space. Specifically, it measures the improvement in feature alignment, which is the sum of cosine similarities between pairs of feature vectors within a given class when CUDA is used. We analyze how CUDA affects the feature alignment gain of each class across the 4 models for the validation dataset. The anticipated result was mostly positive feature alignment gain i.e. gain in feature alignment using CUDA across all 4 models. In our reproduction of this study, we observe from Figure-4 that feature alignment gain is positive for most of the classes when use CUDA.
When feature vectors within a class exhibit high similarity, they tend to cluster closely together within the feature representation space. This closeness results in reduced intra-class distance (Song el al., 2015 meaning that feature vectors belonging to the same class are positioned closer to each other. Consequently, the likelihood of intrusions from feature vectors of forcign class diminished.
4.1.5 Hlpee--Paaameer Analysis
The paper proposed training on RIDE, but we opted for LDAM-DRW as LDAM-DRW showed a higher increase in performance when paired with CUDA, which means that the magnitude of change in performance with change in hyperparameter will be higher. For the acceptance rate, we trained the model LDAM-DRW with CUDA for 5 equally spaced values from 0.1 to 0.9. An acceptance rate of 0.1 means that the threshold for accepting the augmented samples is low, leading to higher augmentation strength. An acceptance rate of 0.9 means that the threshold for accepting the augmented samples is high, leading to lower augmentation strength. For augmentation probability, we trained the model LDAM-DRW with CUDA for 5 equally spaced values from 0.3 to 0.7. An augmentation probability of 0.3 means that most of the original images are retained when forming the data loader. An augmentation probability of 0.7 means that most of the original images are replaced by the augmented images when forming the data loader. The anticipated result was a concavity in the value of validation accuracy when the values of acceptance rate and probability of augmentation are increased. In our reproduction of the hyperparameter analysis studies from Figuro-5) we can observe that acceptance rate and probability augmentating at show a concavity in the performance. For the hyperparameler number of test samples, we trained the model LDAM- with CUDA for three different values of 1, 10, and 100. The anticipatod result was steep increase accuracy between to 10 values of T and slight increase between 10 to 100 values of T. In our reproduction of the hyperparameter analysis, we can see that the performance increases slightly with an increase in the number of test samples from Figure-5
9

---

Published in Transactions on Machine Learning Research (06/2024)
vithout
Venilla
Whbhot Canistii Cudarwith cumculari
CUDA Lithost Claststte CLtbA with satwie
C2.0NN
Mhen
163 0x 12,00 Ulen
4.00
Figure 6: Component analysis on Class-wisc Score and Curriculum Learning, Comparison of accuracy on different augmentation techniques. The leftmost graph illustrates the comparison of accuracy among Vanilla, CUDA without a Class-wise Score, and the original CUDA version with a Class-wise wise Score. The middle graph demonstrates the comparison of accuracy between CUDA without Curriculum Learning and the original CUDA version with Curriculum Learning. The rightmost graph represents the comparison of accuracy for 9 different augmentation methods: CUDA, Auto Augmentation-CIFARR policy, Auto Augmenlatiin-Imagee policy, Auto Augmentation-SVII policy, Dada- Dada-CIFAR, Dada- ImageNe, Fast Auto Eigmentation-CIEAl CIFAR, Fast Auto Augmentation-Image ImageNet policy, and Random Augmentation. We observe that the original it- eration of CUDA (represented by the green bar) consistently oulperforms its competitors across all three studies.
4.1.6 Curriculum Learning
Curriculum Learning is a training strategy designed to enhance machine learning models by progressively exposing them to increasingly complex or challenging data during training. Previous works (Zhou et al. 2020 have shown that curriculum learning can improve the accuracy of LTR models. In the context of the CUDA algorithm, the LOL (Learning Objective Level) scores for each class initially start zero and are iteratively updated the end of each epoch based on the model's performance with augmented images, After 200 epochs, optimal combination of LOL scores is achieved, leading to the final model performance. To assess the impact of curriculum learning accuracy, two-step approach is employed. In the first step, model is trained using the standard CUDA procedure. Subsequently, the LOL scores obtained from this initial training run are extracted and utilized as fixed scores in a subsequent run. In this second run, the model trained without updating the LOL scores. The anticipated result was a reduction in accuracy on the second run when CUDA is trained without curriculum learning. In our reproduction of this component analysis, we can observe from Figure-6 that there is decrease in performance across all models when CUDA trained without curriculum learning.
4.1.7 Classwise Score
To examine the validity of class-wise augmentation of CUDA, we apply CUDA with the same strength of DA for all classes. Instead of computing the LOL score class-wisely, we computed only one LOL score for the entire dataset by uniformly random sampling instances in the training dataset regardless of class. The anticipated result was a reduction accuracy when CUDA is trained without the class-wise score. In our reproduction ion of this component analysis, we can observe from Figure-Oaa significant performance degradation of CUDA across all 5 models without class-wise score compared to CUDA.
4.1.8 Comparison with other Augmentation techniques
We compare the performance of CUDA with other augmentation techniques: Into-Augmentationn (CIFAR, ImageNet, and SVHN policy) (Cubuk al. 2019)) Fast Auto Augmentation (CIFAR, ImageNet, and SVHN policy) (Lim al.) 2019), DADA (Li al.) 2020), and Rann-Auggentation (m=1, n=2) (Cubuk et al. 2020) for the model LDAM-DRW. The anticipated result was that CUDA outperforms all other existing data
10

---

Published in Transactions on Machine Learning Research (06/2024)
Embedding SNE
Clss
Distance between class and 2
22.06
Oees
Distance between class and
58.72
Distance between class and 3
44.78
Average head to head distance
41.85
Distance between class 97 and 98
17.39
Distance between class 98 and 99
14.35
Distance between class 97 and 99
12.84
Average tail to tail distance
14.86
Difference between h2h and t2t distances without CUDA
26.99
-30
Clsss
Dass
Distance between class and Distance between class and
21.12
Okiss
43.85
Distance between class and
44.67
Average head to head distance
36.54
Distance between class 97 98
16.93
Distance between class 98
14.79
Distance between class 97 and 99
17.57
Average to tail distance
16.43
Difference between h2h and t2t distances with CUDA
20.11
-10
20
Figure Leftmost Feature reprosentation of vanilla CE, inter-class distances for vanilla CE, Feature representation of CE+CUDA, rightmost inter-class distances for CE+CUDA. In the feature representation space, the red, purple, and brown instances represent three tail classes (97, 98, 99) and the green, yellow, and blue instances represent three head classes (1, 2, 3). The dotted line connects the means of 2 head classes, and the solid line connects the means of 2 tail classes. The distances in the table are mcasured between the means of the classes. It is evident that after using CUDA, the difference between average head-to-head distance (h2h) and average tail-to-tail distance (t2t) has diminished.
augmentation techniques. In our reproduction of this comparison, Figure- 6 reveals that CUDA consistently outperforms all other augmentation techniques.
4.2 Results beyond the original paper
Section 4.2.1 delves into findings related to third motivation of exploring the changes in feature repre- sentation space after using CUDA. Section 4.2.2 consists of analysis of the performance of CUDA for the augmentation operation cutout which was implemented in the original code but not explicitly stated to be used in the paper. Section 4.2.3 consists of an analysis of accuracy gain when CUDA used across three different imbalance ratios.
4.2.1 Quantitative and Qualitative analysis of feature embedding
We analyzed the feature embedding derived from the validation dataset of the imbalanced CIFAR-100-LT dataset, building upon prior methodologies (Huang et al.) 2016 Song et al. 2015) aimed at addressing class imbalance through enhanced feature representation. Specifically, we visualized the 2-dimensional fcature representations of three head classes (1, 2, 3) and three tail classes (97, 98, 99) using t-SNE.
The quantitative aspect we examined was the variance in inter-class distances, as detailed in section 4.1.3, This variance can be quantified by the difference between the average distance within head classes and that within tail classes. Notably, we observed a diminished difference from the tables in Figure-7 when employing CUDA compared to the vanilla version, indicating improvement in handling class imbalance. We also
11

---

Published in Transactions on Machine Learning Research (06/2024)
Compair accuracy without
Oazelottrsst cuteet) Caserwith ansut)
Figure 8: The image represents the comparison of the accuracy of CUDA when paired with cutout, when not paired with cutout, and the vanilla version.
Algorithm CEUM CE-DRW+CUDA SHAM-DR+CCUDE BS+CUDA RIDE(3 +CUeA
RR=100
IR=50
IR=10
0.93
1.23
-1.46
6.37
5.68
3.93
5.04
3.42
0.22
5.23
6.01
3.23
0.66
1.63
2.08
Table 2: The table represents the analysis of gain in accuracy with 3 different versions of CIFAR-100-LT with an imbalance ratio:10,50, and 100.
observe slight increase in the average distance of means of tail classes denoting a wider classification boundary than before between tail classes.
The qualitative aspect we examined Following the adoption of CUDA, noted that the clusters corresponding to tail classes exhibited greater coherence from Figure-7 There was reduction in the intrusion of imposter vectors into these clusters, signifying enhanced separability.
the representation of the tail class clusters in the feature space.
4.2.2 Cutout
The usage of the augmentation operation Cutout not explicitly mentioned in the original paper. However, the official implementation of CUDA also had argument to either Cutout has proved to improve the generalization performance of CNNs (DeVries & Taylor 2017). Our analysis focused on investigating how Cutout affects the performance of CUDA for the CE (cross-entropy) model to validate the fact that the original authors used the cutout augmentation operation during training for all models. Our findings in Figure-8 suggest. that incorporating the cutout augmentation operation during training is crucial for achieving the reported accuracy levels stated in the original paper for all models.
not use the cutout operation.
4.2.3 Gain in accuracy for imbalance ratio
We conducted a comparative analysis of the gain in accuracy among five models when paired with CUDA across three different imbalance ratios: 100, 50, and 10. Imbalance ratios reflect the disparity in class distribution within the dataset, with higher ratios indicating more pronounced class imbalances. Our findings in Table-2 reveal a general trend where the gain in accuracy diminishes the imbalance ratio decreases. We can infer that CUDA performs better when the imbalance ratio of the dataset is higher.
12

---

Published in Transactions on Machine Learning Research (06/2024)
5 Discussion
The experimental results reported in this study effectively support the first claim of the original paper, indicating that using an algorithm to find class-wise augmentation strength can enhance validation accuracy (section 4. However, due to time constraints, the performance of CUDA with model BCL. could not be studicd. Despite this limitation, we can still validate the first claim, which states that employing a class-wise degree of auginentation improves performance based on available data. The heatmaps in Figure-2 represent the growth of augmentation strength over the epochs, serving to validate the original paper's second assumption. Before the 160th epoch, there was a stark difference in LOL scores between the majority classes (0-49) and minority classes (50-100). However, after 200 epochs of training, this contrast is greatly reduced. A holistic view of the evolution of LOL scores across epochs (section 4.1.2) still confirms the original paper's second claim, that using stronger augmentation on majority classes and milder augmentation on minority classes results in better model performance than the opposite strategy.
By measuring the variation of weight L1-norm (section 4.1.3), the reproduction study demonstrates how CUDA effectively addresses the imbalance problem by reducing the variance of weight L1-norm, helping the model to give equal importance to cach class as claimed in the original paper. Through further analysis, we inferred that CUDA improves the balance of feature representation in terms of inter-class distances. In addition, we replicated the feature aligmment gain analysis (section 4.1.4), confirming CUDA's effectiveness in mitigating the imbalance issue by boosting feature alignment, as stated in the original paper. Through further analysis, we inferred that CUDA minimizes intra-class distances between instances of the same class, particularly the tail classes. To validate these two inferences regarding balanced inter-class distances and lowered intra-class distances, we compared the feature representation space of the vanilla and CUDA versions (section 4.2.1). Our findings supported both hypotheses, demonstrating that CUDA improves feature representation by promoting balance in inter-class distances and decreasing the invasion of foreign instances by reducing intra-class distances.
The hyperparameter sensitivity analysis for augmentation probability and acceptance rate on validation accuracy demonstrates the concavity claimed in the original paper (Fiuure-5.. Our findings in (sections 4. 1.6 and 4.1.7) demonstrate the impact of curriculum and class-wise score on CUDA performance, verifying their importance as claimed in the original paper. We discovered that cutout augmentation plays a significant role in the accuracy gain observed when using CUDA, even more than curriculum learning or class-wise score (section 4.2.3). We claim the original authors had used cutout augmentation during training across all models but have not explicitly stated this in the paper. The stated accuracy values in the original paper can only be attained when CUDA is combined with cutout augmentation. This dependency may limit the generalizability of CUDA across different sccnarios. Our analysis of the imbalance ratio indicates that CUDA's 's efficiency diminishes as we move towards dalasets with lower imbalance (section 4.2.3). From this, we can also infer that CUDA is not beneficial when dealing with balanced datasets. The reliance on a larger imbalance ratio and cutout augmentation is shortcoming in the CUDA methodology.
5.1 What was easy and what was difficult
The original paper was really easy to follow. The section on the repository for CIFAR-100-LT was written clearly. The description of the arguments that we pass during training was properly stated. The algorithm that makes up CUDA was completely logical in its implementation. The lack of significant barriers in setting up the code enhances its portability. The dependencies were not mentioned clcarly by the author requiring additional time to find the versions by trial and error. The original paper also included a performance comparison on datasets ImageNet- and Inaturalist-Ie 18, the section of the original repository for ImageNet- LT and Inat-18 contains rodundant code and uncleaned up code. The paper claims to deviate from the model recipes of BCL and NCL to ensure fair comparison. However, it fails to clearly state these deviations, making it difficult. to assess their performance in this study.
13

---

Published in Transactions on Machine Learning Research (06/2024)
5.2
Communication with original authors
The initial allempls to contact the authors through the email IDs given in the paper were not successful. We were able to contact the authors through Linked In in the latter half of the study. The authors were able to clarify our doubts about implementing the component analysis for curriculum learning and class-wise scores. Regrettably, the authors were unable to provide a clear recipe for BCL.
References
Sumyeong Ahn, Jongwoo Ko, and Se- Young Yun. Cuda: Curriculum of data augmentation for long- tailed recognition, 2023.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249 209, 2018
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-clistibuution-awaee margin loss. Advances in neural information processing systems, 32, 2019.
Hsin-Ping Chou, Shil-Chieh Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: Rebalanced mixup. ArXiu, abs/2007.03945 009443, 2020. URL https://api www.nticschollee.e........c.caee htt.:///....orrrrrruurr000000000 22004444
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoangment: Learning augmentation strategies from data. In Procoedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113 123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition workshops, pp. 702-703, 2020.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1706 04552, 2017.
Siming Fu, Xiaoxuan He, Xinpeng Ding, Yuchen Cao, and Hualiang Wang. Uniformly distributed category prototype-guiedd vision-language framework for long-tail recognition, 2023.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21 (9):1633 12841 2009.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5375- 5384, 2016. doi: 10.1109/CC /CV..2016..2.....
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URL https://openreviee: http:///ww....oorrr//a//////////
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for rep-
resentation learning. In International Conference on Learning Representations, 2021. URL https: //openreview.netee http://ntwnt.toormmmdtotttaaaaaa
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13896-13905, 2020.
Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentatiin. In European Conference on Compuler Vision, pp. 580 595. Springer, 2020.
Sungbin Lim, Ildoo Kim, Tacsup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Sculki Park, Youngkyu Hong. Byeongho Heo, Sangdoo Yun, and Jin Young Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6887 6896, 2022.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, al. Balanced meta-softmax for long-tailed visual rocognition. Advances in neural information processing systems, 33:4175-4186, 2020.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding, 2015.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by routing diverse distribution-awwrre experts. In International Conference on Learning Represenlations, 2021. URL
https://openrevierr htt.:///........oooooooooooaaaoa
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and paftern recognition, pp. 9719 9728, 2020.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognilion. In Proceedlings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 6908 6917, 2022.
A Augmentation preset
We use 22 different augmentation operations for CUDA each having its parameter. The details of each of these operations have been described in Table 1. The magnitude parameter divides the augmentation parameter into 30 values linearly. For example, if the max value for Rotate is 30 and the min value is 0, the magnitude of the parameter for Rotate is defined by
Myotate (s) = (30 0)/30 8, thus Mrotate (1) = 1 = (30 0)/30
15

---

Published in Transactions on Machine Learning Research (06/2024)
GRAMPLM
Figure 9: The 22 augmentation operations we use for CUDA
16

---

Published in Transactions on Machine Learning Research (06/2024)
Table 3:
Operation Parameter Description
Operation
Description
Flip On/Off
Flip top and bottom
Mirror On/Off
Flip left and right
Edge Enhancement On/Off
Increasing the contrast of the pixels around the targeted edges
0
-1
0
Detail On/Off
Utilize convolutional kernel
10
-1
0
1
Smooth-On/On
Utilize convolutional kernel
1 5
1
ratoContrast On/Off
Remove specific percent of the lightest and darkest pixels
Equalize On/Off
Apply non-linear mapping to make uniform distribution Negate the image
Invert On/Oft
Gaussian Blur
Blurring an image using Gaussian function with radius [0,2]
Resize Crop
Resizing and center random cropping with scale [1,1.3]
Rotate
Rotate the image with angle [0,30]
Posterize
Reduce the number of bits for each channel in the range [0,4]
Solarize
Invert all pixel values above threshold in the range [0,256]
SolarizeAdd
Adding value and run solarize in the range [0,110]
Color
Colorize grayscale values in the range [0.1, 1.9]
Contrast
Adjust the distance between colors in the range [0.1,1.9]
Brightness
Adjust image brightness in the range [0.1,1.9]
Sharpness
Adjust image sharpness in the range [0.1,1.9]
Shear X
Shearing X-axis in the range [0,0.3]
Shear Y
Shearing Y-axis in the range [0,0.3]
Translate X
Shift. X-axis in the range [0,100]
Translate
Shift Y-axis the range [0,100]
17


Reviews:
[{"review_id": "0mg9fRzBIY", "reviewer": "Reviewer_jwo3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nS1. This work studies a well-motivated problem (long-tailed recognition). this study introduces an algorithm that uses strength-based augmentation in conjunction with a Level of Learning (LOL) score to generate augmented samples. This approach addresses the distinct needs of majority and minority classes, tailoring the complexity of augmentations to achieve improved performance.\n\nS2. Experiments that evaluate CUDA on various LTR models such as CE, CE DRW, LDAM DRW, BS, and RIDE demonstrate the versatility and broad applicability of CUDA on a wide range of base models. This cross-model testing facilitates a deeper understanding of effectiveness with CUDA.\n\nS3. Additional analysis also explores the contribution of Curriculum Learning and class-wise scores to the model's performance. This in-depth analysis allows identifying key factors that influence the outcome and lead to better fine-tuning of the algorithm.\n\nWeakness:\n\nW1. While providing improvements on imbalanced datasets, the efficacy of CUDA diminishes as it is used with datasets having lower degrees of imbalance. This suggests that the advanced features of CUDA might not be essential or beneficial when dealing with balanced datasets. Authors should consider backup strategy for CUDA when handling the datasets not much imbalanced, or provide a criteria on the class-wise distribution for methodoldy selection.\n\nW2. The role of cutout augmentation was found to be significant, potentially overshadowing the contribution of Curriculum Learning and class-wise scores. Such dependency may limit the generalizability of the CUDA approach across different contexts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduce CUDA, which aims at improving the accuracy of Long-Tailed Recognition (LTR) models, particularly with imbalanced datasets. CUDA introduces a class-wise data augmentation technique that employs 'Strength based augmentation' and a score for 'Level of Learning (LOL)' to generate augmented samples. The strength-based augmentation strengthens the augmentation for majority classes and weakens the augmentation for minority classes. The LOL score adjusts dynamically based on the model's prediction success, leading to a form of curriculum learning strategy. \n\nExperiments across different LTR models have shown model validation accuracy improvements when CUDA is used. Moreover, the study provides analytical insights and conducts various evaluations on the approach's effectiveness across different imbalance ratios, hyperparameters, and comparative performance with other augmentation methods. However, the experiments also show that CUDA would not incorporate performance improvement on datasets with lower imbalance, suggesting that it doesn't yield significant improvements on balanced datasets. Furthermore, the results indicate the significant role of cutout augmentation, even overshadowing that of curriculum learning or class-wise scores on the accuracy gain observed when using CUDA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address weakness points that I mentioned.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kCnS5Hl2iM", "reviewer": "Reviewer_89HM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper reproduces what seems to be a substantial portion of the paper and probes additional points in the design space.\n* Experiments and Code are released for reproducibility.\n\nWeaknesses: \n* It is difficult to understand where the paper is going or what the conclusions are. The figure captions don't help with understanding the figures. It is difficult to understand what is expected behavior in the reproduction, what is novel behavior, when deviations from the paper occur, etc.\n* Writing and presentation style can be improved. Paper has many typos and Latex errors that significantly impact clarity of the paper. Commas and periods tend to have unnecessary space around them, while places that require a space do not have one. Informal words inappropriate for technical writing such as \"gonna\" used. Capitalization is used inappropriately.\n* Figures are messy and have inconsistent style. Figure 2 is, for example, a photo that seems directly taken from the CUDA paper. Figure 3 is an image of a table rather than a latex table. Figure 4 requires zooming to see details. Figure 5 has an illegible x-axis. Other figures are small, have visible borders, and in general can be improved with how information is presented. Figure 7, for example, uses 2 different colors (red/green) for the exact same concept, CUDA.\n* First 4 pages are essentially a summary of the CUDA paper, but the reader will likely have to read that paper anyway to make sense of the reproduction. Section 3.1.2 seems less clear than the original explanation, partially because there appear to be typos and the presentation in bullet form lacks motivating explanation.\n* Code could be organized further.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a reproducibility paper for \"CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition\" (Ahn et al., 2023). The paper reproduces aspects of the original paper as well as probes new dimensions of the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Required:\n* Fix clarity issues noted in weaknesses. Please fix the writing to make the paper more clear. Update the figures so they look clean and have summarizing captions.\n* Please state what the intention/goal of each section is so the reader can understand what the takeaway is. The reader should be able to tell if the results are positive or negative without having to reference the original paper.\n\nStrengthen:\n* Quantify the degree of reproducibility achieved", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DWwIHE0dzp", "reviewer": "Reviewer_MBg6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The method description and experimental setup are meticulously clear, offering an understanding of the CUDA technique and its implementation.\n\nWeaknesses:\n\n1. Lack of insight is evident in certain sections, such as Section 4.1.3, where the paper claims a significant decrease in the standard deviation of weight L1 norm. It is essential to delve deeper into these observations to provide meaningful insights into the underlying mechanisms or implications of such findings.\n\n2. There is room for additional evaluation or experimentation, particularly regarding the study on class-wise feature cosine similarity. Exploring this aspect further could potentially yield valuable insights and discoveries that could benefit the broader research community.\n\n3. The motivation behind reproducing the original paper is not adequately clarified. It's essential to elucidate the significance and relevance of replicating the study to the audience, outlining the key contributions or insights that the replication study aims to offer.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper replicated the methodology outlined in the CUDA paper; Conducted experiments on the Cifar100-LT dataset to evaluate the effectiveness of the proposed CUDA technique; and analyzed and compared the performance of various models with and without CUDA augmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. The absence of tags on the red and blue curves in Figure 5 creates ambiguity regarding their meanings. \n2. Figure 3 illustrates that adding CUDA to the method CE +CMO did not result in improvement for the IR = 10. Why claim observe a consistent increase in accuracy?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a reproducibility paper for "CUDA: Curriculum of Data Augmentation for Long‐Tailed Recognition". I think its quality is generally acceptable as a reproducibility paper. Two reviewers vote for acceptance and one for rejection. More specifically, a positive reviewer has left the following post-rebuttal comments:

> The paper is a reproducibility work, which may be useful for those who are interested in studying related algorithms. That said, the paper is rather complex to read otherwise as it lacks a traditional story and the graphs have a lot of distracting detail. The authors did put an effort to clean up the paper presentation; however, some minor issues persist---Figure 2 is still an image from another paper and there are still some typos when scanning through ("kaggle" and "linked-in" are lowercase). Overall, I do see value in this study if someone is specifically interested in the related algorithms, though the results are more difficult to generalize beyond that.

I would like to accept the submission. Since this submission has a sole author, I strongly suggest the author finding a real researcher to proofread the final version of the manuscript (not an LLM), given that reviewers pointed out the formal writing (language and formatting) issue as a major issue that significantly impacted clarity.
Decision: Accept (poster)