LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems
Authors: Pengkun Wang
Zhe Zhao
HaiBin Wen
Fanfu Wang
Binwu Wang
Qingfu Zhang
Yang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Long-tailed learning
Large Language Model
Data Augmentation
Primary area: deep_learning_architectures

Abstract: The long-tailed distribution is the underlying nature of real-world data, and it presents unprecedented challenges for training deep learning models. Existing long-tailed learning paradigms based on re-balancing or data augmentation have partially alleviated the long-tailed problem. However, they still have limitations, such as relying on manually designed augmentation strategies, having a limited search space, and using fixed augmentation strategies. To address these limitations, this paper proposes a novel LLM-based long-tailed data augmentation framework called LLM-AutoDA, which leverages large-scale pretrained models to automatically search for the optimal augmentation strategies suitable for long-tailed data distributions. In addition, it applies this strategy to the original imbalanced data to create an augmented dataset and fine-tune the underlying long-tailed learning model. The performance improvement on the validation set serves as a reward signal to update the generation model, enabling the generation of more effective augmentation strategies in the next iteration. We conducted extensive experiments on multiple mainstream long-tailed learning benchmarks. The results show that LLM-AutoDA outperforms state-of-the-art data augmentation methods and other re-balancing methods significantly.

LLM-AutoDA:
Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems
Pengkun Wangl.2; Zhe 1.30 Haibin Wen', Fanfu Wang", Binwu Wang', Qingfu Zhang?! Yang Wang .22 University of Science and Technology of China (USTC), Hefei, China
Suzhou Institute for Advanced Research, USTC, Suzhou, China
City University of Hong Kong, Hong Kong, China MMrongAAI, Suzhou, China
Lanzhou University, Lanzhou, China
(pengkun wus.c.edu.cn, zz4543 mail.ustc. edu.en, haibin65535 gegigllammmm wangff21 @ lzuu.edu.cn, 2bw2024 @ ustc.edu.co qingfu.zhang @ cityu.edu.hk, angyan @ ustc.edu.cn)
Abstract
The long-tailed distribution is the underlying nature of real-world data, and it presents unprecedented challenges for training deep learning models. Existing long-tailed learning paradigms based on re-balancing or data augmentation have partially alleviated the long-tailed problem. However, they still have limitations, such as relying on manually designed augmentation strategies, having a limited search space, and using fixed augmentation strategies. To address these limi- tations, this paper proposes a novel LLM-based long-tailed data augmentation framework called LLM-AAtODA, which leverages large-scale pretrained mod- els to automatically search for the optimal augmentation strategies suitable for long-tailed data distributions. In addition, it applies this strategy the original imbalanced data to create an augmented dataset and fine-tune the underlying long- tailed learning model. The performance improvement as reward signal to update the generation model, enabling the generation of more effective augmentation strategies in the next iteration. We conducted exten- sive experiments on multiple mainstream long-tailed learning benchmarks. The results show that methods and other re-balancing methods significantly. The code is available in https::// thub. com Lab tomm LLLLLLLLMLAACALCAL LT-AUG.
the validation serves
outperforms state-of-the-art data augmentation
Introduction
As revolutionary technology, deep learning has shown a broad and significant impact on various tasks, including image classification [21], object detection [34], natural language processing [19], and many interdisciplinary research problems [40, 47]. The success of these endeavors relies heavily on the support of large-scale manually curated datasets, e.g., ImageNet [35]. However, for the convenience of training and evaluating models, most artificially constructed datasets typically follow the assumption of uniform distribution, which contradicts the real-world data distribution, i.e., long- tailed distribution. This deviation between the ideal and real distributions has resulted in many deep
Pengkun Wang and Zhe Zhao contributed equally to this work. t Cooresponding authors: Qingfu Zhang and Yang Wang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompts
Fixed DA Strategy
Fixed Strategy
Random DA
Optimal DA Strategy
Class-wise
Class-wise
DA
DA
Result
Result
Result
(a) Traditional Long-tailed DA (e.g,, AutoAugment)
(b) Strategy Fixed Long-tailed DA (e.g., CUDA, DODA)
(c) LLM-driven Long-taie tailed DA (e.g., LLM-AutoDA)
Different long-tailed data augmentation paradigms. (a) The traditional augmentation paradigm randomly samples augmentations from the fixed strategy. (b) The strategy fixed augmen- tation paradigm samples augmentations from the fixed strategy according to the data distribution. (c) The LLM-driven augmentation paradigm combines LLMs with long-tailed learning to learn the optimal augmentation strategy.
Figure/ I:
models trained on balanced datasets failing produce satisfactory results in real-world applications, e.g., they only perform well few classes and ignore many vulnerable classes [4, 36).
To address the ubiquitous long-tailed problem, researchers have been continuously proposing various carefully designed research paradigms. One popular approach is to rebalance the training data by oversampling the tail classes or undersampling the head classes [6, 13]. However, this approach cannot fundamentally address the problem of insufficient global information, even though the data for the tail classes increases significantly. Another line of work focuses on designing specialized loss functions or reweighting strategies to alleviate the impact of class imbalance [11, 5, 37, 39]. However, these methods either introduce additional computational overhead or require careful manual design.
Recently, using data augmentation (DA) to improve long-tailed learning has attracted significant attention from researchers and is considered a viable research paradigm [44]. For example, FASA [46] enhances the tail classes by generating class-level features based on Gaussian priors. Remix [8] achieves this goal through a rebalanced mixup approach. However, these DA-based methods either manipulate high-dimensionall information in the feature space or directly apply tradi- tional transformations (e.g.. flipping, cropping, and rotation) to expand the training and generate diverse data, without considering the underlying relationships between data augmentation and class classes. To avoid ineffective augmentation, some studies suggest applying different augmentations to different classes [41]. Typically, CUDA [2] improves the overall performance of models by dynamically adjusting the augmentation intensity for each class during training. Considering the issue of augmentation method, thereby avoiding weak classes being sacrificed. Unfortunately, they still have significant limitations: (i) these strategies are often based munually designed human knowledge and experience, which may be suboptimal for specific data and tasks. (ii) the search space of these strategies is often limited. (iii) these fixed strategies lack flexibility to adapt to changes in the data distribution during the training process.
in augmentation, DODA [39] allows each class to choose its own suitable
To address the above limitations, we leverage the recently popular large language models (LLMs)) [29, 24, 45] to provide augmentation suggestions for long-tailes learning. We first designed feasible and straightforward framework called SimpleLLM, which guides the LLM to generate augmentation strategies and apply them to long-tailed learning by providing specific prompts. Analysis revealed that the augmentation strategies generated by SimpleI.LM are comparable to the effectiveness of CUDA [2] and DODA [39]. Figure/ illustrates the differences between this framework and previous methods.

---

Furthermore. inspired by AutoML [17]. particularly automated data augmentation [10], we propose LLM-AutoDA, novel LLM-based long-tailed data augmentation framework. LLM-AutoDA leverages large-scale pretrained models to automatically search for the optimal augmentation strategies suitable for long-tailed data distributions. Specifically, we first define broad search space that includes augmentation operations and their parameters. Then, we train an augmentation strategy generation model that generates augmentation strategies based on the class-wise statistics the long- tailed data. This strategy is applied to the original imbalanced data to create augmented dataset, which is used to fine-tune the underlying long-tailed learning model. Importantly, the performance improvement the validation set serves the generation more effective augmentation strategies in the next iteration. This process is repeated until the performance converges or the computational budget is exhausted.
reward signal to update the generation model, enabling
Compared to previous long-tailed data augmentation methods, LLM-AutoDA offers several advan- tages: (i) it leverages LLMSs to automatically learn augmentation strategies tailored to the charac- teristies of long-tailed data, without relying on human expertise.
(ii) has a more extensive search space, allowing it to discover more novel strategies. (iii) it can dynamically adjust the augmentation strategies based on performance feedback during the training process, providing flexibility and robust- ness. Extensive experiments multiple mainstream long-tailed learning benchmarks demonstrate that LLM-AutoDA outperforms sfate-ofthhe-aee data augmentation methods and other rebalancing techniques significantly.
The main contributions of this work are summarized as follows:
New augmentation paradigm: We combine LLMs with long-tai tail data augmentation for the first time, providing a novel perspective for efficient long-tail learning.
New automated framework: We propose novel AutoML framework called LLM-AutoDA, which automates the search for effective data augmentation strategies for long-tailed learning, significantly reducing the cost of manually designing augmentation strategies.
Compelling empirical results: We conduct extensive experiments multiple mainstream long- tailde benchmarks, demonstrating the superiority of L.LM-AutoDA compared to state-of-the-art methods.
In-depth analysis and insights: We provide detailed analysis and insights into the discovered augmentation strategies, guiding future research in long-tailes learning.
2 Related Work
2.1 Long-tailed Learning (LTL)
Various approaches have been proposed to address the long-tailed learning problem, including re- balancing the training data through over-sampling the tail classes [6. 13] or under-sampling the head classes [12. 14], modifying loss functions or adjusting class weights during training [22, 11], and decoupling representation and classifier learning [18]. Among these methods, data augmentation has emerged as a promising solution for long-tailed learning [8, 20, 25, 46). The key idea is to generate additional samples for the tail classes to alleviate the data imbalance issue. Recent research has attempted to design sophisticated strategies by observing performance changes during the training process to adjust augmentation operator [39] or intensity [2]. However, these methods still rely on hand-crafted augmentation strategies that may not be optimal for the specific long-tailed data and have limited search space for discovering novel and effective strategies. In contrast, LLM-AutoDA automatically learss data augmentation strategies tailored to the long-tailed data distribution without manual design.
2.2 Large Language Models (LLMs)
Large language models (e.g., BERT [19], GPT [29, 30, 31, and T5 [31]) have achieved remarkable success in various natural language processing tasks. They exhibit strong generalization abilities and can be easily fine-tuned for downstream tasks with limited labeled data [16, 28]. Researchers have explored the potential of L.MSs in automating algorithm design and implementation, such as
generating source code [7], optimizing hyperparameters [42]. and designing neural architectures [32]. However, the interaction between large and small language models and its impact on improving the
3

---

design of small models have been less explored, particularly in the context of data augmentation for long-tailed learning. LLM-AutoDA aims bridge this gap by harnessing the knowledge and generative capabilities of large language models to discover effective data augmentation strategies tailored long-tailed distributions automatically. By defining a rich search space augmentation operations and training an augmentation strategy model conditioned on the taasswiise statistics, LLM-AutoDA can generate adaptive and optimized augmentations specifically designed for the given long-tailed data. This novel approach opens up new possibilities for leveraging the interaction between large and small language models to improve the design of machine learning algorithms in various imbalanced learning scenarios.
3 LLM x LTL: Can LLMs Provide DA Strategies for Long-tailed Learning?
In this section, we attempt analyze whether LLMs can be applied long-tailed learning and how to implement this learning paradigm.
SimpleLLM. As shown in Figure 1(b), strategy fixed DA is the current mainstream paradigm for long-tailed data augmentation. It utilizes carefully designed augmentation strategies to dynamically adjust augmentation operators intensities during the training process, allow- ing different classes to choose advantageous augmentation methods. In this paradigm, the key step is to design high-quality augmentation strategy. When dealing with balanced data distributions, we often employ class-independent augmentation strategies, which apply the same data augmentation ing with imbalanced data distributions, this class-independent augmentation strategy can poten- tially sacrifice certain classes, thus requiring the design of class-dependent augmentation strate- gies. However, the manual design process for such strategies is highly complex and costly.
all classes. However, mentioned in DODA [39], when deal-
Recent research has shown that LLMs can replace many manually engineered
Non-parametric Prompt
LLMS
Augmentation Strategy
tasks [7]. Inspired by
Question:
this, we first designed simple yet efficient paradigm for generating augmentation strategies called Simplel.LMM As shown in Figure 2, initially constructed a data augmentation-thhen prompt from the perspective of prompt engineering, including task description, algorithm input, algorithm output, parameter interpretation, etc. We then input this prompt into pre-trained LLMs to generate a functional function that conforms to the prompt, i.e., an algorithm implementation containing augmentation strategies. Finally, this augmentation strategy is applied the conventional training process of long-tailed learning. is worth noting that this paradigm allows us to generate multiple augmentation strategies suitable for long-tailed learning at low cost.
requested design
novel algorithm given set of augmentation techniques
Long-tailed nang-tailed Model
Figure 2: Strategy generation paradigm of SimpleLLM.
Comparative Analysis. To further validate the effectiveness of the augmentation strategies generated by this paradigm, we conducted experiments several mainstream long-tailed learning baselines and integrated SimpleLLM with them. addition, we compared the latest state-of-the-al long-tailed DA methods, CUDA and DODA, under the same settings. The experimental results, as shown in Figure 4, reveal that SimpleLLM achieves acceptable average performance, comparable to CUDA and DODA, indicating that LLMs can be used as generators of augmentation strategies to enhance the performance long-tailed learning.
1FFAR-100-L7 (IR=100) dataset. We selected
Under this paradigm, we believe that with appropriate prompts, augmentation strategies similar to CUDA and DODA can also be generated by L.MMs. other words, within search space, we can obtain multiple similar locally optimal strategies.

---

Crossover Prompt
Update
Grade
Result
Question:
You requisted to design a novel algonthmm sugmentation techniques,
Sampling
Strategy Repository
Update
Result
new elgorithm that has a totally different form from ones.
Strategy Grade
Stratagy Grade
Tranffer
Strategy Grade
Tet
Mutation Prompt
Question:
Store
Please assist mh in creating new algorithm different can be modified version of the algorithi provided.
Inthialtattin
Input
Verify
C LLMS
Augmentation Strategy
Long-tailed Model
Training
Result
Figure 3: Overview of LLM-AutoDA, LLM-AutoDA leverages large-scale pretrained models to automatically search for optimal augmentation strategies suitable long-tailed data distributions.
4 LLM-AutoDA: A Resourceful Adviser for Long-tailed Learning
4.1
Framework
The overall framework of LLM-AutoDA is illustrated in Figure 3. The framework consists of two interactive modules: the L.L.-based data augmentation strategy generation module and the long-tailed learning training and evaluation module.
LLM-based Data Augmentation Strategy Generation. As shown in Figure 3 (left, pink), to design the DA strategies automatically, LLM-AutoDA incorporates pre-trained LLM C as search operator. Using prompt engineering techniques, series of prompt templates are designed to incorporate prior knowledge about data augmentation into the generation process of LLM. LLM generates diverse data augmentation strategies based on these prompts, including both natural language descriptions and Python code implementations. Furthermore, the generated strategies are stored in a strategy pool and interact with the long-tail learning model search for the optimal data augmentation strategy.
Long-tailed Learning Training and Evaluation. As shown Figure (right, pink), LLM- ALM-AutDDA utilizes pretrained long-tailed learning model . for fine-tuning on given long-tailed distributed dataset D. At the beginning of each training epoch, the algorithm adaptively determines the DA operator A. and DA intensity E, for each class based on information such the accuracy of each class in the previous epoch and historical accuracy.
The key aspect of LLM-AutoDA lies in the synergy between the DA strategies generated by LLMs and the long-tailed learning model. This close interaction allows the discovered DA strategies dynamically align with the model training process, effectively enhancing the performance of long-tailed learning in targeted manner.
Strategy Evaluation
To evaluate the performance of candidate data augmentation strategies, we insert them into the model training process, conduct small amount of additional training on the training set, and then test accuracy on the validation set, using the accuracy as the fitness score for the algorithm. Assuming the data augmentation function generated by L.LM denoted as fang and the training-testing function denoted as T. the evaluation process can be represented follows:
Fitness (faug) T(faug: Cckp, N1t)
(I)
5

---

where Cckp is the starting checkpoint epoch number and Ne is the epoch number of fine-tune. The function T injects fang into the training flow:
T(fugg Cokp, N1)
(eCCealFFine tune(faug. Derain: Cckp: N)))
(2)
The Finelune function starts training from the enky checkpoint and performs N₃x epochs of training on the training set trrain- At the beginning of each epoch, it dynamically selects data augmentation methods for each class using faug.
A (4)
(1-1), E( fang (W((-), aa a
( A (t)
(3)
Here, A represents the DA selection matrix for class c at time 1, E represents the corresponding augmentation intensity, w represents the weights of each augmentation method on class from the previous time step, a (t-1) represents the accuracy of class c at time - 1, and H(- represents the historical accuracy of class when using different augmentation methods in the previous step.
After training, the model is evaluated using the no augmented validation set Pval. and the overall accuracy AcCeal is obtained as the fitness score for faug.
Fitness(fawa = Acc ['inetune(fawa Dtrain, Eckp, N Nt), Dvat)
A higher fitness score indicates better performance of the algorithm on long-tailed distributions. By injecting candidate algorithms into the real training process and evaluating them on the validation set, we can accurately and efficiently measure their actual ability to address the long-tailed problem.
4.3 LLM-based Search Operator
I. LLM-AuttDA leverages Pretrained Language Models (PLMs) to automatically generate data aug- mentation algorithms. To guide the PLM in generating algorithms that meet specific requirements, we employ prompt engineering techniques and carefully design a series of prompts. By incorporating task descriptions, input-output formats, novelty requirements, and other prior knowledge through prompts, constrain the generation process of the PLM within the desired search space. We design the following three types of search operators, corresponding to different prompt templates:
Initialization operator I: Based on the task description prompt Prask and the knowledge base
(0) N'
of data augmentation X., set of randomly initialized population algorithms A i=1 is generated.
Crossover operator E: Building upon Presk. Np parent algorithms A 1 N from the current population are used references. along with the incorporation of knowledge base X. The PLM is required to generate N₂ new algorithms A ( = N. that are different in both form and logic from the existing algorithms, thereby expanding the search space.
Mutation operator M: Based on Prosks Nm individuals ( N...
#=1 are selected from the current population, and local improvement directions are provided. The PLM is tasked with generating
a mutated algorithm A for each A within its neighborhood for further exploration.
Taking the crossover operator E as an example, its prompt PF: can be represented as follows:
PE(Prask: (2 KC, Dume)) = Ptask + Pref (14 1 + (KC) +CC)+Paays Passy Pormat fune)
(5)
where Presk represents the task description, A;i N represents N parent algorithms, and Dfune represents the domain of the objective function. Prey formats the parent algorithms into reference code, Pasfs requires the generation of new algorithms that are completely different from the existing ones, and P. Pormat specifies the input and output of the objective function.
Once we have obtained the prompt PE. we input it into the pretrained language model C, and as result, we obtain N. new crossover algorithms.
{4/)) N. [(Prs ( K, Dfune))
(6)
Each Aa typically consists of a natural language description of the algorithm and its corresponding Python code implementation. Similarly, the prompts P, and Par for the initialization operator / and the mutation operator M can constructed in a similar manner, with the main difference lying in the introduced prior information.

---

5 Experiments
5.1 Experimental Settings
Datasets and Metrics. Like most long-tailed learning methods, we conducted experiments on several mainstream long-tailed learning datasets, including CIFAR-IO0-LT [5], ImageNet-LT [26]. and iNatu- ralist 2018 [38]. Among them, CIFAR-I00-LT is the long-tailed version of CIFAR-100, with various imbalance ratios. To validate the effectiveness of LLM-AutoDA addressing the long-tailed problem, we selected three testing environments: 50, 100, 200. Compared to CIFAR-100-LT, both ImageNet- LT and iNaturalist 2018 have more classes and larger data sizes. It worth noting that, similar to SFARR110-LL1 ImageNet-Li LT is a long-tailes version artificially constructed from the well-known ImageNet [35] dataset. On the other hand, iNaturalist 2018 a naturally occurring long-tailed dataset collected from the real world. We used the official complete versions of these datasets, and detailed in- formation about the datasets is provided in Appendix B. We use Top-1 accuracy as the evaluation met- ric and provide the performance of subsets based on the class divisions provided by the official datasets. Baselines. Following the settings of CUDA [2]. we considered various research theories when selecting the base- lines. addition to the classic cross-entropy loss (CE) [15],
CIFAR-100-LT (IR=100)
CUDA
DODA
SimpleLLM
Top-1 Accuracy
48
we also validated different data augmentation methods on other baselines, such as
46
loss-based re-balancing methods:
CE-DRW [5], LDAM-DRW [5], Balanced Softmax (BS) [33]. and model-based re-balancing methods: RIDE [43]. BCL [48]. In terms of data augmenta- tion methods, we compared LLM-AutoDA with the latest SOTA methods: CUDA [2] and DODA [39]. We observed their advantages and disadvantages by combining these DA methods with the long-tailed baselines. Appendix A.
44
42
CE
CE-DRW LDAM-DRW Baselines
RIDE
BCL
Figure 4: Average accuracy (%) on CIFAR-100-LY dataset (Im- balance ratio=100) with CUDA and DODA. SimpleLLM is parable to the effectiveness CUDA and DODA when combined with long-tailed learning baselines.
The relevant descriptions of the baselines are also provided in
Implementation Details. All our models are implemented based on PyTorch [27]. We trained and evaluated the models on 2 NVIDIA Tesla A100 GPUs and reported the experimental results. We utilized the powerful gpt-3.5-turbo for strategy generation and employed AEL [23] for strategy optimization. In the experimental process, we first trained the models for 50 epochs without using augmentation strategies, then continued training with augmentation strategies for an additional 20 epochs, employing novel evaluation mechanism. Additionally, during the final evaluation stage of long-tailed learning, we adopted the same settings as DODA [39] for all baseline methods and our approach.
5.2 Comparison with the State-of-the-art
Results on CIFAR-100-LE We first evaluated LLM-AutoDA and other long-tailed data augmentation (DA) methods on CIFAR- SEEARR-10-LLYY 100-LT dataset (IR = 50, 100). The experimental results are shown in Table From the results, can be observed that both SimpleLLM and the improved version LLM-AutoDA significantly improve the global accuracy of the model compared to the original long-tailed learning baseline, achieving robust improvements. In the horizontal comparison with long- DA methods CUDA [2] and DODA [39], as analyzed earlier. SimpleLLM achieves comparable performance to the former through a non-optimized way. This indicates that within our framework, a locally optimal strategy can replace a carefully designed complex strategy. In addition, LLMM-AutDA brings more significant and stable gains, reflecting that existing long-tailed DA methods may be suboptimal strategies within our augmentation strategy space, while
can provide the
7

---

Table 1: Accuracy (%) on CIFAR-100-LT dataset (Imbalance ratio=(50, 100)) with SOTA DA methods. Blod indicates the best performance while underline indicates the second best. (+) and (-) indicate the relative gain.
Method
Head
Medium
Tail
All
Head
Medium
Tail
All
[E[15] CE + CUDA
63.9
36.2
15.2
43.8 (+0.0)
65.6
36.2
40.1 (40.0)
68.3
38
13.7
46.2 (+2.4)
70.7
41.4
9.3
42.0 (+3.9)
(+4.2)
74.8
(+6.4)
CE + SimpleLLM
21.4
39.9
13.1
48.0 (+4.2)
72.5
44.9
9.8
44.0 (45.9)
CE + LLM-AutoDA
72.3
40.0
14.1
48.6 (44.8)
74,9
45.3
9.6
45.0 (+6.9)
CE-DRW [5]
60.6
39.0
22.9
45.0 (+0.0)
63.4
41.2
15.7
41.4 (+0.0)
CE-DRW CUDA
63.8
48.0
37.0
52.5
63.5
46.9 (45.5)
CE-DRW DODA
63.4
47.4
38,9
52.5 (47.5)
60.2
51.9
29.6
48.1 (46.7)
CE-DRW SimpleLLAM
62.3
49.4
38.8
52.7 (47.7)
62.1
49.6
27.9
47.5 (+6.1)
CE-DRW LLM-AmoDA
63.1
48.4
39.3
52.8
62.9
50.7
29.9
(47.3)
LDAM-DRW [5]
63.0
41.2
25.1
47.2 (+0.0)
62.8
42.6
21.1
43.2 (+0.0)
LDAM-DRW CUDA
66.2
46.2
26.4
50.8 (43.6)
66.0
49.5
22.1
47.1 (+3.9)
LDAM- DRW DODA
64.7
(+3.3)
(+5.1)
LDAM-DRW SimpleLLM
65.1
45.2
27.3
50.1 (+2.9)
65.7
49.4
23.9
47.5 (+4.3)
LDAM-DRW LLM-AutoDA
66.7
46.1
27.4
51.2 (44.0)
66.7
50.1
26.3
48.8 (45.6)
BS [33]
60.3
41.3
34.3
47.9 (+0.0)
$9.6
42.3
23.7
42.8 (+0.0)
BS CUDA
63.6
48.4
37.3
52.7 (44.8)
62.5
49.1
29.4
47.9 (45.1)
BS + DODA
62.2
51.2
41.5
54.0 (+6.1)
63.1
49.3
31.2
48.7 (45.9)
BS + SimpleLLM
62.4
37.7
52.4 (44.5)
62.4
48.8
30.6
(45.3)
BS L.A--AuODA
63.3
50.5
40.2
53.9 (+6.0)
63.3
50.0
31.0
49.0 (+6.2)
RIDE [43]
65.7
47.7
31.8
52.2 (+0.0)
65.7
48.6
25.0
47.5 (40.0)
RIDE CUDA
67.8
47.0
33.4
53.1 (+0.9)
67.9
$1.2
27.6
50.0 (+2.5)
RIDE DODA
68.2
29.3
$2.1
68.7
25.7
(+2.1)
RIDE SimpleLLM
67.3
46.8
30.8
52.3 (40.1)
69.3
48.8
25,4
49.0 (+1.5)
RIDE LLM-AutoDA
67.1
47.3
32.7
52.8 (40.6)
69.1
50.2
28.1 1
50.2 (42.7)
BCL [48]
61.6
43.1
34.3
49.1 (+0.0)
63.1
42.9
23.9
44.2 (40.0)
BCL CUDA
64,0
47.4
39.4
52.7 (43.6)
64.7
49.7
(44.6)
BCL. DODA
64.9
48.0
40.6
53.6 (+4.5)
66.0
50.7
33.8
51.0 (+6.8)
SimpleLLM
54.0 (44.9)
(44.8)
BCL LLM-AAD
64.9
49.2
44.1
54.8 (+5.7)
66.6
50.6
33.1
51.0 (+6.8)
GFTA.5
CFT4
Cladbo1 Oprs
Orele
170
Eogelogoiiion with Bodatio
Tach popolatio Evedatio
Figure 5: Impact of different LLMs the performance of learn- ing models.
Figure 6: Impact of differ- ent population numbers the mutation prompts.
Figure 7: Impact of differ- ent population numbers in the crossover prompts.
optimal augmentation strategy through continuous optimization. To evaluate the effectiveness of LLM-AutoDA in highly imbalanced scenarios, we adjusted the imbalance ratio to 200 and conducted comparative experiments in Appendix D.
We also conducted comparative experiments on large-scale benchmark datasets, ImageNet-LT and iNaturalist 2018. As expected, different long-tailed learning methods showed significant performance improvement when integrated with LLM-AutoDA. Similar to the highly imbalanced setting mentioned earlier, both of these large-scale datasets are inherently highly imbalanced. Therefore, the augmentation strategies provided by LLM- AutoDA can consistently demonstrate superiority in various imbalanced environments. Importantly, I.L.-AutoDA does not rely on meticulous manual design, which reduces the optimization cost on large-scale datasets.
Results ImageNet-LT and iNaturalist 2018.

---

Table 2: Accuracy (%) ImageNet-LT and iNaturalist 2018 datasets with SOTA DA methods. Blod indicates the best performance while underline indicates the second best. (+) and (-) indicate the relative gain.
Image
iNaturalist 2018
Mothod
Head
Medium
Tail
Head
Medium
Tail
CE
64.0
33.8
5.8
41.6 (+0.0)
73.9
63.5
55.5
61.0 (+0.0)
CE CUDCUDA CUDA
67.1
47.1
13.4
47.2 (+5.6)
74.6
65.0
57.2
62.5 (+1.5)
DODA
67.4
47.5
13.9
48.1 (+6.5)
74.9
66.0
58.4
63.6 (+2.6)
CE+ LLM-AuteDA
68.2
47.1
14.3
50.4 (+8.8)
75.1
66.3
58.9
64.0 (+3.0)
CE-DRW [5]
61.7
47.3
28.8
50.1 (+0.0)
68.2
67.3
66.4
67.0 (-0.0)
CE-DRW CUDA
61.7
48.4
30.5
51.1 ( (+1.0)
68.8
67.9
66.5
67.4 (-0.4)
CE-DRW DODA
62.4
48.5
31.3
52.2
69.0
68.2
67.8
68.2
CE-DRW LLM-AuteDA
62.8
48,3
31.7
51,6 (+1.5)
68.8
68.8
68.1
68,7 (41.7)
L.AM-DRW [5]
60.4
46.9
30.7
49.8 (+1.0)
66.1 (-(1.0
LDAM-DRW CUDA
48.2
31.2
51.5((1.7)
66.8
67.3 (+1.2)
LDAM-DRW DODA
63.7
48.6
31.9
52.4 (+2.6)
68.6
68.1
67.9
68.7 (+2.6)
LDAM-DRW LLM-AutoDA
63.3
49.4
32.4
52.5 (+2.7)
68.0
69.4
68.6
69.5 (+3.4)
BS [33]
60.9
48.8
32.1
(+0.0)
65.7
67.4
67.5
67.3 (+0.0)
CUDA
61.8
49.1
31.8
51,5
67.6
68.2
68.3
68.2 (+0.9)
BS DODA
61.9
49.5
32.4
32.0(++1.0)
68.1
68.9
69.5
69.4 (+2.1)
BS LLM-AmoDDA
62.5
50.0
32.8
52.5 (+1.))
68.0
69.1
69.9
69.8 (+2.5)
RIDE [43]
64.9
50.4
34.4
53.6 (+0.0)
70.4
71.8
71.8
71.6 (+0.0)
RIDE CUDA
66.0
51.7
34.7
54.7
70.6
72.6
72.7
72.4 (+1.4)
RIDE DODA
66.6
$1.9
35.9
55.8 (+2.2)
70.9
72.4
73.9
73.7 (+2.8)
LLM-AutoDA
52.3
(+2.9)
73.8
73.9 (+3.0)
[48]
53.5
(+0.0)
72.4
71.8
71.8 (+0.0)
BCL CUDA
66.8
53.9
36.6
56.3 (+0.7)
70.8
72.7
72.0
72.2 (+0.4)
BCL. DODA
66.9
54.1
37.4
56.9 (+1.3)
71.2
73.2
73.4
73.7 (+19)
BCL LLM-AutoDA
55.1
38.3
(+19)
70.9
73.6
74.7
74.2 (42.4)
5.3 More Analysis and Discussion
Do different LLMs produce differentiated effects? In the aforementioned experiments, we used GPT-3.5 [3] as the LLM to respond to the designed prompts. To analyze whether LLM-AutoDA is dependent specific LLM models (e.g.. GPT-3.5), replaced the LLM model in LLM-AutoDA with other popular methods such as GPT-4 [i] and Claude-3-Opus, and conducted experiments. The experimental results, shown in Figure 5, demonstrate that all three LLMs exhibit consistent performance trends when selecting augmentation strategies in different score ranges. For instance, the augmentation strategies with scores around 12, while
they all show high performance augmentation strategies with excessively high scores lead to performance degradation across the three LLMs due to insufficient diversity.
Do different population numbers have an impact on perfor- mance? In ALM-AuttDD
employed two different types of prompts: crossover prompts and mutation prompts. Crossover prompts involve transforming multiple parent populations into a single population, while mutation prompts replace the current augmentation strategy with equivalent one.
40
30
Throughout the iterative process of framework evolution, when performing crossover and mutation operations, we need to spec- ify the number of populations generated each time. Figures and 7 illustrate the scores of strategies generated by two dif- ferent mutation prompts (ml, m2) and two different crossover prompts (cl, c2), respectively. It can be observed that different prompts exhibit consistent trends in score variations. Particu- larly, compared to el, e2 demonstrates higher score variance, which indirectly reflects the bias in its prompt content.
11
0.1 Intonsiry
Operater
0.3
3
0.4
Figure 8: Visualization of the pro- cess of finding the optimal so- lution for different augmentation paradigms.
Why are fixed strategy methods often local optima? Strategy fixed data augmentation methods, such as CUDA and DODA, aim adapt to long-tailed distributions by dynamically adjusting the augmentation operators or intensities. However, their focus is limited. We visualized the loss variations when searching for optimal augmentation strategies using different paradigms. From Figure 8, it can be observed that CUDA (i.c., green plane) and DODA (i.c., red

---

plane) can only search for local optimal strategies on single plane, while LLM-AutoDA is capable of flexibly searching for the points with the lowest loss across the entire curved surface to obtain global optimal solution.
6 Conclusion
Existing paradigms for long-tailed learning have partially alleviated the long-tailed problem but still have limitations. To address this, this paper presents I.LM-driven long-tailed data augmen- tation framework called LLM-AutoDA, which utilizes large-scale pre-trained language models to automatically search for data augmentation strategies optimized for long-tailed data distributions. Ex- periments on multiple mainstream benchmark datasets demonstrate that LLM-AutoDA outperforms state-of-the-art methods.
Acknowledgements
This work was supported by the Natural Science Foundation of China Youth Project (No. 62402472), the Natural Science Foundation of Jiangsu Province of China Youth Project (No. BK20240461), the Research Grants Council of the Hong Kong Special Administrative Region, China (GRF Project No. CityU 11215723), National Natural Science Foundation of China (No.62072427, No. 122299270. the Project of Stable Support for Youth Team in Basic Research Field, CAS (No. YSBR-005), and Academic Leaders Cultivation Program, USTC.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2330308877, 2023.
[2] Sumyeong Ahn, Jongwoo Ko, and Yo-Young Yun. Cuda: Curriculum of data augmentation for long-tailed recognition. In The Eleventh International Conference on Learning Representations, 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-259, 2018.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabll-distribution-aae margin loss. Advances neural information processing systems, 32, 2019.
[6] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321 357, 2002.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2xXv:22007.03374 2021.
[8] Hsin-Ping Chou, Shih-Chich Chang, Jia-Yu Yu Pan, Wei Wei, and Da-Cheng Juan. Remix: rebalanced mixup. In European Conference Computer Vision, pages 95-110. Springer, 2020.
[9] Peng Chu. Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long- tailed data. In European Conference on Computer Vision, pages 694-710, Springer, 2020.
[10] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: the Learning augmentation strategies from data. Proceedings the IEEF/CVF Conference on
Computer Vision and Pattern Recognition, pages 113-123, 2019. 2019
10

---

[11] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 0268-9277, 2019.
loss based
[12] Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In Workshop on learning from imbalanced datasets II, volume 11, 2003.
[13] Hui Han, Wen-Yuat Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference intelligent computing. pages 878 887. Springer, 2005.
[14] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering. 21(9):1263-1288 2009.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- tion. arXiv preprint arXiv:1801.0614 14, 2018.
[17] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
[18] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for loig-tailed recognition. arXiv preprint arXiv: 2019.
[19] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[20] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13896-13905, 2020.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[22] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988 2988, 2017.
[23] Fei Liu, Xialiang Tong, Mingxuan Yuan, and Qingfu Zhang. Algorithm evolution using large language model. arXiv preprint arXiv:2311.15249, 2023.
[24] Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning, 2024.
[25] Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2970-2979) 2020.
[26] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[27] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems Workshops, 2017.
11

---

[28] ME Peters, M Neumann, M lyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer. Deep contextualized word representations. arxiv. org. arXiv preprint arXiv:1802.05365 2018.
[29] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):, 2019.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(14)): 2020.
[32] Md Hafizur Rahman and Prabuddha Chakraborty. Lemo-nade: Multi-parameter neural architec- ture discovery with llms. arXiv preprint arXiv:2402. 18443, 2024.
[33] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International joueraalionaJouualll of computer vision, 1 115(3):11-2522 2015.
[36] Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin- Yan Han, and Yu-Feng Li. Long- tail learning with foundation model: Heavy fine-tuning hurts. In Proceedings of the 41 International Conference on Machine Learning, pages 450144555 450099, 2024.
[37] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11662-11671, 2020.
[38] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 8769-8778, 2018
[39] Binwu Wang, Pengkun Wang, Wei Xu, Xu Wang, Yudong Zhang, Kun Wang, and Yang Wang. Kill two birds with one stone: Rethinking data augmentation for deep long-tailed learning. In The Twelfth International Conference on Learning Representations, 2024.
[40] Pengkun Wang, Chuancai Ge, Zhengyang Zhou, Xu Wang, Yuantao Li, and Yang Wang. Joint gated co-attention based multi-moda networks for subregion house price prediction. IEEE Transactions on Knowledge & Data Engineering, 35(02):1667-1680 2023.
[41] Pengkun Wang, Xu Wang, Binwu Wang, Yudong Zhang, Lei Bai, and Yang Wang. Long-tailed time series classification via feature space rebalancing. In International Conference on Database Systems for Advanced Applications, pages 151-166. Springer, 2023.
[42] Siyin Wang, Shimin Li, Tianxiang Sun, Jinlan Fu, Qinyuan Cheng, Jiasheng Ye, Junjie Ye, Xipeng Qiu, and Xuanjing Huang. Llm can achieve self-regulation via hyperparameter aware generation. arXiv preprint
240.2212..112.11 2024.
[43] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recogni- tion by routing diverse distribution-awarr experts. In International Conference on Learning Representations, 2021.
[44] Wei Xu, Pengkun Wang, Zhe Zhao, Binwu Wang, Xu Wang, and Yang Wang. When imbalance meets imbalance: Structure-driven learning for imbalanced graph classification. In Proceedings of the ACM on Web Conference 2024, pages 905-913, 2024.
12

---

[45] Shunyu Yao. Fei Liu, Xi Lin. Zhichao Lu, Zhenkun Wang, and Qingfu Zhang. Multi-objectinn evolution of heuristic using large language model. arXiv preprint arXiv:2409.16867 2024.
[46] Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
[47] Guibin Zhang, Kun Wang, Wei Huang, Yanwei Yue, Yang Wang, Roger Zimmermann, Aojun Zhou, Dawei Cheng, Jin Zeng, and Yuxuan Liang. Graph lottery ticket automated. In The Twelfth International Conference on Learning Representations, 2024.
[48] Jianggang Zhu, Zheng Wang. Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Bal- anced contrastive learning for long-tailed visual recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6908 6917, 2022.
13

---

Appendix
LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems
The content of the Appendix is summarized follows:
1) in Sec. A, we summarize long-tailed learning baselines and data augmentation baselines.
2) in Sec. B, we demonstrate the details of datasets.
3) in Sec. C, We present the pseudocode corresponding to the LLM-AAttDA.
4) in Sec. D, we illustrate more detailed empirical results and analyses.
5) in Sec. E, we show the prompts used in our framework.
6) in Sec. F, we discuss the limitations of our framework.
7) in Sec. G, discuss the broader impacts of our framework.
A Baselines Details
Cross Cross-Entropy Loss (CE) [15] As classic classification loss function, cross-entropy (CE) is widely used both balanced and imbalanced data distributions, directly computing the loss based on the true labels of samples and the predicted probability distributions of the model. Although simple and effective, CE tends to overly focus on head classes while overlooking tail classes in long-tailed scenarios.
Loss-based Re-balancing Strategies
CE-DRW [5] combines seighitting of training samples based the inverse class frequency and gradient reversal that considers class frequencies during backpropagation, enhancing the focus on tail classes.
LDA M-DRWW [5] further improves upon CE-DRW by introducing learnable amplification factors automatically adjust the weight of each class, better optimizing the inter-class balance. Balanced Softmax (BS) [33] re-weights the logits based on the prior class probabilities, giving higher attention tail classes.
Model-based Re-balancing Strategies
RIDE [43] ensembles multiple expert models, each focusing different class distributions, and
adaptively combines their outputs based on the test data to adapt to distribution shifts.
BCL [48] for CNN classifiers, optimizes the losses of head and tail components while considering intra-class variance and inter-class distances to enhance the separability of classification boundaries.
Data Augmentation Methods
CUDA [2] proposes contrastive learning- based automatic data augmentation method that gener- ates transformations during training, significantly boosting performance in long-tailed scenarios. DODA [39] dynamically adjusts the data augmentation strategies for different classes from distribution perspective, ensuring sufficient augmentation for tail classes while avoiding over- augmentation for head classes.
The above baseline methods offer unique insights into addressing long-tailed distributions, laying the foundation for research. The experimental section will comprehensively compare and analyze the performance of these methods across different datasets and evaluation metrics.
B Evaluation Datasets
To comprehensively evaluate the effectiveness of our proposed method, we conduct experiments four representative long-tailed datasets: CIFAR 100-LT, ImageNet-LT, iNaturalist 2018, and Places365-LT. These datasets cover diverse domains and exhibit varying degrees of class imbalance, providing comprehensive and challenging testbed for long-tailed learning algorithms.
14

---

Table 3: Statistics of the long-tailed datasets.
Dataset
# Classes
# Train
# Test
Max Imbalance Ratio
CIFAR 100-LT
100
50,000
10,000
100
ImageNet-LT
1,000
115,846
50,000
256
iNaturalist 2018
8,142
437,513
24,426
500
the long-tailes version the renowned CIFARI00 dataset, comprising 60,000 32 32 color images across 100 classes. The long-tailed distribution is induced by exponentially decreasing the number of samples per class, with a maximum imbalance ratio 100.
CIFARI00-LT
ImageNet-LT [26] is long-tailed subset of the large-scale ImageNet dataset, containing 115,000 images spanning 1,000 classes. The class cardinalities follow Pareto distribution with 6, leading maximum imbalance ratio of 256.
Natuuralsst 2018 [38] is real-world dataset reflecting the long-tailed distribution nature, com- prising approximately 450,000 images across 8,142 species categories. Due the drastic variation in the number of images per species, this dataset has maximum imbalance ratio of 500, posing a significant challenge with extreme class imbalance and high intra-class variation.
C Pseudocode
Algorithm 1: LLM-AuttDA: Automatic Data Augmentation with Language Models
1: Input:
2: LLog--tail taigetailed datasets Dtrain: Deal
3: Pretrained language model L
4: Initial data augmentation policies K.
5: Long-tailed learning model fo with parameters 0
6: Output:
7: Optimal data augmentation algorithm A*
8: Final model fe
9: Define task description prompt Ptask
10: Define exploration operator prompt template Pg
Define mutation operator prompt template Pas
12: Initialize algorithm population
{A4
(0)
) = INITIALIZE(C, Peasks K.)
for each generation do
(A(t) NN,, SELECT (1A (1)1-1)) 1N
14:
)i=1) (Select parents}
(A(") N.
J==1 EXPLORE(C, PE, (A Np ) (Explore)
(1) No
15:
(A(")) MUTATE(C, PMs, (AA REl
(0) No KC) (Mutate)
17:
(A
"NN JN (1(t)N No U 1L4 Jim1 ( (1) Nm
(0) No
Jimi
for each !! do
18:
fitness EVALUATE(A So, Dirain, D Cekps Nst) 20: end for
end for
22:
A =A. fitness; (Select algorithm with highest fitness)
23: return Jo
=0
where INITIALIZE uses an initialization operator to generate random population of algorithms, EXPLORE and MUTATE correspond to the exploration and mutation operators respectively. In each generation, a subset of individuals is selected from the previous generation as parents, then new algorithms are generated using the exploration and mutation operators, and merged into the population. For each candidate algorithm, EVALUATE is called to compute its fitness:
15

---

function EVALUATE(A, (A, fo. Derain Doat, Cokps Nfc): fg FINETUNE(A, A. fo. Drrain; Eckp: f1)
return ACCURACY (So. Dvat)= =0
FINETUNE starts from the Cckp-th checkpoint and incrementally trains for Nee epochs on the training set, with the algorithm dynamically selecting augmentations for each class:
function FINETUNE(A, Sa. D Drrein: Cokp: st Load checkpoint of fe at epoch Cokp
for Eckp to Eckp + y: do
for each class e do
A
_ A(W! H
acce acc
(e-1) Ala-1) d(o-1) .e)
Ee
end for
Train fo for one epoch on Derain using )
Elee for augmentation
end for
return fo =0
where ( are the weights of augmentation techniques for class c, H! is the history of accuracies for c, and acce (e-1) is the accuracy in the previous epoch.
D Future Anylysis
D.1 Highly Imbalanced Scenarios
To evaluate the effectiveness of LLM-AutoDA in highly imbalanced scenarios, we adjusted the imbalance ratio 200 and conducted comparative experiments. The experimental results, as shown in Figure 9, demonstrate that LLM-AutoDA consistently outperforms other long-tailed data augmen- tation methods.
CIFAR-100-LT (IR=200)
44
Top-1 Accuracy (%)
42
40
38
CUDA
DODA
36
LLM-AutoDA
CE
CE-DRW LDAM-DRW BS
RIDE
BCL
Baselines
Figure 9: Accuracy (%) more imbalanced CIFAR-100-LLT dataset (Imbalance ratio=200) with
SOTA DA methods.
16

---

D.2 Visualization of Selection Process
Rotate
GaussianBlur
Invert
Equalize
AutoContrast
Smooth
Detail
EdgeEnhance
Mirror
Flip
20
40
60
80
100
120
140
160
180
200
epoch
Figure 10: Trends in the augmentation intensities and number of times different strategies selected.
In the figurel3, visualize the selection process of the data augmentation strategies provided by the model. We train the discovered data augmentation methods on CIFAR-I00 with an imbalance ratio of 100 under the bi-entropy loss, where the x-axis the epoch. The y-axis represents the data augmentation techniques that may be used in this work, including Mirror, EdgeEnhance, Detail, Smooth, AutoContrast, Equalize, Invert, GaussianBlur, Rotate, and Flip. The augmentation intensity ranges from () to 1 for all of these techniques. For samples of the first class, on the right side, different data augmentation methods are represented, with dot in the grid indicating that the method represented by that row is selected, and dots of different colors indicating multiple selections. The height in the grid represents the intensity given by our strategy. the early stages of training, the selected data augmentation strategies are relatively random, but as the model continues training, the trends in the selected data augmentation methods, intensities, and number of times selected gradually stabilize. This shows that the data augmentation methods provided by the large model can effectively achieve convergence on the long- tailed learning model. This demonstrates the feasibility of using a large model to search for and design data augmentation strategies for long-tailed learning models.
D.3
Cost Analysis
Using LLM-AutoDA to optimize data augmentation strategies for long-tailed recognition tasks requires significantly less time compared to the manual design of data augmentation strategies by humans. LLLM-AutDA can obtain highly effective strategies within mere 2 hours under a given framework.
17

---

E
Prompts
You are requested to design a novel algorithm that, given a set of augmentation techniques, selects several of them based on the change in per-class accuracy between the current training time point and the previous one, to be employed in the subsequent training phase, with the aim of enhancing the model's ability to tackle long- tail problems. This algorithm should deviate from existing methodologies present in the literature.
Describe the new algorithm and main steps in one sentence, place the sentence inside curly braces, Next, implement it in Python as a function named get aug_ aug_type. This function should accept 6 input(s): seigh_''tt', 'ACCs', "History HHsss, 'lats chose matix', 'lats chooseehsheeeeeeeeeeeeeeeeeeec _exees'', 'epoch'. The function should return 1 output(s):
'chose matrix,chose_cee exts'. aug_ weight is two-dimensional integer array initialized to I, used to record the historical weight information for each category (indexed by rows) across every augmentation technique (indexed by columns). ACCs is a one-dimensional integer array showcasing the performance of each category at the current training instant, specifically, the count of correct predictions within that category. History ACCs is a two-dimensional integer array that records the number of correct predictions made by each augmentation technique (column-wise) the last time they were employed for every category. lats_chose chose matrix dimensional Boolean array indicating whether specific augmentation techniques (by column index) were utilized for each category (row index) in the previous training step; True signifies usage, while False denotes non-usage. lats chose exts is two-dimensional float array representing the application intensity of each augmentation method across classifications at the current point in time, with range from 0 1, where higher numbers imply greater enhancement strength. epoch is an integer denoting the current training epoch, indicating the number of completed training cycles. chose matrix is two- dimensional Boolean array marking which augmentation techniques (by column index) will be employed in the next training step; True values indicate adoption, and False, rejection. chose exts is a two- dimensional float array signifying the intensity of applying each augmentation technique to individual categories in the upcoming time step, also ranging from 0 to 1, with larger values indicating more substantial augmentation efforts. All are Numpy arrays.
two-
chtse_eets
Do not give additional explanations.
Figure 11: An example of initialization prompts.
18

---

You are requested to design novel algorithm that, given a set of augmentation techniques, selects severul of them based on the change in per-class accuracy between the current training time point and the previous onc, to be employod the subsequent training phase, with the aim of eahancing model's ability tackle long-tail problems. This algorithm should deviate from existing methodologies present in the literature.
have existing algorithms with their codes as follows:
No.1 algoridhm and the corresponding code are:
The algorichm dynamically adjusts augmentation tochnique selection and intensity based on per-elass accuracy improvements, historical usage, and a decay factor address class imbulance and enhance model performance on underrepessented classes.
import numpy as np import random
def get aug typotaug Weight,ACCs, History_ ACCss lats chose matix, lats chose exts.opochh)
ols num,num aug, type History ACCs.shape
f solve weight as selfaug weight
for cidx range(cls, numn:
indices = lats chose matix(cidx]
assert indices.anye Polass index [cidx] has no chose aug (num of aug must 0) aug weightf cidx [[nddios]] = Ap..hhere(ACss(iddy History ACCsfeidx[[indies)) +
aug weightfcidx]jidiiee
aug_ weight[cidss [[indcess] 1)
aug weight - up. wpp.mxximummauggg. weight, 1)
chose augg np zeroscsss(css num ,umm aug astpoolltyyo
chose exts op rap..andoommoom hpp.random.mnd/11 man(""ltt llats chose shas.s
aug list = (I for i in rango(mum aug__tgggg type)]
for in rangotels num):
indexes rundom choiccsssoiiisiuugg..ooooo list, weights aug, weightfi, [Lolistt) k= MAX N for index in indexes:
chose mugf [index] - True
return chose jug. chose_eettt _eetss
Please help me create a new algorithm that has totally different form from the given ones.
Describe the new algorithm and main steps in one sentence, place the sentence inside curly braces. Next, implement it in Python as function named get nug type. This function should accept inputis): 'aug. weight', 'ACCs', 'History ACCS, 'lats chose matix', 'lals, chose exis', 'epoch'. The function should return oulpui(s): 'chosee malris, chose, exts'. aug weight two-dinensiooall integer arryy initialized 1. used to record the historical weight information for each category (indexed by rows) across every uugmentation technique (indexed by columns). ACC's is one- dimensional inleger array sbowcasing the performance of each category at current training instant, spocifically, the count correct predictions within that category. History ACCs is rwo-diimesiooall integer army that records the number of correct predictions made by each augmentation technique (column-- wise) the last time they were employed for every category. lots chose matrix two-dimensional Boolcan array indicating whether specific augmentation techniques (by column index) were utilized for each category (row index) in the previous training two-dimensional float array representing the
step; True signifies usage, while False denotes non-usage. lats chose application intensity of cach augmentation method across classifications at the current point time, with range from to I. where higher numbers imply greater calancement strength. opoch is an integar denoting the current training epoch, indicating the number of completed training cycles. chose matrix is two-dimensional Boolean ammy marking which augmentation techniques (by column index) will be employed in the next training step: True values indicate adoption, and False, rejection. chose exts is wwo-dimensional float armay signifying the intensity of applying each mugmentation technique to individual categories in the upcommng time step, also ranging from to 1, with larger values indicaling more subslantial augmentation efforts. All are Numpy arrays.
Do not give additional explanations.
Figure 12: An example of crossover prompts.
19

---

You are requested to design novel algorithm that, given a set of augmentation techniques, selects severul of them based on the change in per-class accuracy between the current training time point and the previous onc, to be employod the subscquent training phase, with the aim of ealancing model's ability tackle long-tail problems. This algorithm should deviate from existing methodologies present in the literature.
code follows Algorithm description: The algorithm dynamically adjusts augmentation technique selection and intensity bused on pen-class accuracy improvementss, historical usage, and decay factor to address class imbalance and cahance model performance on undoregresented classes.
have one algorithm
Code:
import numpy
import rundom
def' get_ aug hypotaug Weight,CCs History_ lats chose matix, lats chose exts.epoch):
els num,num aug type History ACCs.shape
solve weight as weight
for cids rangefels num):
indices lats chose, matix(cids]
assert indices any( f'class index (cidx) has no chose aug (aum of aug must oy mug. weightf cidx [[indioes] = np.ppphee where(ACCsfeiid > History CCCfeeixx[indiess
aug. weightfeids (ecnffcix]]dddiidd [indieess +
auy, weightfcidx ceaphficixxiniiicess [indices] 1)
aug weight - up www.mummmaauuu.....m weight, 1)
chose aug up spp.eeros((cls num num aug astye..astpe(bboo)) chose exts = rando llats chose exts shape)
aug list = (i for i in rango(mum, ang typo)]
for in range(cls num):
indoxes = random www.oooooooiccaaug.ooooooooo list, weights aug_ weightfi. Jaolist(), k- 1) MAX
index in indexes:
chose [index] True
return chose aug, chose exts
Please assist creating a new algorithm
different can modified version the algorithm provided.
First, describe the new algurithm and main steps in one senience, place the sentence inside curly braces. implement in Python as function named get puwg_uggggggg type. This function should accept input(s): 'aug. weight, 'ACCs' 'History _ACCss,, 'lats chose matix', 'lats chose exts', opech', The function should return ourpun(s):: 'chosc matrix,chose chose exts'. aug weight two-dimensional integer army initialized to 1, used to record the historical weight information for each category (indexed by nows) ncross every mugmentation technique (indexed by columns). ACCs is one-dimensional integer array showcasing the performance each calegory
current truining inslant, specifically, the count of correct predictions within that categury. History ACCss owo-dinensional integer armyy that records the number of correct predictions made by each augmentation tochnique (column- wise) the last time they were employed for every calegory. lats chose matrix is twe-dimensional Boolean array indicating whether spocific augmentation techniques (by column index) were utilized for each category (row index) the previous training step: True signifies usage, while False denotes non-usage. lats chose mpplication intonsity of cach augmentation method across classifieations at the current point in time, with range from to I, where higher numbers imply greater enhancement strength. epoch an integer denoting the current training epoch, indicating the number of completed training cycles, chose matrix is two-dimensional Boolean arryy marking which augmentation techniques (by column index) will be employed in the next training step; True values indicate adoption, and Falsc, rejection. chose exts is two-dimensional float array signifying the intensity of applying each augmeutation tochnique to individual categorics in the upcoming time step, also ranging from I, with larger values indicating more substantial augmentation efforts, All are Numpy armys..
twn-dimensional float array representing the
Do not give additional explamations.
Figure 13: An example of mutation prompts.
20

---

Limitations
This paper aims innovate the data augmentation paradigm in long-tailed learning, which greatly improves the degree of freedom of long-tailed data augmentation. However, there are still some limitations of our approach. For example, the scoring mechanism for augmentation strategies is not perfect, and a more comprehensive scoring mechanism is needed. In addition, how to break through search space limitation and generate novel augmentation methods is also problem to solved.
G Broader Impacts
Traditional data augmentation is not the best choice for long-tail learning, and recent long-tail data augmentation methods still lack degrees of freedom. The positive impact of our method is that it can give large number of augmentation strategies suitable for long-tail learning in short time, which greatly reduces the time and human cost consumed to design the strategy. Of course, there is also small negative impact, that is, is easy
dependence LLMss
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We describe the contributions of this paper in detail.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this paper in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We provide a complete demonstration process and data support.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details and code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: We provide the complete code as well as the details of the publicly available datasets used.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nig) cc/
public/ guides Codeeubomi ss casooooPccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: n ps cc/ /pppplll c/gut des CodeSubm's ss onPol ccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the relevant details and analysis of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported the average results of multiple tests in the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the code for easy reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt /ttt://///urppae/ee http:////bcc/ccccccccccu...oooo
Answer: [Yes]
Justification:
We are fully qualified.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the implications in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We give references to the datasets and code used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' saww thcode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "sLNI74Owpb", "reviewer": "Reviewer_JafW", "summary": "This work introduces gradient-free black-box optimization algorithms to formulate appropriate data augmentation methods, achieving some performance improvements. Utilizing LLM for evolutionary strategies is interesting, but the final augmentation strategy remains a black box. This makes it difficult to validate the authors' claims about targeting long-tail data and does not fully align with their stated motivation. I believe this work requires further improvement.", "questions": "See Weaknesses*", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. By leveraging large-scale pre-trained language models, LLM-AutoDA can automatically generate and optimize data augmentation strategies without manual design, reducing labor and time costs.\n\n2. The framework can dynamically adjust augmentation strategies based on performance feedback from the validation set, ensuring the strategy remains optimal throughout the training process.\n\n3. The introduction of gradient-free black-box optimization is a novel idea.", "weaknesses": "1. The authors' survey of the long-tail recognition field is insufficient. For example, in the introduction, they state that existing long-tail data augmentation methods either augment in feature space or directly use traditional methods. I have two counterpoints to this claim. First, existing long-tail data augmentation methods are not limited to the feature space, such as CMO (CVPR2022). Second, what are the drawbacks of augmentation in the feature space? What is the authors' intended motivation?\n\n2. The authors aim to develop targeted data augmentation strategies for each class, but LLM cannot evaluate the characteristics and deficiencies of image data. I am personally skeptical about formulating strategies through prompts alone, and the experimental results do not show distinct differences compared to other methods.\n\n3. This work introduces gradientless black-box optimization, and the authors attempt to use LLM as a generator of expansion strategies, using the validation set performance as an evaluation metric, and letting LLM undergo evolutionary evolution. However, the authors did not provide the augmentation strategies obtained through evolutionary processes, making it hard to prove the work's claimed focus on long-tail problems.\n\n4. The selected comparison methods are insufficient. The authors should conduct extensive comparisons with existing long-tail data augmentation methods, such as OFA (ECCV2020), GistNet, CMO (CVPR2022), H2T (AAAI 2024), and FUR (IJCV2024). Additionally, comparisons with vision foundation model-based methods, such as LPT (ICLR2023) and PEL (ICML2024), should be included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for these pertinent and in-depth comments, which are very helpful in improving our work and clarifying our contributions. Let me respond to your comments one by one:\n\n**W1:** Thank you for your advice. We have done our best to supplement the comparative experiments of our method and more data augmentation methods (including CMO) and present **the results in Table 1 in ONE-PAGE PDF**. We will update this section in the revised version to comprehensively review the different types of methods and their pros and cons.\n\nAs for the second point, I am sorry for causing your misunderstanding. We want to express that the current long-tailed learning methods based on data augmentation include data-level augmentation methods and feature-level augmentation methods, as highlighted in a recent long-tailed survey [1]. In addition, we strongly believe in augmenting the feature dimension, which is orthogonal to the traditional data dimension. **However, as formulated by CUDA [2] and DODA [3], previous approaches can come with potential sacrificial effects if a class-independent augmentation strategy is imposed on all classes. Therefore, based on this motivation, we further propose new augmentation paradigms.**\n\nIn general, we mention these works to make our survey more comprehensive, even though our work is not focused on feature dimension improvement. I hope you can understand our intention.\n\n[1] Deep long-tailed learning: A survey. IEEE TPAMI 2023.  \n[2] CUDA: Curriculum of Data Augmentation for Long-tailed Recognition. ICLR 2023.  \n[3] Kill Two Birds with One Stone: Rethinking Data Augmentation for Deep Long-tailed Learning. ICLR 2024.\n\n  \n\n**W2 and W3:** You bring up a good point. With textual hints alone, LLMs may struggle to accurately grasp the characteristics and limitations of image data. **Therefore, based on the same point of view as yours, our data augmentation discovery framework consists of an LLM-based evolution process and a long-tailed training process, where,**\n- The LLM-based evolution process will learn existing successful data augmentation methods and discover unseen data augmentation methods through mutation and crossover processes.\n- The long-tail learning model evaluates the true effect of these newly discovered data augmentations through the real long-tail distribution environment and dynamic data augmentation during training.\n\nThat is, **rather than letting LLMs devise class-specific data augmentation strategies based on text, we let LLMs learn from existing methods and invent new ones, and then evaluate the methods based on the ability of the long-tailed model to evaluate image data**. By interacting with the LLM and the long-tailed model in this way, we can achieve a **positive loop between evaluating the features of the image data and innovating the augmentation algorithm**. \n\nTo give you a better understanding, **we add an example in Figure 1 in ONE-PAGE PDF** to demonstrate the discovery process of the algorithm. In addition, we compare the performance improvement curves of the augmentation strategies in the three evolution stages, to illustrate that our framework can generate corresponding data augmentation strategies towards methods that are beneficial for long-tailed learning.\n\n  \n\n**W4:** Your suggestion is to the point. Thank you for your experience in the field to help us improve our work. Limited by the training time, we tried our best to reproduce part of the baselines in the limited time and have added the **corresponding comparison experiment in  Table 1 in ONE-PAGE PDF**. We'll add a more comprehensive comparison to cover more of the latest long-tailed data augmentation methods in the revised version.\n\nThank you again for your valuable advice. In the revised version, we will supplement the related work review and add the baseline as appropriate. At the same time, we are also happy to agree with you and apologize for the misunderstanding caused. We also look forward to further communication with you and listening to your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8oMxK2bC3", "reviewer": "Reviewer_37A8", "summary": "The paper proposes LLM-AutoDA to select the optimal augmentation strategies with the help of pre-trained LLMs. Specifically, the authors carefully designed various prompts to instruct an LLM to design new algorithms or mutate existing algorithms with the goal of improving the validation performance on long-tail classification tasks. Experiments show the effectiveness of LLM-AutoDA on three datasets under the long-tail setting.", "questions": "- How does the method perform compared with popular AutoDA works, such as AutoAugment and the simple RandAugment algorithm?\n\n- In previous AutoDA work, AutoAugment learns the probability and magnitude of applying multiple augmentations to a target dataset; AdaTransform[3] and AdaAug[2] can learn class-specific and instance-specific augmentation strategies. What are the benefits of using LLM to select the augmentation parameters? \n\n- The paper listed some example prompts in the appendix to ask the LLM to design a new algorithm or mutate an existing algorithm. Can the authors provide some examples of the new algorithms output from the LLM? How are they different from the manually designed method?", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Applying LLM in data augmentation for long-tail datasets is somewhat novel.\n\n- Real-life datasets often follow a long-tail distribution. The method is solving an important problem.\n\n- The method shows slight improvements in three datasets under the long-tail setting.", "weaknesses": "- The formulation of providing class-specific validation accuracy and carefully designed text prompts to deploy LLM in data augmentation is somewhat novel. However, other parts of the method have been studied before. The LLM is an off-the-shelf method. The idea of searching for optimal augmentation configuration is proposed in AutoAugment, and the approach of using the population-based method to select a good augmentation strategy is studied in PBA[1]. Class-specific augmentation strategies are studied in AdaAug[2] and AdaTransform[3]. \n \n- The proposed method is very computationally expensive. It uses a large pre-trained model to process the augmentations. It trains a population of models instead of a single model. If we look at the best-performing BCL + LLM-AutoDA baseline in Table 1, LLM-AutoDA slightly leads DODA (+1.2%) with IR=50 and performs the same with DODA with IR=100 in terms of the “all” test accuracy.  What is the computation cost of DODA when compared to LLM-AutoDA? It raises the concern whether the huge computation power in LLM-AutoDA is justified.\n\n- The clarity of the paper needs to be improved. I found that the notations introduced are not explained clearly. For example, I do not understand the meaning of “the weights of each augmentation method”, $W$; what is the definition of the selection matrix $A$? Is it a zero-one matrix in {0,1}$^{C \\times M}$, where $C$ is the number of classes, and $M$ is the number of augmentations? Sometimes, the same symbol is used to represent different concepts. For example, $A$ is the selection matrix and the population algorithm. Are the selection matrices the same as the population algorithms? It seems that the set notation of $A$ in lines 224-231 misses the parentheses. The unclear definition and explanation of the notations make it hard to understand the algorithm precisely.\n\n(minor) There are quite some typos in the manuscript, for example\n- Inconsistent use of “fine-tune” and “finetune” in line 204\n- Subscript in $f_{aug}$ in line 206\n- “Blod” should be “bold” in Table 1 and Table 2 captions\n\n__References__\n\n[1] Ho, Daniel, et al. \"Population based augmentation: Efficient learning of augmentation policy schedules.\" International conference on machine learning. PMLR, 2019.\n\n[2] Cheung, Tsz-Him, and Dit-Yan Yeung. \"Adaaug: Learning class-and instance-adaptive data augmentation policies.\" International Conference on Learning Representations. 2021.\n\n[3] Tang, Zhiqiang, et al. \"Adatransform: Adaptive data transformation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed comments and questions. These feedbacks are invaluable for improving our work. Let me respond to your questions one by one:\n\nFirst, from your review comments, I noticed that our work may have caused some misunderstandings. For example, you thought our method \"uses LLM to select augmentation parameters\" and \"trains a set of models\". Your summary of our method as \"carefully designing various prompts\" \"to improve validation performance on long-tail classification tasks\" is also not as accurate as the summaries by Reviewer JafW, q3qy, and 2Q2t. Therefore, please allow me to reintroduce our research.\n\n- Our research is not a data augmentation method that uses LLMs and prompt engineering to improve model performance, but a new process for discovering long-tailed data augmentation strategies. In the discovery process, the LLM continuously innovates and modifies existing data augmentation methods, while the long-tailed model evaluates these innovative methods through its image learning ability in long-tailed learning, guiding the evolution direction of the methods. **Reviewers JafW, q3qy, and 2Q2t all emphasized the importance of this process, so we supplemented some of the obtained strategies and performance improvements during the process in Figure 1 in ONE-PAGE PDF.**\n\n- Although this discovery process uses LLMs and long-tailed model training, it is **much less time-consuming and labor-intensive** than manually designing new data augmentation methods.\n\n- In this discovery process, we obtained brand-new data augmentation strategies that are universally effective on different long-tailed baselines and datasets, and have no higher cost in real-world usage, comparable to existing data augmentation strategies.\n\nNext, I will reply to your questions one by one:\n\n**W1:** Thank you for your opinion. First, LLM is indeed existing, but the current trend is to utilize or improve LLM in more practical domains to promote domain development. Therefore, **as the other three reviewers mentioned, \"this integration of LLM and long-tail learning is both creative and timely\".** Moreover, selecting better augmentation strategies itself is an ongoing research area, and our essential goal is to promote progress in this field.\n\nTherefore, our method is innovative in the following aspects: \n- We introduced LLM into the field of long-tailed data augmentation, leveraging its powerful language understanding and generation capabilities, combined with a dynamic data augmentation framework, successfully discovering data augmentation strategies suitable for long-tailed learning that surpass human-designed ones; \n- We designed a complete LLM-driven data augmentation pipeline specifically for long-tailed problems, systematically studying how to utilize LLM to mitigate the challenges brought by long-tailed distributions. This is obviously different from improving model performance through prompts.\n\n**W2:** My previous reply has addressed this question. In fact, **we only run the model multiple times during the discovery process of new strategies.** Once a data augmentation strategy is discovered (which takes about 6 hours), we can directly use it in testing and generalize it to datasets and long-tailed baselines unseen during the discovery process (as shown in the experimental results in this paper). In this process, the complexity of the discovered data augmentation strategies is consistent with manually designed strategies.\n\n**W3:** Thank you very much for your valuable comments. We will improve and unify the definition and usage of symbols in the revised paper to enhance its readability and rigor. In our algorithm, each data augmentation method is assigned a weight $w \\in [0, 1]$, representing the probability of selecting that method. We will clarify the meaning of weight $w$ in the symbol definition section to eliminate ambiguity. The selection matrix $A \\in \\{0, 1\\}^{C \\times M}$ is a binary matrix, where $C$ represents the number of categories and $M$ represents the number of augmentation methods. The matrix element $a_{ij} = 1$ indicates that the $j$-th augmentation method is selected for category $i$; $a_{ij} = 0$ means not selected.\n\n**Q1:** We provided some updated performance comparisons of automatic data augmentation. In the appendix, we continue to add some comparisons with the latest baselines to demonstrate our performance. In the revised version, we will increase the comparative experiments with AutoAugment and RandAugment to comprehensively evaluate the performance of our method.\n\n**Q2:** As answered at the beginning, we do not use LLM to do the same thing as auto, but focus on discovering new data augmentation strategies, not just searching for parameters.\n\n**Q3:** In Figure 1 in ONE-PAGE PDF, we list some novel augmentation algorithms designed by LLM and show the changes in performance improvement. In Tabel 1, we will enhance more relevant analysis and examples.\n\nThank you again for your detailed review comments. We will conduct a comprehensive review and correction of the symbol definitions and usage in the paper accordingly. If you have any other suggestions, we will humbly listen and seriously improve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "woB1RIejIg", "reviewer": "Reviewer_q3qy", "summary": "This paper proposes to leverage large language models (LLMs) to help automatically facilitate data augmentation for long-tailed learning. It first discusses the limitations of traditional re-balancing or data augmentation methods. Then it proposes a novel LLM-based augmentation framework LLM-AutoDA. LLM-AutoDA automatically searches for the optimal augmentation strategies and re-balance the long-tailed data. The experimental results demonstrate the effectiveness of LLM-AutoDA on multiple long-tailed datasets.", "questions": "See weaknesses above.", "limitations": "The limitations have been discussed in Appendix Section F.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work provides pioneering research on leveraging LLMs for data augmentation in long-tail learning. To the best of my knowledge, it is the first work that combines long-tail learning and LLMs, and I believe it will have certain contributions.\n2. The discussion of the weaknesses of previous methods makes sense. Previous methods adopt data augmentation within a limited knowledge space and might struggle to solve the long-tail problem. The proposed method breaks this limitation.\n3. The proposed method is general and scalable. It can be further enhanced with the update of large language models.\n4. The empirical studies and the ablation studies are well conducted.\n5. The paper is well-written and easy to follow.", "weaknesses": "1. It is better to provide some example augmentation strategies that the LLM generates and briefly analyze why such generated strategies can help learning.\n2. What does 1/|gap| in the x axis of Figure 5 mean? How will it impact the performance of different LLMs?\n3. Some related works regarding re-sampling and data augmentation are missing [1].\n4. It is better to unify the figure and text size in the appendix, such as Figure 9 (too large), Figures 11-13 (different sizes)\n\n[1] How re-sampling helps for long-tail learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. These suggestions are very helpful for improving the quality of the paper. I will respond to your questions one by one:\n\n**W1:** Your suggestion is very pertinent. In the revised version, we will supplement some examples of augmentation strategies generated by LLM and briefly analyze them.\n\nFor example, we can showcase a specific augmentation combination generated by LLM for a long-tailed class and discuss how this combination can effectively improve the learning effect of that class. Through example analysis, readers can have a more intuitive understanding of the working principle and advantages of LLM-generated strategies.\n\n**W2:** I apologize for the lack of clarity in the description of Figure 5. Here, |gap| refers to the performance gap between the augmentation strategy generated by LLM and the optimal strategy. The larger 1/|gap| is, the closer the LLM-generated strategy is to the optimal, and the better the performance.\n\nDifferent LLMs may have varying performances when dealing with long-tailed problems. Figure 5 aims to compare the performance curves of different LLMs and reveal their ability to narrow the gap with the optimal strategy. We will add clearer legend explanations in the revised version to facilitate readers' understanding.\n\n**W3:** Thank you for providing the references. We indeed overlooked this. In the revised version, we will supplement the discussion of some representative methods in the field of resampling and data augmentation in the related work section, especially the paper you mentioned, to make our related work more comprehensive.\n\n**W4:** You are right. We will carefully check and adjust the format in the appendix, especially Figures 9 and 11-13, to make their sizes and layouts more consistent.\n\nThank you again for your feedback. These comments are enlightening for us. We will carefully revise the paper accordingly to present higher-quality work. If you have any other suggestions, we will humbly accept them and actively improve.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kP41BL6zS", "reviewer": "Reviewer_2Q2t", "summary": "The paper presents a novel approach by ingeniously leveraging large language models (LLMs) to generate data augmentation strategies for long-tailed problems. Given the recent rapid advancements in LLMs and their potential applications beyond natural language processing, this integration of LLMs with long-tailed learning is both creative and timely.", "questions": "It is interesting that the authors upgrade the data augmentation paradigm of long-tailed learning, I wonder if this paradigm can also be combined with other domains with interesting impact. If the author can give relevant analysis, it will help to improve the audience of the paper.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) The proposed LLM-AutoDA framework generates augmentation strategies and incorporates a feedback loop mechanism to optimize these strategies iteratively. The idea is novel and interesting.\n\n2) The paper offers in-depth analysis and insights into the discovered augmentation strategies, which not only aids in understanding the model's working principles but also provides valuable guidance for future research in long-tailed learning.  From a practical perspective, this method significantly reduces the cost of manually designing augmentation strategies, which has important practical value for researchers and practitioners dealing with imbalanced datasets.\n\n3) The paper presents extensive experiments on multiple mainstream long-tailed learning benchmarks, convincingly demonstrating the method's effectiveness across various datasets and imbalance ratios.", "weaknesses": "1) The paper proposes multiple prompts to collaborate to update the data augmentation strategy, please explain what impact they will have when multiple prompts collaborate.\n\n2) Can you try to evaluate and compare the performance of SimpleLLM and LLM-AutoDA under different long-tailed settings such as different imbalance ratios? I think this better reflects the ability of the searched strategy to adapt to different long-tailed scenarios.\n\n3) This paper has some syntax errors, please improve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments and questions. Let me respond to them one by one:\n\n**W1:** In our framework, different prompts play complementary roles. For example, some prompts are responsible for generating new augmentation algorithms, while others mutate existing algorithms. Through this collaboration, we can strike a balance between exploring new augmentation spaces and leveraging known effective algorithms.\n\nThe collaboration of multiple prompts helps the search strategy more comprehensively cover the design space of data augmentation. Different prompts can optimize the augmentation strategy from different perspectives, jointly promoting performance improvement. We will supplement more analysis in the paper to explain the benefits of this collaborative mechanism.\n\n**W2:** You made a great suggestion. Comparing the performance of the two methods under different imbalance ratios can more comprehensively evaluate the ability of the searched strategies to adapt to different long-tailed scenarios.\n\nIn the revised version, we will add more experiments with different imbalance ratios to systematically analyze the strengths and weaknesses of SimpleLLM and LLM-AutoDA in dealing with different long-tail distributions. This will provide readers with richer experimental results and demonstrate the robustness of our method.\n\n**W3:** Thank you for the reminder. We will carefully check the entire text again, correct the grammatical errors found, and improve the expression quality of the paper.\n\n**Q1:** You raised an interesting direction for reflection. We believe that the paradigm of using LLM to assist data augmentation has the potential to be extended to other tasks and bring inspiration to more fields.\n\nFor example, in natural language processing, LLM can generate targeted augmentation strategies based on the characteristics of text data, such as synonym replacement and back-translation, to alleviate data sparsity problems. In the field of speech recognition, LLM may also help optimize audio augmentation and improve the generalization of models.\n\nIn the revised version, we will add some extensibility analysis in the discussion section to explore the possibilities and potential impacts of combining this paradigm with other fields.\n\nThank you again for your feedback, which is very helpful for improving our work. We will carefully revise the paper, hoping that the revised version can answer your doubts and meet the publication standards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We sincerely appreciate your valuable comments and suggestions. We are encouraged by the positive feedback highlighting the novelty, significance, and potential impact of our work in the field of long-tailed learning.

We are delighted to receive many favorable assessments. Reviewer q3qy recognizes our work as pioneering research on leveraging LLMs for data augmentation in long-tailed learning, while Reviewer 2Q2t acknowledges the novelty and practicality of our framework. Reviewer JafW finds our approach of utilizing LLMs for evolutionary strategies to be interesting, noting that it reduces labor and time costs. At the same time, we will actively clarify some misunderstandings that our paper may have caused. We have provided a detailed introduction to address Reviewer 37A8's concerns, aiming to better explain the logic behind our work.

Once again, we sincerely thank all reviewers for their constructive feedback. We believe that addressing the concerns raised will significantly enhance the quality and impact of our work. We look forward to further improving our submission based on your valuable input.

Meta-review: This paper presents a novel framework, LLM-AutoDA, which leverages large language models (LLMs) for long-tailed data augmentation. The framework fine-tunes a long-tailed learning model using augmented training data, where performance improvements on the validation set serve as a reward signal to iteratively refine the data augmentation strategies. The experimental results demonstrate that LLM-AutoDA significantly outperforms state-of-the-art data augmentation and re-balancing methods.

The paper received four positive scores following the rebuttal period (5, 5, 7, 8), indicating recognition of its novelty, particularly in addressing long-tailed problems. However, there are several important issues that the authors should carefully consider, even if the paper is accepted:

1. Fair Comparisons with Baselines: While using LLMs in machine learning is novel, LLMs inherently contain external knowledge. It is crucial to ensure fair comparisons with other methods. The authors should carefully design additional baselines and thoughtfully select benchmarks to evaluate performance. An in-depth analysis of which components contribute most to addressing long-tailed problems is also needed.

2. Transparency and Interpretability: Models used in the machine learning community should prioritize transparency. In the context of image recognition tasks, deep neural networks are already opaque, and the inclusion of black-box optimization, such as LLMs, exacerbates this issue. The authors should discuss the implications of this lack of transparency in detail and explore how insights into long-tail problems can still be gained.
Decision: Accept (poster)