LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems

Authors: pengkun  wang
zhe  zhao
haibin  wen
fanfu  wang
binwu  wang
qingfu  zhang
yang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:11:44

Abstract: The long-tailed distribution is the underlying nature of real-world data, and it presents unprecedented challenges for training deep learning models. Existing long-tailed learning paradigms based on re-balancing or data augmentation have partially alleviated the long-tailed problem. However, they still have limitations, such as relying on manually designed augmentation strategies, having a limited search space, and using fixed augmentation strategies. To address these limitations, this paper proposes a novel LLM-based long-tailed data augmentation framework called LLM-AutoDA, which leverages large-scale pretrained models to automatically search for the optimal augmentation strategies suitable for long-tailed data distributions. In addition, it applies this strategy to the original imbalanced data to create an augmented dataset and fine-tune the underlying long-tailed learning model. The performance improvement on the validation set serves as a reward signal to update the generation model, enabling the generation of more effective augmentation strategies in the next iteration. We conducted extensive experiments on multiple mainstream long-tailed learning benchmarks. The results show that LLM-AutoDA outperforms state-of-the-art data augmentation methods and other re-balancing methods significantly.

Main paper: 
 introduction: 
 As a revolutionary technology, deep learning has shown a broad and significant impact on various tasks, including image classification [21] , object detection [34] , natural language processing [19] , and many interdisciplinary research problems [40, 47] . The success of these endeavors relies heavily on the support of large-scale manually curated datasets, e.g., ImageNet [35] . However, for the convenience of training and evaluating models, most artificially constructed datasets typically follow the assumption of uniform distribution, which contradicts the real-world data distribution, i.e., longtailed distribution. This deviation between the ideal and real distributions has resulted in many deep models trained on balanced datasets failing to produce satisfactory results in real-world applications, e.g., they only perform well on a few classes and ignore many vulnerable classes [4, 36] .To address the ubiquitous long-tailed problem, researchers have been continuously proposing various carefully designed research paradigms. One popular approach is to rebalance the training data by oversampling the tail classes or undersampling the head classes [6, 13] . However, this approach cannot fundamentally address the problem of insufficient global information, even though the data for the tail classes increases significantly. Another line of work focuses on designing specialized loss functions or reweighting strategies to alleviate the impact of class imbalance [11, 5, 37, 39] . However, these methods either introduce additional computational overhead or require careful manual design.Furthermore, inspired by AutoML [17] , particularly automated data augmentation [10] , we propose LLM-AutoDA, a novel LLM-based long-tailed data augmentation framework. LLM-AutoDA leverages large-scale pretrained models to automatically search for the optimal augmentation strategies suitable for long-tailed data distributions. Specifically, we first define a broad search space that includes augmentation operations and their parameters. Then, we train an augmentation strategy generation model that generates augmentation strategies based on the class-wise statistics of the longtailed data. This strategy is applied to the original imbalanced data to create an augmented dataset, which is used to fine-tune the underlying long-tailed learning model. Importantly, the performance improvement on the validation set serves as a reward signal to update the generation model, enabling the generation of more effective augmentation strategies in the next iteration. This process is repeated until the performance converges or the computational budget is exhausted.Compared to previous long-tailed data augmentation methods, LLM-AutoDA offers several advantages: (i) it leverages LLMs to automatically learn augmentation strategies tailored to the characteristics of long-tailed data, without relying on human expertise. (ii) it has a more extensive search space, allowing it to discover more novel strategies. (iii) it can dynamically adjust the augmentation strategies based on performance feedback during the training process, providing flexibility and robustness. Extensive experiments on multiple mainstream long-tailed learning benchmarks demonstrate that LLM-AutoDA outperforms state-of-the-art data augmentation methods and other rebalancing techniques significantly.The main contributions of this work are summarized as follows:• New augmentation paradigm: We combine LLMs with long-tail data augmentation for the first time, providing a novel perspective for efficient long-tail learning. • New automated framework: We propose a novel AutoML framework called LLM-AutoDA, which automates the search for effective data augmentation strategies for long-tailed learning, significantly reducing the cost of manually designing augmentation strategies. • Compelling empirical results: We conduct extensive experiments on multiple mainstream longtailde benchmarks, demonstrating the superiority of LLM-AutoDA compared to state-of-the-art methods. • In-depth analysis and insights: We provide detailed analysis and insights into the discovered augmentation strategies, guiding future research in long-tailed learning.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We describe the contributions of this paper in detail. 
long-tailed learning (ltl): 
 Various approaches have been proposed to address the long-tailed learning problem, including rebalancing the training data through over-sampling the tail classes [6, 13] or under-sampling the head classes [12, 14] , modifying loss functions or adjusting class weights during training [22, 11] , and decoupling representation and classifier learning [18] . Among these methods, data augmentation has emerged as a promising solution for long-tailed learning [8, 9, 20, 25, 46] . The key idea is to generate additional samples for the tail classes to alleviate the data imbalance issue. Recent research has attempted to design sophisticated strategies by observing performance changes during the training process to adjust augmentation operator [39] or intensity [2] . However, these methods still rely on hand-crafted augmentation strategies that may not be optimal for the specific long-tailed data and have limited search space for discovering novel and effective strategies. In contrast, LLM-AutoDA automatically learns data augmentation strategies tailored to the long-tailed data distribution without manual design.Large language models (e.g., BERT [19] , GPT [29, 30, 3] , and T5 [31] ) have achieved remarkable success in various natural language processing tasks. They exhibit strong generalization abilities and can be easily fine-tuned for downstream tasks with limited labeled data [16, 28] . Researchers have explored the potential of LLMs in automating algorithm design and implementation, such as generating source code [7] , optimizing hyperparameters [42] , and designing neural architectures [32] . However, the interaction between large and small language models and its impact on improving the design of small models have been less explored, particularly in the context of data augmentation for long-tailed learning. LLM-AutoDA aims to bridge this gap by harnessing the knowledge and generative capabilities of large language models to discover effective data augmentation strategies tailored to long-tailed distributions automatically. By defining a rich search space of augmentation operations and training an augmentation strategy model conditioned on the class-wise statistics, LLM-AutoDA can generate adaptive and optimized augmentations specifically designed for the given long-tailed data. This novel approach opens up new possibilities for leveraging the interaction between large and small language models to improve the design of machine learning algorithms in various imbalanced learning scenarios.3 LLM × LTL: Can LLMs Provide DA Strategies for Long-tailed Learning?In this section, we attempt to analyze whether LLMs can be applied to long-tailed learning and how to implement this learning paradigm.SimpleLLM. As shown in Figure 1 (b), strategy fixed DA is the current mainstream paradigm for long-tailed data augmentation. It utilizes carefully designed augmentation strategies to dynamically adjust augmentation operators or intensities during the training process, allowing different classes to choose advantageous augmentation methods. In this paradigm, the key step is to design a high-quality augmentation strategy. When dealing with balanced data distributions, we often employ class-independent augmentation strategies, which apply the same data augmentation to all classes. However, as mentioned in DODA [39] , when dealing with imbalanced data distributions, this class-independent augmentation strategy can potentially sacrifice certain classes, thus requiring the design of class-dependent augmentation strategies. However, the manual design process for such strategies is highly complex and costly.You are requested to design a novel algorithm that, given a set of augmentation techniques ... ... Recent research has shown that LLMs can replace many manually engineered tasks [7] .Inspired by this, we first designed a simple yet efficient paradigm for generating augmentation strategies called SimpleLLM. As shown in Figure 2 , we initially constructed a data augmentation-themed prompt from the perspective of prompt engineering, including task description, algorithm input, algorithm output, parameter interpretation, etc. We then input this prompt into pre-trained LLMs to generate a functional function that conforms to the prompt, i.e., an algorithm implementation containing augmentation strategies. Finally, this augmentation strategy is applied to the conventional training process of long-tailed learning. It is worth noting that this paradigm allows us to generate multiple augmentation strategies suitable for long-tailed learning at a low cost.Comparative Analysis. To further validate the effectiveness of the augmentation strategies generated by this paradigm, we conducted experiments on CIFAR-100-LT (IR=100) dataset. We selected several mainstream long-tailed learning baselines and integrated SimpleLLM with them. In addition, we compared the latest state-of-the-art long-tailed DA methods, CUDA and DODA, under the same settings. The experimental results, as shown in Figure 4 , reveal that SimpleLLM achieves acceptable average performance, comparable to CUDA and DODA, indicating that LLMs can be used as generators of augmentation strategies to enhance the performance of long-tailed learning.Under this paradigm, we believe that with appropriate prompts, augmentation strategies similar to CUDA and DODA can also be generated by LLMs. In other words, within a search space, we can obtain multiple similar locally optimal strategies. LLM-based Data Augmentation Strategy Generation. As shown in Figure 3 (left, pink), to design the DA strategies automatically, LLM-AutoDA incorporates a pre-trained LLM L as a search operator. Using prompt engineering techniques, a series of prompt templates are designed to incorporate prior knowledge about data augmentation into the generation process of LLM. LLM generates diverse data augmentation strategies based on these prompts, including both natural language descriptions and Python code implementations. Furthermore, the generated strategies are stored in a strategy pool and interact with the long-tail learning model to search for the optimal data augmentation strategy. The key aspect of LLM-AutoDA lies in the synergy between the DA strategies generated by LLMs and the long-tailed learning model. This close interaction allows the discovered DA strategies to dynamically align with the model training process, effectively enhancing the performance of long-tailed learning in a targeted manner.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of this paper in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Answer: [Yes] Justification: We provide the complete code as well as the details of the publicly available datasets used.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
strategy evaluation: 
 To evaluate the performance of candidate data augmentation strategies, we insert them into the model training process, conduct a small amount of additional training on the training set, and then test the accuracy on the validation set, using the accuracy as the fitness score for the algorithm. Assuming the data augmentation function generated by LLM is denoted as f aug and the training-testing function is denoted as T , the evaluation process can be represented as follows:EQUATIONwhere e ckp is the starting checkpoint epoch number and N f t is the epoch number of fine-tune. The function T injects f aug into the training flow:EQUATIONThe F inetune function starts training from the e ckp checkpoint and performs N f t epochs of training on the training set D train . At the beginning of each epoch, it dynamically selects data augmentation methods for each class using f aug.EQUATIONHere, Ac represents the DA selection matrix for class c at time t, Ec represents the corresponding augmentation intensity, W After training, the model is evaluated using the no augmented validation set D val , and the overall accuracy Acc val is obtained as the fitness score for f aug .EQUATIONA higher fitness score indicates better performance of the algorithm on long-tailed distributions. By injecting candidate algorithms into the real training process and evaluating them on the validation set, we can accurately and efficiently measure their actual ability to address the long-tailed problem.LLM-AutoDA leverages Pretrained Language Models (PLMs) to automatically generate data augmentation algorithms. To guide the PLM in generating algorithms that meet specific requirements, we employ prompt engineering techniques and carefully design a series of prompts. By incorporating task descriptions, input-output formats, novelty requirements, and other prior knowledge through prompts, we can constrain the generation process of the PLM within the desired search space. We design the following three types of search operators, corresponding to different prompt templates:• Initialization operator I: Based on the task description prompt P task and the knowledge base of data augmentation K, a set of randomly initialized population algorithms A (0) i N i=1 is generated. • Crossover operator E: Building upon P task , N p parent algorithms A (t) i i = 1 Np from the current population are used as references, along with the incorporation of knowledge base K. The PLM is required to generate N e new algorithms A (t) j j = 1 Ne that are different in both form and logic from the existing algorithms, thereby expanding the search space.• Mutation operator M : Based on P task , N m individuals A (t) i Nm i=1 are selected from the current population, and local improvement directions are provided. The PLM is tasked with generating a mutated algorithmÂ(t) i for each A (t)i within its neighborhood for further exploration. Taking the crossover operator E as an example, its prompt P E can be represented as follows:P E (P task , {A (t) i } Np i=1 , K, D f unc ) = P task +P ref ({A (t) i } Np i=1 )+P know (K)+P dif f +P f ormat (D f unc )(5) where P task represents the task description, A i i = 1 N represents N parent algorithms, and Df unc represents the domain of the objective function. P ref formats the parent algorithms into reference code, P dif f requires the generation of new algorithms that are completely different from the existing ones, and P f ormat specifies the input and output of the objective function.Once we have obtained the prompt P E , we input it into the pretrained language model L, and as a result, we obtain N e new crossover algorithms.EQUATIONEachA (t)j typically consists of a natural language description of the algorithm and its corresponding Python code implementation. Similarly, the prompts P I and P M for the initialization operator I and the mutation operator M can be constructed in a similar manner, with the main difference lying in the introduced prior information. 
experimental settings: 
 Datasets and Metrics. Like most long-tailed learning methods, we conducted experiments on several mainstream long-tailed learning datasets, including CIFAR-100-LT [5] , ImageNet-LT [26] , and iNaturalist 2018 [38] . Among them, CIFAR-100-LT is the long-tailed version of CIFAR-100, with various imbalance ratios. To validate the effectiveness of LLM-AutoDA in addressing the long-tailed problem, we selected three testing environments: 50, 100, 200. Compared to CIFAR-100-LT, both ImageNet-LT and iNaturalist 2018 have more classes and larger data sizes. It is worth noting that, similar to CIFAR-100-LT, ImageNet-LT is a long-tailed version artificially constructed from the well-known ImageNet [35] dataset. On the other hand, iNaturalist 2018 is a naturally occurring long-tailed dataset collected from the real world. We used the official complete versions of these datasets, and detailed information about the datasets is provided in Appendix B. We use Top-1 accuracy as the evaluation metric and provide the performance of subsets based on the class divisions provided by the official datasets. Following the settings of CUDA [2] , we considered various research theories when selecting the baselines. In addition to the classic cross-entropy loss (CE) [15] , we also validated different data augmentation methods on other baselines, such as loss-based re-balancing methods: CE-DRW [5] , LDAM-DRW [5] , Balanced Softmax (BS) [33] , and model-based re-balancing methods: RIDE [43] , BCL [48] . In terms of data augmentation methods, we compared LLM-AutoDA with the latest SOTA methods: CUDA [2] and DODA [39] . We observed their advantages and disadvantages by combining these DA methods with the long-tailed baselines. The relevant descriptions of the baselines are also provided in Appendix A.Implementation Details. All our models are implemented based on PyTorch [27] . We trained and evaluated the models on 2 NVIDIA Tesla A100 GPUs and reported the experimental results. We utilized the powerful gpt-3.5-turbo for strategy generation and employed AEL [23] for strategy optimization. In the experimental process, we first trained the models for 50 epochs without using augmentation strategies, then continued training with augmentation strategies for an additional 20 epochs, employing a novel evaluation mechanism. Additionally, during the final evaluation stage of long-tailed learning, we adopted the same settings as DODA [39] for all baseline methods and our approach.Results on CIFAR-100-LT. We first evaluated LLM-AutoDA and other long-tailed data augmentation (DA) methods on CIFAR-100-LT dataset (IR = 50, 100). The experimental results are shown in Table 1 . From the results, it can be observed that both SimpleLLM and the improved version LLM-AutoDA significantly improve the global accuracy of the model compared to the original long-tailed learning baseline, achieving robust improvements. In the horizontal comparison with long-tailed DA methods CUDA [2] and DODA [39] , as analyzed earlier, SimpleLLM achieves comparable performance to the former through a non-optimized way. This indicates that within our framework, a locally optimal strategy can replace a carefully designed complex strategy. In addition, LLM-AutoDA brings more significant and stable gains, reflecting that existing long-tailed DA methods may be suboptimal strategies within our augmentation strategy space, while LLM-AutoDA can provide the optimal augmentation strategy through continuous optimization. To evaluate the effectiveness of LLM-AutoDA in highly imbalanced scenarios, we adjusted the imbalance ratio to 200 and conducted comparative experiments in Appendix D.Results on ImageNet-LT and iNaturalist 2018. We also conducted comparative experiments on large-scale benchmark datasets, ImageNet-LT and iNaturalist 2018. As expected, different long-tailed learning methods showed significant performance improvement when integrated with LLM-AutoDA. Similar to the highly imbalanced setting mentioned earlier, both of these large-scale datasets are inherently highly imbalanced. Therefore, the augmentation strategies provided by LLM-AutoDA can consistently demonstrate superiority in various imbalanced environments. Importantly, LLM-AutoDA does not rely on meticulous manual design, which reduces the optimization cost on large-scale datasets. Do different LLMs produce differentiated effects? In the aforementioned experiments, we used GPT-3.5 [3] as the LLM to respond to the designed prompts. To analyze whether LLM-AutoDA is dependent on specific LLM models (e.g., GPT-3.5), we replaced the LLM model in LLM-AutoDA with other popular methods such as GPT-4 [1] and Claude-3-Opus, and conducted experiments.The experimental results, shown in Figure 5 , demonstrate that all three LLMs exhibit consistent performance trends when selecting augmentation strategies in different score ranges. For instance, they all show high performance near the augmentation strategies with scores around 12, while augmentation strategies with excessively high scores lead to performance degradation across the three LLMs due to insufficient diversity. Do different population numbers have an impact on performance? In LLM-AutoDA, we employed two different types of prompts: crossover prompts and mutation prompts. Crossover prompts involve transforming multiple parent populations into a single population, while mutation prompts replace the current augmentation strategy with an equivalent one.2 S H U D W R U , Q W H Q V L W \ /RVVThroughout the iterative process of framework evolution, when performing crossover and mutation operations, we need to specify the number of populations generated each time. Figures 6 and 7 illustrate the scores of strategies generated by two different mutation prompts (m1, m2) and two different crossover prompts (e1, e2), respectively. It can be observed that different prompts exhibit consistent trends in score variations. Particularly, compared to e1, e2 demonstrates a higher score variance, which indirectly reflects the bias in its prompt content.Why are fixed strategy methods often local optima? Strategy fixed data augmentation methods, such as CUDA and DODA, aim to adapt to long-tailed distributions by dynamically adjusting the augmentation operators or intensities. However, their focus is limited. We visualized the loss variations when searching for optimal augmentation strategies using different paradigms. From Figure 8 , it can be observed that CUDA (i.e., green plane) and DODA (i.e., red plane) can only search for local optimal strategies on a single plane, while LLM-AutoDA is capable of flexibly searching for the points with the lowest loss across the entire curved surface to obtain a global optimal solution. 
conclusion: 
 Existing paradigms for long-tailed learning have partially alleviated the long-tailed problem but still have limitations. To address this, this paper presents an LLM-driven long-tailed data augmentation framework called LLM-AutoDA, which utilizes large-scale pre-trained language models to automatically search for data augmentation strategies optimized for long-tailed data distributions. Experiments on multiple mainstream benchmark datasets demonstrate that LLM-AutoDA outperforms state-of-the-art methods. {A (t) i } Np i=1 = SELECT({A (t-1) i } N i=1 ) {Select parents} 15: {A (t) j } Ne j=1 = EXPLORE(L, P E , {A (t) i } Np i=1 , K) {Explore} 16: { Â(t) i } Nm i=1 = MUTATE(L, P M , {A (t) i } Nm i=1 , K) {Mutate} 17: {A (t) i } N i=1 = {A (t) i } Np i=1 ∪ {A (t) j } Ne j=1 ∪ { Â(t) i } Nm i=1 18:for eachA (t) i do 19: f itness (t) i = EVALUATE(A (t) i , f θ , D train , D val , e ckp , N f t ) 20:end for 21: end for 22: A * = Ai f itness i {Select algorithm with highest fitness} 23: return A * , f θ =0where INITIALIZE uses an initialization operator to generate a random population of algorithms, EXPLORE and MUTATE correspond to the exploration and mutation operators respectively. In each generation, a subset of individuals is selected from the previous generation as parents, then new algorithms are generated using the exploration and mutation operators, and merged into the population.For each candidate algorithm, EVALUATE is called to compute its fitness:function EVALUATE(A, f θ , D train , D val , e ckp , N f t ): f ′ θ = FINETUNE(A, f θ , D train , e ckp , N f t ) return ACCURACY(f ′ θ , D val ) =0FINETUNE starts from the e ckp -th checkpoint and incrementally trains for N f t epochs on the training set, with the algorithm A dynamically selecting augmentations for each class: is the accuracy in the previous epoch.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the relevant details and analysis of the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We reported the average results of multiple tests in the experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided the code for easy reproduction.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We are fully qualified.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the implications in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We give references to the datasets and code used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To evaluate the effectiveness of LLM-AutoDA in highly imbalanced scenarios, we adjusted the imbalance ratio to 200 and conducted comparative experiments. The experimental results, as shown in Figure 9 , demonstrate that LLM-AutoDA consistently outperforms other long-tailed data augmentation methods. In the figure13, we visualize the selection process of the data augmentation strategies provided by the model. We train the discovered data augmentation methods on CIFAR-100 with an imbalance ratio of 100 under the bi-entropy loss, where the x-axis is the epoch. The y-axis represents the data augmentation techniques that may be used in this work, including Mirror, EdgeEnhance, Detail, Smooth, AutoContrast, Equalize, Invert, GaussianBlur, Rotate, and Flip. The augmentation intensity ranges from 0 to 1 for all of these techniques. For samples of the first class, on the right side, different data augmentation methods are represented, with a dot in the grid indicating that the method represented by that row is selected, and dots of different colors indicating multiple selections. The height in the grid represents the intensity given by our strategy. In the early stages of training, the selected data augmentation strategies are relatively random, but as the model continues training, the trends in the selected data augmentation methods, intensities, and number of times selected gradually stabilize. This shows that the data augmentation methods provided by the large model can effectively achieve convergence on the long-tailed learning model. This demonstrates the feasibility of using a large model to search for and design data augmentation strategies for long-tailed learning models.Using LLM-AutoDA to optimize data augmentation strategies for long-tailed recognition tasks requires significantly less time compared to the manual design of data augmentation strategies by humans. LLM-AutoDA can obtain highly effective strategies within a mere 2 3 hours under a given framework.You are requested to design a novel algorithm that, given a set of augmentation techniques, selects several of them based on the change in per-class accuracy between the current training time point and the previous one, to be employed in the subsequent training phase, with the aim of enhancing the model's ability to tackle longtail problems. This algorithm should deviate from existing methodologies present in the literature.Describe the new algorithm and main steps in one sentence, place the sentence inside curly braces, Next, implement it in Python as a function named get_aug_type. This function should accept 6 input(s): 'aug_weight', 'ACCs', 'History_ACCs', 'lats_chose_matix', 'lats_chose_exts', 'epoch'. The function should return 1 output(s): 'chose_matrix,chose_exts'. aug_weight is a two-dimensional integer array initialized to 1, used to record the historical weight information for each category (indexed by rows) across every augmentation technique (indexed by columns). ACCs is a one-dimensional integer array showcasing the performance of each category at the current training instant, specifically, the count of correct predictions within that category. History_ACCs is a two-dimensional integer array that records the number of correct predictions made by each augmentation technique (column-wise) the last time they were employed for every category. lats_chose_matrix is a twodimensional Boolean array indicating whether specific augmentation techniques (by column index) were utilized for each category (row index) in the previous training step; True signifies usage, while False denotes non-usage. lats_chose_exts is a two-dimensional float array representing the application intensity of each augmentation method across classifications at the current point in time, with a range from 0 to 1, where higher numbers imply greater enhancement strength. Describe the new algorithm and main steps in one sentence, place the sentence inside curly braces. Next, implement it in Python as a function named get_aug_type. This function should accept 6 input(s): 'aug_weight', 'ACCs', 'History_ACCs', 'lats_chose_matix', 'lats_chose_exts', 'epoch'. The function should return 1 output(s): 'chose_matrix,chose_exts'. aug_weight is a two-dimensional integer array initialized to 1, used to record the historical weight information for each category (indexed by rows) across every augmentation technique (indexed by columns). ACCs is a one-dimensional integer array showcasing the performance of each category at the current training instant, specifically, the count of correct predictions within that category. History_ACCs is a two-dimensional integer array that records the number of correct predictions made by each augmentation technique (columnwise) the last time they were employed for every category. lats_chose_matrix is a two-dimensional Boolean array indicating whether specific augmentation techniques (by column index) were utilized for each category (row index) in the previous training step; True signifies usage, while False denotes non-usage. lats_chose_exts is a two-dimensional float array representing the application intensity of each augmentation method across classifications at the current point in time, with a range from 0 to 1, where higher numbers imply greater enhancement strength. First, describe the new algorithm and main steps in one sentence, place the sentence inside curly braces.Next, implement it in Python as a function named get_aug_type. This function should accept 6 input(s): 'aug_weight', 'ACCs', 'History_ACCs', 'lats_chose_matix', 'lats_chose_exts', 'epoch'. The function should return 1 output(s): 'chose_matrix,chose_exts'. aug_weight is a two-dimensional integer array initialized to 1, used to record the historical weight information for each category (indexed by rows) across every augmentation technique (indexed by columns). ACCs is a one-dimensional integer array showcasing the performance of each category at the current training instant, specifically, the count of correct predictions within that category. History_ACCs is a two-dimensional integer array that records the number of correct predictions made by each augmentation technique (columnwise) the last time they were employed for every category. lats_chose_matrix is a two-dimensional Boolean array indicating whether specific augmentation techniques (by column index) were utilized for each category (row index) in the previous training step; True signifies usage, while False denotes non-usage. lats_chose_exts is a two-dimensional float array representing the application intensity of each augmentation method across classifications at the current point in time, with a range from 0 to 1, where higher numbers imply greater enhancement strength. This paper aims to innovate the data augmentation paradigm in long-tailed learning, which greatly improves the degree of freedom of long-tailed data augmentation. However, there are still some limitations of our approach. For example, the scoring mechanism for augmentation strategies is not perfect, and a more comprehensive scoring mechanism is needed. In addition, how to break through the search space limitation and generate novel augmentation methods is also a problem to be solved.Traditional data augmentation is not the best choice for long-tail learning, and recent long-tail data augmentation methods still lack degrees of freedom. The positive impact of our method is that it can give a large number of augmentation strategies suitable for long-tail learning in a short time, which greatly reduces the time and human cost consumed to design the strategy. Of course, there is also a small negative impact, that is, it is easy to cause dependence on LLMs.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper presents a novel framework, LLM-AutoDA, which leverages large language models (LLMs) for long-tailed data augmentation. The framework fine-tunes a long-tailed learning model using augmented training data, where performance improvements on the validation set serve as a reward signal to iteratively refine the data augmentation strategies. The experimental results demonstrate that LLM-AutoDA significantly outperforms state-of-the-art data augmentation and re-balancing methods.

The paper received four positive scores following the rebuttal period (5, 5, 7, 8), indicating recognition of its novelty, particularly in addressing long-tailed problems. However, there are several important issues that the authors should carefully consider, even if the paper is accepted:

1. Fair Comparisons with Baselines: While using LLMs in machine learning is novel, LLMs inherently contain external knowledge. It is crucial to ensure fair comparisons with other methods. The authors should carefully design additional baselines and thoughtfully select benchmarks to evaluate performance. An in-depth analysis of which components contribute most to addressing long-tailed problems is also needed.

2. Transparency and Interpretability: Models used in the machine learning community should prioritize transparency. In the context of image recognition tasks, deep neural networks are already opaque, and the inclusion of black-box optimization, such as LLMs, exacerbates this issue. The authors should discuss the implications of this lack of transparency in detail and explore how insights into long-tail problems can still be gained.

Reviews: review: paper_summary: paper_summary: This work introduces gradient-free black-box optimization algorithms to formulate appropriate data augmentation methods, achieving some performance improvements. Utilizing LLM for evolutionary strategies is interesting, but the final augmentation strategy remains a black box. This makes it difficult to validate the authors' claims about targeting long-tail data and does not fully align with their stated motivation. I believe this work requires further improvement.
strength_weakness: strength_weakness: 1. The authors' survey of the long-tail recognition field is insufficient. For example, in the introduction, they state that existing long-tail data augmentation methods either augment in feature space or directly use traditional methods. I have two counterpoints to this claim. First, existing long-tail data augmentation methods are not limited to the feature space, such as CMO (CVPR2022). Second, what are the drawbacks of augmentation in the feature space? What is the authors' intended motivation?

2. The authors aim to develop targeted data augmentation strategies for each class, but LLM cannot evaluate the characteristics and deficiencies of image data. I am personally skeptical about formulating strategies through prompts alone, and the experimental results do not show distinct differences compared to other methods.

3. This work introduces gradientless black-box optimization, and the authors attempt to use LLM as a generator of expansion strategies, using the validation set performance as an evaluation metric, and letting LLM undergo evolutionary evolution. However, the authors did not provide the augmentation strategies obtained through evolutionary processes, making it hard to prove the work's claimed focus on long-tail problems.

4. The selected comparison methods are insufficient. The authors should conduct extensive comparisons with existing long-tail data augmentation methods, such as OFA (ECCV2020), GistNet, CMO (CVPR2022), H2T (AAAI 2024), and FUR (IJCV2024). Additionally, comparisons with vision foundation model-based methods, such as LPT (ICLR2023) and PEL (ICML2024), should be included.
questions: questions: See Weaknesses*
limitations: limitations: None
paper_summary: paper_summary: The paper proposes LLM-AutoDA to select the optimal augmentation strategies with the help of pre-trained LLMs. Specifically, the authors carefully designed various prompts to instruct an LLM to design new algorithms or mutate existing algorithms with the goal of improving the validation performance on long-tail classification tasks. Experiments show the effectiveness of LLM-AutoDA on three datasets under the long-tail setting.
strength_weakness: strength_weakness: - The formulation of providing class-specific validation accuracy and carefully designed text prompts to deploy LLM in data augmentation is somewhat novel. However, other parts of the method have been studied before. The LLM is an off-the-shelf method. The idea of searching for optimal augmentation configuration is proposed in AutoAugment, and the approach of using the population-based method to select a good augmentation strategy is studied in PBA[1]. Class-specific augmentation strategies are studied in AdaAug[2] and AdaTransform[3]. 
 
- The proposed method is very computationally expensive. It uses a large pre-trained model to process the augmentations. It trains a population of models instead of a single model. If we look at the best-performing BCL + LLM-AutoDA baseline in Table 1, LLM-AutoDA slightly leads DODA (+1.2%) with IR=50 and performs the same with DODA with IR=100 in terms of the “all” test accuracy.  What is the computation cost of DODA when compared to LLM-AutoDA? It raises the concern whether the huge computation power in LLM-AutoDA is justified.

- The clarity of the paper needs to be improved. I found that the notations introduced are not explained clearly. For example, I do not understand the meaning of “the weights of each augmentation method”, $W$; what is the definition of the selection matrix $A$? Is it a zero-one matrix in {0,1}$^{C \times M}$, where $C$ is the number of classes, and $M$ is the number of augmentations? Sometimes, the same symbol is used to represent different concepts. For example, $A$ is the selection matrix and the population algorithm. Are the selection matrices the same as the population algorithms? It seems that the set notation of $A$ in lines 224-231 misses the parentheses. The unclear definition and explanation of the notations make it hard to understand the algorithm precisely.

(minor) There are quite some typos in the manuscript, for example
- Inconsistent use of “fine-tune” and “finetune” in line 204
- Subscript in $f_{aug}$ in line 206
- “Blod” should be “bold” in Table 1 and Table 2 captions

__References__

[1] Ho, Daniel, et al. "Population based augmentation: Efficient learning of augmentation policy schedules." International conference on machine learning. PMLR, 2019.

[2] Cheung, Tsz-Him, and Dit-Yan Yeung. "Adaaug: Learning class-and instance-adaptive data augmentation policies." International Conference on Learning Representations. 2021.

[3] Tang, Zhiqiang, et al. "Adatransform: Adaptive data transformation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
questions: questions: - How does the method perform compared with popular AutoDA works, such as AutoAugment and the simple RandAugment algorithm?

- In previous AutoDA work, AutoAugment learns the probability and magnitude of applying multiple augmentations to a target dataset; AdaTransform[3] and AdaAug[2] can learn class-specific and instance-specific augmentation strategies. What are the benefits of using LLM to select the augmentation parameters? 

- The paper listed some example prompts in the appendix to ask the LLM to design a new algorithm or mutate an existing algorithm. Can the authors provide some examples of the new algorithms output from the LLM? How are they different from the manually designed method?
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper proposes to leverage large language models (LLMs) to help automatically facilitate data augmentation for long-tailed learning. It first discusses the limitations of traditional re-balancing or data augmentation methods. Then it proposes a novel LLM-based augmentation framework LLM-AutoDA. LLM-AutoDA automatically searches for the optimal augmentation strategies and re-balance the long-tailed data. The experimental results demonstrate the effectiveness of LLM-AutoDA on multiple long-tailed datasets.
strength_weakness: strength_weakness: 1. It is better to provide some example augmentation strategies that the LLM generates and briefly analyze why such generated strategies can help learning.
2. What does 1/|gap| in the x axis of Figure 5 mean? How will it impact the performance of different LLMs?
3. Some related works regarding re-sampling and data augmentation are missing [1].
4. It is better to unify the figure and text size in the appendix, such as Figure 9 (too large), Figures 11-13 (different sizes)

[1] How re-sampling helps for long-tail learning?
questions: questions: See weaknesses above.
limitations: limitations: The limitations have been discussed in Appendix Section F.
paper_summary: paper_summary: The paper presents a novel approach by ingeniously leveraging large language models (LLMs) to generate data augmentation strategies for long-tailed problems. Given the recent rapid advancements in LLMs and their potential applications beyond natural language processing, this integration of LLMs with long-tailed learning is both creative and timely.
strength_weakness: strength_weakness: 1) The paper proposes multiple prompts to collaborate to update the data augmentation strategy, please explain what impact they will have when multiple prompts collaborate.

2) Can you try to evaluate and compare the performance of SimpleLLM and LLM-AutoDA under different long-tailed settings such as different imbalance ratios? I think this better reflects the ability of the searched strategy to adapt to different long-tailed scenarios.

3) This paper has some syntax errors, please improve.
questions: questions: It is interesting that the authors upgrade the data augmentation paradigm of long-tailed learning, I wonder if this paradigm can also be combined with other domains with interesting impact. If the author can give relevant analysis, it will help to improve the audience of the paper.
limitations: limitations: Yes

Comments: title: Clarifying Contributions and Addressing Concerns
Thanks for your efforts
Thanks for your efforts
To authors
Thank you and reply to your concerns
Thank you and reply to your concerns
Clarification of the problem
To authors
comment: Dear Reviewer,

Thank you for your valuable feedback and for taking the time to read our response to Reviewer 37A8. We greatly appreciate your positive comments and the concerns you have raised. Your perspective has provided us with an opportunity to further clarify and strengthen the contributions of our work. Please allow us to provide a preliminary response.

1. Method of generating samples: In fact, the generation of new methods is not limited to combining existing methods; we also encourage the creation of algorithms that differ from existing individual algorithms. This is similar to a trade-off between utilizing existing knowledge and encouraging innovation. In this process, the continuous evaluation and selection of real long-tailed models have precipitated effective insights during the constant balancing of combination and innovation. Insights are distilled during this process. The dual evidence of the performance change curve and the final results also demonstrate this point. We believe that this process of distilling insights through evolution in the trade-off, as well as the visible and observable process and results, are sufficient to illustrate the findings of this paper.

2. Whether it leans more towards a black box: The use of large models and black-box optimization certainly introduces some black-box elements, but this does not mean that the overall research leans towards a black box. We believe that the strategies generated from this process should bring insights to the field (the provided evidence proves this point), and the combination of LLM's creativity and the evaluation of long-tailed learning itself is an insight (many reviewers have acknowledged this). Our research employs some black-box factors, but globally, it has intrinsically explainable motivations and can provide evidence to supplement post-hoc explanations.

3. Whether it belongs to the reinforcement learning paradigm: To some extent, it shares some common ideas with reinforcement learning, but it is difficult to fully describe it as such. Our research is a holistic approach that involves the organic combination of the learning and creative capabilities of large models with long-tailed learning. It is challenging to summarize it as reinforcement learning. Similar primitive ideas of research based on evaluation to guide a certain trade-off exist in many areas of artificial intelligence, such as evolutionary computation.

Furthermore, we summarize our response to Reviewer 37A8 to answer the question of new insights. The 0th generation algorithm introduced adaptive weight adjustment to personalize augmentation strategies for each category. The 2nd generation expanded to a global perspective, introducing a cross-category resource balancing mechanism to address the overall data distribution problem. The 5th generation further enhanced adaptability and exploration, balancing exploitation and exploration to uncover the optimization potential of tail categories. We believe this aligns with the insights proposed by humans in state-of-the-art long-tailed research from the perspective of category balance. If you prefer to see data augmentation methods at the image feature level or different from mainstream perspectives, we can make slight modifications to the initial definition prompts or seeds. If you have any other questions, we will still actively respond, or you can wait for our revised version.

Thank you again for your efforts!
Thank you for your valuable feedback and for taking the time to re-evaluate our work. We greatly appreciate your recognition of the improvements made in the revised manuscript, particularly the strengthened experimental results and the further explanations provided on the augmentation strategies discovered by the LLM.

We are pleased to know that our efforts have addressed your concerns and have led to an increase in your assessment score. Your recommendation to continue improving the clarity and fixing the typos in the final version of the manuscript is well noted, and we will make sure to address these issues before the final submission.

Once again, we would like to express our gratitude for your constructive comments and suggestions throughout the review process. We sincerely feel that you are helping us improve the quality of this paper, and we are grateful for that.
Thank you very much for your feedback and recognition. We are delighted to see that our comment has addressed your main concerns and further demonstrated the effectiveness of large language models in tackling the challenges of long-tailed data distributions within our framework. Your comments, along with those from other reviewers, have greatly helped us improve the paper. Once again, we sincerely appreciate your valuable opinions and suggestions.
Thanks for the explanation. The response strengthened the experimental results by comparing the proposed method with other automated data augmentation baselines. The authors also provided further explanations on the augmentation strategy discovered by the LLM. I am increasing the score from 3 to 5. I also recommend the authors continue improving the clarity and fixing the typos in the final version of the manuscript.
I appreciate the authors' response, which has largely addressed my concerns. Furthermore, I noticed that in the public response and in the reply to reviewer 37A8, the authors provided additional information about the evolutionary process and related strategies. I commend the efforts of both the reviewers and the authors. I believe that the demonstration of the process and the insights revealed highlight the effectiveness of large language models in addressing the challenges of long-tailed data distributions within this framework. This further convinces me that this paper will bring new vitality to the community. Based on these considerations, I have decided to raise my rating to 8.
I have read your response to Reviewer 37A8, and I share the same concerns. Based on the three examples you provided, it seems that the black-box optimization using LLM mainly results in combinations of different methods to augment samples. I would like the authors to provide new insights into which aspects of the long-tail distribution challenge require more attention, making the use of LLMs more transparent. As it stands, the work still leans towards being a black box. Can I understand this work as a reinforcement learning paradigm, but using LLMs to generate solutions? Overall, I hope the contributions of this work can be more transparent.
To address your other concern, we've analyzed the algorithms presented in the PDF to better illustrate the evolution of data augmentation and the new insights it brings.

We've chosen to focus on the 0th, 2nd, and 5th iterations of our augmentation method to demonstrate the progressive optimization process for long-tailed data augmentation in long-tailed scenarios.

#### 0th Iteration Method:

**Motivation in Long-Tailed Context**: In long-tailed data, different categories have vastly different needs and effects for augmentation. To adapt to these differences, a method that can adaptively adjust augmentation strategies for different categories is needed.

**Formal Description**: This method dynamically adjusts the weight matrix `aug_weight` of augmentation methods, increasing it as accuracy improves and decreasing it as accuracy declines. For category i and augmentation method j, the weight update formula is:

```
aug_weight[i][j] = aug_weight[i][j] + 1, if Accs[i] > History_Accs[i][j]
aug_weight[i][j] = aug_weight[i][j] - 1, if Accs[i] ≤ History_Accs[i][j]
```

**Insight**: Through adaptive weight adjustment, this method can find the optimal augmentation combination for each category, effectively addressing the problem of large differences in augmentation needs among categories in long-tailed data. Formally, the optimal augmentation combination satisfies:

```
j* = argmax_j(Accs[i] | aug_weight[i][j])
```

#### 2nd Iteration Method:

**Motivation in Long-Tailed Context**: In long-tailed scenarios, considering only the augmentation effect of a single category is insufficient. It's necessary to balance the demand for augmentation resources across different categories globally to alleviate the data imbalance problem.

**Formal Description**: This method introduces an importance evaluation metric `importance`, comprehensively considering current accuracy, historical average accuracy, and cumulative weight:

```
importance[i] = Accs[i] - mean(History_Accs[i]) × epoch + sum(aug_weight[i])
```

Categories are ranked according to importance, prioritizing augmentation methods for categories with high importance (usually tail categories).

**Insight**: By quantifying and comparing the demand for augmentation across different categories globally, this method can more fairly and effectively allocate augmentation resources in long-tailed data. The design of the importance metric balances three key factors:

```
importance[i] ~ Accs[i] (current performance) - mean(History_Accs[i]) (historical performance) + sum(aug_weight[i]) (resource occupation)
```

This global balancing mechanism helps alleviate data imbalance in long-tailed problems and promotes learning for tail categories.

#### 5th Iteration Method:

**Motivation in Long-Tailed Context**: In long-tailed scenarios, tail categories lack sample quantity and diversity. Relying solely on known effective augmentations is insufficient; continuous exploration of new augmentation possibilities is needed to uncover more optimization potential.

**Formal Description**: This method introduces a gain evaluation metric `delta_acc` within categories, measuring the improvement of current accuracy relative to historical weighted accuracy:

```
delta_acc[i] = Accs[i] - sum(History_Accs[i] × aug_weight[i])
```

Truly effective augmentation methods are selected based on `delta_acc`, while prioritizing previously untried methods to balance exploitation and exploration.

**Insight**: Through gain evaluation and exploration mechanisms, this method achieves fine-grained selection and continuous optimization of augmentation methods within each category. Formally:

```
chose_matrix[i][j] = True <=> last_chose_matrix[i][j] = False and j = argmax_j(delta_acc[i])
```

This balance is particularly important in long-tailed scenarios because the optimization space for tail categories is often difficult to fully exploit. It's necessary to continuously try new possibilities while utilizing known effective augmentations to overcome the lack of sample diversity.
In summary, these three generations demonstrate the evolution of our algorithm for long-tailed recognition tasks:

1. The 0th iteration introduced adaptive weight adjustment, personalizing augmentation strategies for each category.

2. The 2nd iteration expanded to a global view, introducing cross-category resource balancing to address overall data distribution.

3. The 5th iteration enhanced adaptability and exploration, balancing exploitation and exploration to uncover optimization potential for tail categories.

This progression shows how large language models gradually explored strategies for long-tailed problems, introducing new insights with each iteration. The evolving approach offered incremental improvements and fresh perspectives on data augmentation for long-tailed scenarios.
Thank you for your response! We're pleased that our efforts have been recognized and that some of your concerns have been addressed. Let's continue to address your questions:

Regarding your request for comparisons with more automatic augmentation methods, we've done our best to provide a comprehensive answer. As you can see, our method, leveraging the advantages of large language models, shows significant improvements:

### Accuracy(%) on CIFAR-100-LT(IR=100) dataset with Cross-Entropy (CE) Loss

| Method | Head | Medium | Tail | All |
|--------|-----:|-------:|-----:|----:|
| Vanilla | 65.6 | 36.2 | 8.2 | 40.1 |
| AA | 68.6 | 43.7 | 8.0 | 41.7 |
| PBA | 63.3 | 49.5 | 8.1 | 41.9 |
| FA | 64.1 | 49.5 | 8.5 | 42.3 |
| DADA | 69.3 | 41.2 | 7.7 | 41.0 |
| RA | 64.8 | 44.1 | 8.0 | 40.5 |
| LLM-AutoDA | 74.9 | 45.3 | 9.6 | 45.0 |

### Accuracy(%) on CIFAR-100-LT(IR=100) dataset with Balanced Softmax (BS) Loss

| Method | Head | Medium | Tail | All |
|--------|-----:|-------:|-----:|----:|
| Vanilla | 59.6 | 42.3 | 23.7 | 42.8 |
| AA | 63.5 | 49.1 | 25.3 | 47.0 |
| PBA | 63.2 | 51.4 | 24.6 | 47.5 |
| FA | 61.4 | 46.9 | 28.7 | 46.5 |
| DADA | 56.0 | 54.1 | 21.5 | 45.0 |
| RA | 62.8 | 43.3 | 26.9 | 45.2 |
| LLM-AutoDA | 63.3 | 50.0 | 31.0 | 49.0 |

### Accuracy(%) on CIFAR-100-LT(IR=100) dataset with Cross-Entropy and Deferred Re-Weighting (CE-DRW)

| Method | Head | Medium | Tail | All |
|--------|-----:|-------:|-----:|----:|
| Vanilla | 63.4 | 41.2 | 15.7 | 41.4 |
| AA | 65.5 | 53.7 | 15.9 | 46.5 |
| PBA | 64.0 | 59.5 | 15.6 | 47.9 |
| FA | 64.8 | 52.8 | 17.1 | 46.3 |
| DADA | 62.8 | 51.1 | 19.1 | 45.6 |
| RA | 66.4 | 48.9 | 18.2 | 45.8 |
| LLM-AutoDA | 62.9 | 50.7 | 29.9 | 48.7 |

### Accuracy(%) on CIFAR-100-LT(IR=100) dataset with Label-Distribution-Aware Margin and Deferred Re-Weighting (LDAM-DRW)

| Method | Head | Medium | Tail | All |
|--------|-----:|-------:|-----:|----:|
| Vanilla | 62.8 | 42.6 | 21.1 | 43.2 |
| AA | 66.7 | 49.8 | 20.8 | 47.0 |
| PBA | 68.0 | 49.6 | 21.8 | 47.7 |
| FA | 66.1 | 47.8 | 22.5 | 46.6 |
| DADA | 65.8 | 43.8 | 25.1 | 45.9 |
| RA | 64.7 | 40.8 | 23.6 | 44.0 |
| LLM-AutoDA | 66.7 | 50.1 | 26.3 | 48.8 |

These comparative results clearly demonstrate the superiority of our LLM-AutoDA method across various evaluation metrics and loss functions:

1. **Overall Performance**: LLM-AutoDA consistently outperforms other automatic augmentation methods in terms of overall accuracy ("All" column) across all four loss functions. For instance, with CE Loss, LLM-AutoDA achieves 45.0% overall accuracy, which is a substantial 3.1% improvement over the next best method (FA at 42.3%).

2. **Tail Class Performance**: One of the most notable improvements is in the tail classes. LLM-AutoDA shows remarkable gains in tail class accuracy across all loss functions. For example, with CE-DRW, LLM-AutoDA achieves 29.9% tail accuracy, nearly doubling the performance of the next best method (DADA at 19.1%).

3. **Balanced Performance**: LLM-AutoDA maintains strong performance across head, medium, and tail classes, indicating a more balanced learning approach. This is particularly evident with BS Loss, where LLM-AutoDA achieves the highest tail accuracy (31.0%) while maintaining competitive performance in head and medium classes.

4. **Consistency**: Unlike some other methods that may excel with one loss function but underperform with others, LLM-AutoDA shows consistent improvements across all four loss functions. This demonstrates the robustness and versatility of our approach.


These results underscore the effectiveness of leveraging large language models for data augmentation in long-tailed classification tasks. LLM-AutoDA not only improves overall accuracy but also significantly enhances the model's ability to learn from underrepresented classes, addressing a key challenge in imbalanced datasets.
Thanks for the response. I understand the authors are proposing an Automated Data Augmentation method to search for better augmentation strategies for long-tail datasets. I also agree that integrating LLM and long-tail learning is somewhat novel in the review. My major concerns are partly addressed in the response.

- The elaboration of the algorithm and notation in the response for W3 is much more precise than the presentation in the original paper.

- My primary concern is whether using comparatively expensive LLM to discover an augmentation strategy is better than other Automated Data Augmentation methods. A good way to clarify the improvement is to apply other automated searches, e.g., PBA, RandAugment, and Fast AutoAugment, to the long-tail dataset and compare the end classification results with the proposed method.

- The authors provide three screenshots of the code output from the LLM. Can the authors provide further analysis explaining how the generated codes cope with the long-tail problem? Are there any insights or new observations we can find from the LLM strategies?
Dear Reviewer,

Hello. Due to the critical nature of both issues, we have provided a comprehensive response to questions 2 and 3. As you can see, we have stated that we included results in Figure 1 to demonstrate the data augmentation strategies obtained through LLM black-box optimization and their performance changes during the evolution process.

As a reviewer myself, I've also noticed that we seem unable to view the authors' full rebuttal and the additional one-page PDF with experiments in global rebuttal. I believe this might be due to a system error in the settings. I apologize for this inconvenience.

Please wait for the system to update. Thank you for your understanding.
The author has overlooked my third concern, which is that they are unable to provide the data augmentation strategies obtained through LLM black-box optimization. As a result, their claim that this method is specifically tailored to long-tailed problems cannot be verified.