Empowering Large Language Models for Textual Data Augmentation (2024)

Abstract: With the capabilities of understanding and executing natural language instructions, Large language models (LLMs) can potentially act as a powerful tool for textual data augmentation. However, the quality of augmented data depends heavily on the augmentation instructions provided, and the effectiveness can fluctuate across different downstream tasks. While manually crafting and selecting instructions can offer some improvement, this approach faces scalability and consistency issues in practice due to the diversity of downstream tasks. In this work, we address these limitations by proposing a new solution, which can automatically generate a large pool of augmentation instructions and select the most suitable task-informed instructions, thereby empowering LLMs to create high-quality augmented data for different downstream tasks. Empirically, the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods, leading to the best performance on 26 few-shot learning tasks sourced from a wide range of application domains.

heading: 1 Introduction
text: Large language models (LLMs) have recently demonstrated their potential in performing data augmentation on text data (Dai et al., 2023; Chung et al., 2023; Yu et al., 2023; Yoo et al., 2021). Serving as a semantic-preserving transformation function, LLMs transform original texts based on instructions to create diverse and informative data augmentations. With the augmented data, users can further train a spreadable and affordable model (e.g. OPT (Zhang et al., 2022)) to perform specific tasks. Unlike traditional heuristic-based methods such as word swapping (Wei and Zou, 2019) and modelbased methods like back-translation (Fadaee et al.,
*The first two authors contributed equally to this work. Kaize Ding is the corresponding author.
2017), LLMs offer great potential to produce more fluent, diverse, and semantically consistent augmentations for text data, owing to their great understanding and generalization capabilities.
Despite the early success of LLMs for textual data augmentation, existing methods (Dai et al., 2023) that simply prompt LLMs with human-crafted augmentation instructions (i.e., Manual-LLMDA methods) have the following major bottlenecks: (1) Firstly, their efficacy heavily relies on the quality of the augmentation instructions, which are manually engineered by domain experts. This manual process is not only domain knowledge-intensive but also prone to inconsistencies, potentially compromising the quality of augmented data. Subtle variations in how these instructions are formulated can significantly influence the outcomes, as demonstrated by recent studies (Ishibashi et al., 2023; Zhu et al., 2023); (2) Secondly, usually text augmentation instructions are written in a task-agnostic form for a general purpose, however, the lack of context information on downstream tasks could lead to dramatic performance disparity on different downstream tasks, as shown in Fig. 1. Without considering the specific properties of the target tasks, LLM may generate low-quality augmented data (Ribeiro et al., 2020; Wei and Zou, 2019). ar X iv :2 40 4. 17 64
2v 1
[ cs
.C L
] 2
6 A
pr 2
02 4
12734
To address the aforementioned challenges, in this paper, we introduce a new framework – Self-LLMDA that automates augmentation instruction generation and selection, facilitating LLM to generate task-specific augmented data. The initial phase of Self-LLMDA aims to broaden the span of seed augmentation strategies through the generation of diverse and effective instructions based on LLMs. Following this, Self-LLMDA employs a scoring model to identify and select the most relevant instructions that are likely to bolster the performance of target models. Such a new textual data augmentation approach ensures a balance between the generative breadth of augmentation instructions and targeted precision of task-specific guidance for downstream tasks.
In our study, we conduct extensive experiments across a large collection of few-shot learning tasks used in previous studies (Min et al., 2022; Ye et al., 2021; Khashabi et al., 2020). This collection includes 26 different types of tasks across hate speech detection, question answering, natural language inference, and phrase detection datasets. Our study stands out for its extensive coverage of tasks, setting a new benchmark in the application of LLMs for textual data augmentation when compared to previous work (Dai et al., 2023; Li et al., 2023; Chung et al., 2023). The empirical results demonstrate that the proposed approach Self-LLMDA significantly outperforms various baseline methods in generating high-quality augmented textual data. To summarize, our main contributions are as follows: • We introduce a framework Self-LLMDA, which
automates the generation and selection of taskspecific augmentation instructions for LLMs, providing effective data augmentation for text data.
• Through a comprehensive set of experiments, we validate the effectiveness of Self-LLMDA, demonstrating its superior performance in enhancing data quality and model accuracy over existing text data augmentation methods.
• Our in-depth analyses reveal that Self-LLMDA can well generalize across various target models and previously unseen augmentation instructions, demonstrating its versatility and potential for broad applicability.

heading: 2 Related Work

heading: 2.1 Non-LLM Textual Data Augmentation
text: Conventional textual data augmentation methods encompass a variety of techniques aimed at enhanc-
ing the diversity of textual datasets without relying on large language models (i.e., Non-LLMDA methods). Those methods range from simple heuristicbased methods to generative model-based methods. For heuristic-based approaches, such as synonym replacement (Zhang et al., 2016) and word shuffling, stand out for their computational efficiency and simplicity, making them ideal for large-scale data augmentation with minimal computational demands. Another notable example is the Easy Data Augmentation (EDA) technique introduced by Wei and Zou (2019), which employs token-level perturbations—random insertion, deletion, and swapping—to improve performance across a spectrum of text classification tasks.
For model-based approaches, researchers have employed seq2seq and language models for data augmentation. Back-translation (Fadaee et al., 2017) employs translation models to preserve semantic integrity while generating paraphrases (Fadaee et al., 2017). Conditional masked language models like BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) can also be utilized for data augmentation (Cheng et al., 2022; Wu et al., 2018). By masking words within sentences and subsequently generating replacements, these models introduce linguistic variations. Furthermore, other methods (Kumar et al., 2021; Edwards et al., 2023) leverage the capabilities of generative language models like GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2019) for data augmentation. These approaches perform conditional generation based on class labels. Additionally, some studies have explored augmentation in the feature space. Mixup techniques interpolate within word or sentence embeddings (Guo et al., 2019), while others introduce random multiplicative and additive noise to the feature vectors (Kurata et al., 2016). Despite their utility, these conventional Non-LLMDA methods often come with limitations in readability and contextual consistency.

heading: 2.2 LLM-based Textual Data Augmentation
text: Recent advancements in LLMs have demonstrated their superiority in generating high quality and contextually relevant augmented data (Brown et al., 2020). LLMs are increasingly employed as labelpreserving transformation functions, where an original example is transformed or perturbed according to manually crafted instructions (Dai et al., 2023; Yoo et al., 2021; Piedboeuf and Langlais, 2023).
Concurrently, several studies (Chung et al., 2023; Yu et al., 2023; Li et al., 2023; Ubani et al., 2023; Meng et al., 2022) have explored the generation of conceptually similar yet semantically distinct synthetic examples. These methods, however, mostly rely on manual instruction design. In contrast, our work automatically generates label-preserving augmentation instructions by prompting LLMs, thus reducing dependency on manually crafted instructions. Furthermore, we introduce an instruction selection model that chooses appropriate instructions for arbitrary downstream tasks.

heading: 3 Preliminary
text: Problem Definition. Textual data augmentation involves applying a label-preserving transformation function T (·) to a dataset D = {(xi,yi)}ki=1, where each example consists of an input text xi (a sequence of tokens) and a corresponding label yi (also a sequence of tokens). The augmented dataset D′ is generated as follows, ensuring that the output label y′i remains unchanged:
x′i = T (xi),y ′ i = yi. (1)
A target model F is then trained on the union of the original and augmented datasets, D ∪D′, with the training objective defined as:
L(x̂i,ŷi)∈D∪D′(Fθ(x̂i), ŷi). (2)
Therefore, designing an effective transformation functionT (·) that produces high-quality augmented data D′ is crucial for improving the downstream performance of model Fθ.
Manual-LLMDA. For Manual-LLMDA methods, the transformation function T (·) is realized through a combination of an LLM and a manual-crafted instruction Iman (e.g., paraphrasing). The LLM is prompted to generate semantic-preserving transformations of the input text xi for the augmented
dataset D′:
x′i = LLM(Iman,xi),y′i = yi (3)
4 Proposed Approach – Self-LLMDA To reduce the human efforts in designing augmentation instructions and selecting a task-specific instruction for a given task, we propose Self-LLMDA depicted in Fig. 2. The process begins with the LLM generating a diverse set of potential instructions I = {Ij}nj=0 from a given set of seed instructions Iseed = {Iman}:
I = LLM(Iseed). (4)
A selection model S then scores these generated instructions against the dataset D to identify the most suitable instruction I∗:
I∗ = S(I,D). (5)
Based on the selected instruction I∗, the LLM performs data augmentation on D, producing an enhanced augmented dataset D′ for training the target model more effectively.

heading: 4.1 Augmentation Instruction Self-Generation
text: Inspired by the self-instruct methodology (Wang et al., 2022), this phase generates augmentation instructions from a seed set of 13 human-crafted instructions. These seed instructions act as exemplars, guiding the LLMs toward the creation of novel and diverse instructions that maintain the semantic integrity of the input text. To generate a broad and diverse set of augmentation instructions without the bias introduced by a few task examples, we exclude the task-specific data from the instruction generation. This will leverage the zero-shot learning capabilities of LLMs to produce a wide array of potential augmentation instructions. We use the following prompt to encourage LLMs to explore various augmentation techniques:
“Come up with a series of textual data augmentation methods and you need to generate more diverse data augmentation method that can keep the semantic meaning of the input sentence. {Iseed} ”
Through iterative cycles of generation and refinement, we filter out instructions that are too similar to existing ones based on ROUGE-L (Lin, 2004). The unique generated instructions from each iteration are then incorporated back into the seed instruction pool, enriching the seed instructions for subsequent generation rounds. This process is repeated until we reach a collection of 100 augmentation instructions. To ensure diversity and eliminate redundancy, we further refine this set by removing duplicates based on their method names. This filtration results in a final set of 51 unique augmentation instructions.

heading: 4.2 Task-Informed Instruction Selection
text: Recognizing that augmentation instructions may not be universally applicable across different tasks, we implement a selection mechanism, tailored to the specific requirements of each task and its corresponding target model. This process involves a scoring model S to evaluate the suitability of each instruction for the task at hand. The scoring model S, as shown in Fig. 3, outputs a ranking score qj indicating the instruction’s effectiveness based on the pair of instruction and task dataset. Based on the notable instruction-following capabilities of FLAN-T5 (Chung et al., 2022; Raffel et al., 2023), we choose FLAN-T5-Large (Chung et al., 2022; Raffel et al., 2023) as the backbone of our scoring model. The input for scoring model S is:
“Given the dataset for task T and the instruction data, determine if this is a suitable instruction to address the task for model F . Task Dataset: {xi}mi=0 Instruction: Ij . Is this instruction appropriate?”
where T is the task name (e.g. GLUE-RTE), F is the target model name (e.g. OPT-125m). Since most of the tasks did not have a task description and manually designing the task description is time consuming, we utilize the few-shot examples {xi}mi=0 from the dataset as the task description. Here, we calculate qj by assessing the logit value of the “yes” token of the last position of input from FLAN-T5-Large, as shown in Fig. 3. Next, we will introduce the optimization and inference procedure of the scoring model S respectively.
Model Optimization. The instruction selection model is trained to prioritize generated augmentation instructions based on their impact on downstream task performance. Its goal is to assign the highest scores to instructions that lead to the most effective data augmentation. To enhance scalability and computational efficiency, our model optimizes the selection process for a given task D by sampling a subset of augmentation instructions {Ij}nj=0 (where n > 1) from the pool of candidates. The model then computes scores {qj}nj=0, representing the relative effectiveness of each instruction. The optimization objective is formulated as a cross-entropy loss, designed to accurately distinguish between the effectiveness of these instructions {Ij}nj=0. The loss function is given by:
LS = − n∑
j=0
is_max(rj) log σ(qj) (6)
Here, is_max serves as a binary indicator function that identifies the instruction yielding the maximum effectiveness, and σ is softmax that normalizes the qj (a probability generating “yes” token interpreted as a ranking score associated with jth instruction) over the sampled augmentation instructions, rj is the downstream task performance of a trained target model on augmented data D′j ∪ D. Model Inference. When encountering a new task, the selection model S evaluates all potential instructions to determine the most suitable one I∗, denoted by the highest score:
I∗ = Iargmax({qj}|I|j=0) (7)
This optimal instruction, I∗, is then employed to prompt the LLMs to generate augmented data. This selection mechanism ensure the use of the most effective instruction for enhancing data utility across diverse NLP tasks.

heading: 5 Experiment

heading: 5.1 Experimental Setup
text: Evaluation Datasets. In this study, we select 26 few-shot learning tasks spanning a wide range of NLP challenges, sourced from CrossFit (Ye et al., 2021), UnifiedQA (Khashabi et al., 2020), and MetaICL (Min et al., 2022). These datasets were chosen for their diversity, encompassing both classification tasks (Class)—such as natural language inference, paraphrase detection, and hate speech identification—and non-classification (Non-Class) tasks, notably question answering, to ensure a broad evaluation spectrum. The selection of tasks is significantly larger and more diverse than that in other relevant works (Dai et al., 2023; Chung et al., 2023).
To investigate the generalization ability of Self-LLMDA, we split the 26 tasks into training and test tasks as for form “train→ test”. We train the augmentation instruction selection methods on training tasks and evaluated it on test tasks. The task split involves four settings: Class → Class, Class → Non-Class, Non-Class→ Class, and Random → Random, where “Random” represents a mixture of randomly selected tasks *. This design allows us to investigate the performance of selection models when applied across similar and disparate task types, providing insights into their generalizability and effectiveness.
Evaluation Metrics. To handle all types of tasks simultaneously, we unify all downstream tasks, including classification and non-classification tasks, using a text-to-text approach (Raffel et al., 2023). For each task, we feed the input text to the target model Fθ and train it to generate the corresponding target text. We choose OPT (Zhang et al., 2022) from three different sizes (e.g. 125m, 350m and 1.3b*) as our target modelsFθ. During training*,Fθ takes the training example xi as the input, and is optimised to
*Details of training and testing tasks split is in Tab. 9. *Due to GPU memory constraints, the training mini batch size for the 1.3B model is set to 2, while the batch sizes for the 125M and 350M models are set to 8. This difference in batch sizes may cause the 1.3B model to achieve worse performance compared to the 125M and 350M models.
*Detailed hyparameter setting is in Appendix A.
generate yi using the negative likelihood objective function:
LFθ(yi) = − |yi|∑
t=1
logPFθ(y t i |xi,y<ti ) (8)
During inference time, given the test input xtest as well as a set of candidates C, which is either a set of labels (in classification tasks) or answer options (in non-classification tasks), the Fθ computes the conditional probability of each label c ∈ C, where c is a sequence of tokens. The label with the maximum conditional probability is returned as a prediction:
argmaxc∈C
  |c|∑
t=1
logPFθ(c t|xtext, c<t)
  (9)
Specifically, we use macro-F1 for classification tasks, and accuracy for non-classification tasks in our experiment. The overall performance is then quantified by computing the macro-average of these scores across all tasks, encapsulating both accuracy and macro-F1 metrics. To ensure robustness and reduce sampling bias, each experiment under each splitting setting is replicated with five different random seeds. For each few-shot task, we adopt a uniform approach by randomly selecting k = 16 training examples. Following (Min et al., 2022), we did not make perfect label balance between k training examples.
Baseline Methods. In this study, we compare our novel augmentation pipeline, Self-LLMDA, with two different categories of data augmentation methods as baselines: Non-LLMDA and Manual-LLMDA For both Manual-LLMDA and Self-LLMDA, we employ GPT-3.5 Turbo as the backbone LLM. For detailed descriptions of these baseline methods, please see Appendix E. Specifically: • Non-LLMDAmethods. This category includes 13
traditional augmentation techniques: CharacterLevel: Operations such as random swaps, OCR Errors simulation, deletions, insertions, and substitutions. Word-Level: Transformations, including word swaps, deletions, spelling errors, and embedding-based insertions. Contextual-Level: Utilization of language models for word insertions (e.g., using GPT2 (Brown et al., 2020)) and substitutions (e.g., with BERT (Devlin et al., 2018)), and back-translation (Fadaee et al., 2017).
while Bold indicates the best result of the whole table. In each group, the last two rows represent the aggergated result of the whole group of augmentation methods (e.g. average and best).
• Manual-LLMDA methods. This set comprises 13 manually designed augmentation instructions for LLM, including: Character-Level: Perturbations similar to those in Non-LLMDA. Word-Level: Swaps, replacements, and part-of-speech (POS) enhancements. Sentence-Level: Reordering and data mixing strategies. Contextual-Level: Predictive masking, contextual substitutions, and back-translation.
We also report the average and best performance of Non-LLMDA and Manual-LLMDA for better comparison. An extensive ablation study of our taskinformed selection model, presented in § 5.3.

heading: 5.2 Main Results
text: The analysis of experimental results presented in Tab. 1 reveals several findings: Firstly, there is performance inconsistency among the different instructions from Manual-LLMDA. The impact of augmentation instructions varies across different downstream tasks and models. This highlights the difficulty in creating universally effective data aug-
mentation instruction. Secondly, Manual-LLMDA is not always better than Non-LLMDA. In controlled comparisons focusing on specific augmentation topics, Manual-LLMDA’s advantages over Non-LLMDA were not clearly evident. For example, in the contexts of “Back Translation” and “Word Swap”, Non-LLMDA outperformed Manual-LLMDA in 5 out of 12 and 7 out of 12 cases, respectively. Lastly, the experimental results show the superiority of Self-LLMDA. Our proposed model consistently outperformed these baseline methods, highlighting the effectiveness of integrating automatic instruction generation with targeted task-specific instruction selection. This approach not only optimizes performance but also reduces the manual efforts typically required to design effective augmentation strategies, showcasing the potential of our model in enhancing data augmentation practices.

heading: 5.3 Ablation Study
text: We add an ablation study to understand the impact of two key components in our framework: aug-
mentation instruction self-generation and the taskinformed instruction selection. Firstly, we train a task-informed instruction selection model S on manually-crafted instructions from Manual-LLMDA and named it Manual-LLMDA+ to understand the contribution of the contribution of LLM selfgenerated augmentation instructions. Secondly, we test the efficacy of our selection model by comparing three alternative selection strategies: (1) Random-select, which randomly select instruction from the pool of augmentation methods for each task; (2) Empirical-select, which selects the prompt that yielded the highest average performance across training tasks, under the assumption that successful prompts on training tasks will generalize well to test tasks; and (3) LLM-Select, which prompts the LLM to chooses the most suitable instruction from candidates based on its internal decision-making processes. The Results in Tab. 2 show that Self-LLMDA consistently outperforms these alternative methods, indicating the benefits of instruction self-generation and task-informed selection in enhancing model performance.

heading: 5.4 Hyperparameter Analysis
text: Here, we closely examined the impact of two critical hyperparameters on the training of our taskinformed instruction model: n and m. The hyperparameter n specifies the number of augmentation instructions to be sampled for optimizing Eq. 6. It should be noticed that, we only vary n at training time, while at inference, we will calculate the score for all the generated instructions and choose the one with the largest score. On the other hand, m determines the number of examples from the task dataset that are used to represent the task, influencing the model’s performance during both the optimization and inference phases. Our analysis, depicted in Fig. 5, highlights several key findings: (1) Optimal Number of Instructions: We found that setting n = 2 leads to the best performance, outperforming other configurations. This suggests that a pairwise comparison, as formulated in Eq. 6, is most effective for our model’s learning process. (2)
Representative Examples: Interestingly, a smaller number of examples (m) appear to better capture the essence of the tasks. This observation indicates that a larger set of examples could introduce noise, potentially detracting from the model’s ability to accurately represent tasks for instruction selection.

heading: 5.5 In-Depth Analysis of the Task-Informed Instruction Selection Model
text: In this section, we provide a detailed analysis of the performance and generalization capabilities of our instruction selection model S, focusing on its generalizability to unknown augmentation instructions, unknown target models, and the specific case studies of the augmentation instructions it selects.
Generalization to Unknown Augmentation Instructions. In this analysis, we delve into the selection model’s adaptability to unknown augmentation instructions by simulating a dynamic environment where new instructions are generated asynchronously by the LLMs. This scenario mirrors practical applications where the augmentation instruction set can expand without necessitating retraining of the selection model. To test this, we constrained the training phase of the selection model to a limited subset of self-generated augmentation instructions (30% of all generated by the LLMs), utilizing the whole generated instructions for evaluation at inference time.
As the results shown in Fig. 6, we can ob-
serve a performance improvement of our selection model over the best performance of Non-LLMDA and Manual-LLMDA. This indicates the robustness of our selection model in adapting to incremental augmentation instructions, effectively selecting suitable instructions even when faced with previously unknown instructions. These observations highlight the efficacy of our selection model in a dynamic augmentation scenario.
Generalization to Unknown Target Models. Our study extended to evaluate the adaptability of our task-informed selection model across diverse target models. By applying the selection model, initially trained on the task performance of a specific target model, to different models. The results of these experiments are presented in Tab. 3. Our findings show that the augmentation instructions selected by our model remain effective even when applied to different target models. Notably, in most scenarios, our model Self-LLMDA, when transferred to alternate target models, outperformed the best results obtained using Non-LLMDA and Manual-LLMDA. This indicates that the underlying pattern determining instruction effectiveness via our instruction selection model is transferable.
Analysis of Selected Instructions. We conducted a detailed analysis of the augmentation instructions chosen by our selection model, and the findings visualized in Fig. 7. The key insights from this analysis are as follows: (1) Diversity of Selected In-
structions: The distribution of selected instructions showcases a wide variety in the types of augmentations chosen by the model, with 3, 2, and 6 unique data augmentation instructions identified for the 125m, 350m, and 1.3b models under Class→NonClass, respectively. This demonstrates the model’s ability to adapt and select from a broad spectrum of augmentation strategies to meet the specific requirements of different tasks. (2) Variability across Models: The selection patterns exhibit notable differences when the model is applied to various target models. This variability indicates preference differences across different target models. (3) Preference for Paraphrase-Based Instructions: A significant portion of the selected instructions fall into the category of paraphrase-based augmentations, such as “Text Paraphrase”, “Paraphrase”, “Contextual Paraphrase”, and “Sentence Paraphrase”. This preference not only highlights the effectiveness and general applicability of paraphrase-based augmentations but also illustrates our task-informed selection model’s nuanced capability to discern and recommend the most suitable paraphrase variation for a given task.

heading: 6 Conclusion
text: In this work, we introduced Self-LLMDA, a novel framework that leverages the capabilities of LLMs for textual data augmentation. Our approach addresses the challenges associated with traditional data augmentation methods and the limitations of manual instruction generation in LLM-based augmentation. Self-LLMDA automates the generation and selection of augmentation instructions, thereby significantly enhancing the quality and applicability of augmented data across diverse downstream tasks. Tested across 26 diverse few-shot learning tasks, Self-LLMDA consistently outperforms both Non-LLMDA and Manual-LLMDA methods, showcasing its effectiveness and applicability.

heading: 7 Limitations
text: This study acknowledges several constraints that delineate the scope of our current work and outline directions for future research:
• Evaluation on a Limited Range of LLMs: Our experiments were conducted primarily with GPT 3.5 Turbo due to the high costs associated with using OpenAI models. While promising results in Tab. 1 suggest that our proposed Self-LLMDA method could potentially perform even better on more advanced models like GPT 4 Turbo, comprehensive testing was not feasible. Similarly, the computational demands of evaluating open-source LLMs such as LLAMA-70b-chat (Touvron et al., 2023), coupled with the extensive number of tasks in our study, exceeded our resources. Despite these limitations, we are optimistic that Self-LLMDA would exhibit enhanced performance across a broader spectrum of LLMs.
• Meta-Prompting Exploration: Within the Self-LLMDA framework, we employed one meta-prompt to guide the LLM in generating diverse and relevant augmentation instructions. However, our exploration of meta-prompting techniques was limited. We acknowledge that more sophisticated prompt engineering could further refine the quality and effectiveness of generated instructions. Investigating more advanced meta-prompting strategies remains an area for future exploration.
• Analysis of Ensemble Augmentation Methods: Our research did not investigate the potential benefits of combining multiple sets of augmented data (e.g., D∪D′1 ∪D′2). Such ensemble approaches introduce additional complexities, such as determining the optimal number of augmentation instructions to include. While we hypothesize that ensemble augmentation could improve model performance, this aspect falls outside the current study’s scope and is earmarked for subsequent investigation.

heading: A Detailed Experiment Settings
text: Generation Configuration. We utilize gpt-3.5turbo as our backbone LLM for augmentation instruction generation and data augmentation. We set the temperature for both of them as 0.7. For the instruction generation, we follow the generation hyper-parameter setting from Wang et al. (2022). For data augmentation, we utilize the default generation hyper-parameter from Chat Completion. The whole experiment including generating augmentation instructions and generating augmentation data costs us $82 USD in total, according to OpenAI’s pricing ( Input $0.0005 / 1K tokens and output $0.0015 / 1K tokens). However, the total experiment cost around $200 USD for debugging and exploration.
Meta Prompts for Data Augmentation As shown in step 3 in Fig. 2, we also need a meta prompt to encourage Self-LLMDA to augment high quality data. The main reason for this meta-prompt setting is because in some augmentation instructions they will discuss some external tools like word-embedding, other language models, if we did not provide the meta-prompt, the LLM will reject the generation of augmented data. The design of meta prompt is as follows:
Please do the following data augmentation steps to the text delimited by triple backticks. If you need any external resources or data, you can just simulate the environment by yourself and finish that step based on your own knowledge since you are the best language model in word. Augmentation Instructions: Ij , Input Data: xi {Iseed}
Task-informed Instruction Selection. The instruction ranking model is initialized with FLANT5-Large(Radford et al., 2019) and is trained using Adafactor (Shazeer and Stern, 2018) with learning rate 5e-5 and dropout 0.1. We train the selection model for 100 epochs and set the early stop with patience 20 epochs. We employ the validation set from training tasks to select the best checkpoint. The search space for different hyperparameter analysis are as follows:
Target Model Finetuning. We use OPT (Zhang et al., 2022) from 125m, 350m, 1.3b different sizes. For all of them we use AdamW(Loshchilov and Hutter, 2019) as our optimizer with learning rate 5e-5
with 10 training epochs. Due to the constraint of GPU memory, for 125m and 350m we set the batch size as 8, while 1.3b we set the batch size as 2. All these experiments is tested on one NVIDIA A100 A100-40G GPU cards.

heading: B Analysis of Self-Generated Instructions
text: In our analysis, we delve into the characteristics and diversity of the self-generated augmentation prompts created by Manual-LLMDA.
Statistical Information. To facilitate a structured examination, we categorize these prompts based on the textual data augmentation taxonomy outlined by Bayer et al. (2022). The distribution and basic statistics of these various augmentation methods are detailed in Tab. 5.
Naming Conventions. A notable aspect of our analysis involves examining the naming conventions of the augmentation methods. Recognizing that the method names often provide a high-level summary of the augmentation approach (e.g., <method name>), we further explore the linguistic patterns within these names. Specifically, we conduct an analysis focusing on the first and last words of each method name. This approach allows us to gain insights into the thematic and functional aspects of the augmentation methods. The distribution of these first and last words in method names is visually represented in Fig. 8. This visual representation aids in understanding the range and focus of the augmentation techniques generated by Manual-LLMDA.
By analyzing these key linguistic elements, we aim to shed light on the creative breadth and thematic focus of the self-generated augmentation instructions.

heading: C Analysis of Generated Data Across Different Augmentation Methods
text: In this analysis, we aim to discern the differences among the original dataset, non-LLM augmented data, data augmented via human-designed instructions, and data augmented using Manual-LLMDA generated instructions. Our focus is on the surfacelevel characteristics of the augmented content, and we consolidate data across all tasks for a comprehensive view. Key observations from the analysis, as detailed in Tab. 6, include the following:
Length of Content. Data augmented by LLMbased methods, on average, exhibits longer content compared to both the original and traditionally augmented datasets. This increase in length could offer a broader spectrum of training examples, potentially aiding in better generalization of target models. However, it also introduces a challenge of dataset inconsistency and the risk of adding unwanted variations.
Perplexity Scores. Interestingly, LLMaugmented content achieves lower perplexity scores (as measured on GPT2-small) compared to traditional augmentation methods. This suggests that the target model like GPT2-small has a better grasp of content augmented by LLMs. A possible explanation for the higher perplexity scores observed in non-LLM text augmentations is that the character and word-level changes might introduce new, irrelevant tokens into the text, thereby increasing complexity.
Closeness to Original Examples. Compared to non-LLM augmentation methods, LLM-based augmentations tend to produce content that is more closely related or less diverse relative to the original examples. This observation points to a potential trade-off between relevance and diversity in the augmented content generated by LLMs.
Augmentation Instruction Pitfalls Across Tasks The effectiveness of augmentation instructions can vary depending on the specific characteristics of the tasks at hand (Ribeiro et al., 2020; Wei and Zou, 2019). To illustrate this, we present a case study focusing on the augmentation instruction Pronoun replacement: replace pronouns in the text with their corresponding nouns or vice versa, maintaining the
semantic meaning of the sentence. For the sake of brevity, we will use the abbreviation PR to refer to pronoun replacement. We consider two categories of tasks: text entailment (TE) and question answering (QA). As shown in Tab. 8, the results indicate that PR yields suboptimal performance on TE tasks, while it achieves good performance on QA tasks. This discrepancy can be attributed to the inherent characteristics of these tasks. TE tasks heavily rely on capturing the overall semantic meaning and logical relationships within the text, which may not always be preserved when applying pronoun replacement (Gao et al., 2022). In contrast, QA tasks aim to locate and provide specific information relevant to the given question (Joshi et al., 2020). By replacing pronouns with their corresponding nouns, the model can more easily identify the relevant entities and establish a clearer connection between the question and the answer, ultimately benefiting the QA task performance.

heading: D Dataset Collection
text: In Tab. 9, we list all 26 tasks and how we splitting them into training and testing for evaluating the model generalization to unknown downstream tasks. Each task will have 16 training and validation examples but with full test examples. We utilize the code from CrossFit(Ye et al., 2021) to extract and split the training, validation and testing for each task.

heading: E Details of Baseline Methods
text: E.1 Augmentation Methods of Non-LLMDA All of the implementation of Non-LLMDA are from Ma (2019). Here is an elaboration on each of the mentioned Non-LLMDA augmentation methods:
Character-Level Augmentations Random Swap (Belinkov and Bisk, 2018): This involves swapping adjacent characters within words to simulate typos that might occur during typing. For example, "example" might become "exmaple". OCR Replace: Simulating errors commonly introduced by Optical Character Recognition (OCR) software when digitizing text. Characters that look similar, like ’o’ and ’0’ or ’l’ and ’1’, might be substituted for one another. Delete: Randomly removing characters from words to mimic typographical errors or omissions. Insert: Adding extra characters into words at random positions, simulating common typos or spelling errors. Substitute: Replacing characters in words with other characters, not necessarily similar in appearance, to create variations in the text.
Word-Level Augmentations. Swap (Wei and Zou, 2019): Changing the positions of two adjacent words in a sentence to add syntactic variability while largely preserving the sentence’s meaning. Delete: Removing words from sentences randomly to simulate information loss and encourage the model to learn from incomplete data. Spell Error (Coulombe, 2018): Introducing common spelling mistakes into words to mimic human error and increase the model’s exposure to varied spellings. Word2Vector Insert (Morris et al., 2020): Identifying suitable locations in a sentence to insert synonyms or related words based on word embeddings (like word2vec representations), enhancing semantic diversity.
Contextual-Level Augmentations Insert Word using GPT2 (Kumar et al., 2021): Leveraging a pretrained model like GPT2 to generate contextually relevant words to insert into sentences, increasing the complexity and variability of the sentence structures. Substitute Word using BERT (Kumar et al., 2021): Using a model like BERT to identify and replace words with contextually appropriate synonyms or related terms, maintaining the sentence’s overall meaning while altering its surface form. Back-Translation (Fadaee et al., 2017): Translating a sentence into another language and then back into the original language. This process often introduces syntactic and lexical variations, providing a paraphrased version of the original sentence that retains its semantic content.

heading: F Additional Experiment
text: We also compare our method with other data augmentation techniques from Non-LLMDA and Manual-LLMDA. The Non-LLMDA includes EDA (Wei and Zou, 2019) and AEDA (Karimi et al., 2021), while Manual-LLMDA includes GPT3Mix (Yoo et al., 2021) and ZeroGen (Ye et al., 2022). As shown in Tab. 10, our proposed method Self-LLMDA significantly outperforms these baseline methods in the Class→Class and Random→Random settings. However, in the Non-Class→Class setting, Self-LLMDA falls behind GPT3Mix. This may indicate suboptimal transferability of Self-LLMDA in this specific scenario. It is worth noting that GPT3Mix is designed specifically for classification tasks, whereas Self-LLMDA can be applied to a wide range of text-related tasks, demonstrating its versatility and broader applicability.
F.1 Augmentation Instructions of Manual-LLMDA
In Tab. 11, we will represent the manually crafted augmentation instructions. The format of these augmentation instructions is “<method name>: <method instruction>”.

heading: G Self Instructions Generation
text: The instructions automatically generated by LLM is shown in Tab. 12 and Tab. 13.

heading: Future Work
text: This study acknowledges several constraints that delineate the scope of our current work and outline directions for future research:
• Evaluation on a Limited Range of LLMs: Our experiments were conducted primarily with GPT 3. 5 Turbo due to the high costs associated with using OpenAI models. While promising results in Tab. 1 suggest that our proposed Self-LLMDA method could potentially perform even better on more advanced models like GPT 4 Turbo, comprehensive testing was not feasible. Similarly, the computational demands of evaluating open-source LLMs such as LLAMA-70b-chat (Touvron et al. , 2023), coupled with the extensive number of tasks in our study, exceeded our resources. Despite these limitations, we are optimistic that Self-LLMDA would exhibit enhanced performance across a broader spectrum of LLMs. • Meta-Prompting Exploration: Within the Self-LLMDA framework, we employed one meta-prompt to guide the LLM in generating diverse and relevant augmentation instructions. However, our exploration of meta-prompting techniques was limited. We acknowledge that more sophisticated prompt engineering could further refine the quality and effectiveness of generated instructions. Investigating more advanced meta-prompting strategies remains an area for future exploration. • Analysis of Ensemble Augmentation Methods: Our research did not investigate the potential benefits of combining multiple sets of augmented data (e. g. , D∪D′1 ∪D′2). Such ensemble approaches introduce additional complexities, such as determining the optimal number of augmentation instructions to include. While we hypothesize that ensemble augmentation could improve model performance, this aspect falls outside the current study’s scope and is earmarked for subsequent investigation