LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024)

Abstract: Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the lowdata regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https: //github.com/SqueezeAILab/LLM2LLM.

heading: 1 Introduction
text: Pretrained large language models (LLMs) have achieved impressive performance on various benchmarks and datasets that have previously required
*Equal contribution
specialized neural network architectures. For many of these general benchmarks (Hendrycks et al., 2020; Zhong et al., 2023), LLMs are prompted with custom instructions or in-context examples.
However, in various real-world applications, these prompting strategies are not a one-size-fitsall solution. For instance, LLMs have a limit on the amount of input context they can process, thus limiting the number of in-context examples or instructions we can input to make the LLM follow a certain behavior. For simple tasks that are closely aligned with the data that the LLM was pretrained on, extensive prompting may not be necessary. However, applying LLMs to specialized domains (e.g., a specific medical field (Nori et al., 2023) or private data with niche protocols) can be more challenging, often requiring prohibitively long prompts to achieve adequate performance. Even if the prompt length does not exceed the limit, processing long prompts increases the latency and cost of each inference. Additionally, LLMs also tend to forget or ignore information in long contexts (Liu et al., 2023b), leading to potential accuracy drops even when the model can handle long input prompts. While Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) has been developed to address some of these challenges, it may sometimes retrieve irrelevant passages or documents, which can potentially degrade the generation performance. Furthermore, RAG does not necessarily solve the latency and cost issue as processing a long input prompt may still be required.
A promising method for addressing this is finetuning. With the emergence of Parameter Efficient Fine-tuning (PEFT) (Hu et al., 2021; Mangrulkar et al., 2022), the computational resources required to fine-tune a task-specific LLM have decreased significantly. However, herein lies a new problem: successful fine-tuning requires enough training data. This can be challenging for some applications, where we only have access to a small amount
6498
of task-specific data. Often, collecting, cleaning, and labeling additional data can be costly and timeconsuming. So the key question is: how should we increase the user’s training data to be enough for fine-tuning?
Data augmentation is a known method that could help effectively expand the training dataset. For natural language processing (NLP) tasks, one can use approaches such as synonym replacement, character replacement (e.g., by intentionally introducing spelling errors), random swapping, and back translation, just to name a few (Wei and Zou, 2019; Belinkov and Bisk, 2017; Coulombe, 2018; Zhang et al., 2018). However, these approaches fail to effectively expand the training data for fine-tuning LLMs in the case of new and specialized tasks, as we will show later in Section 4.3.
To address this, several recent papers have explored using an LLM to expand the fine-tuning dataset (Dai et al., 2023; Kumar et al., 2020; Zhou et al., 2023; Chen et al., 2023; Cao et al., 2023; Wei et al., 2023; Zhu et al., 2023). This approach has proven to be more effective than traditional data augmentation methods. However, these approaches often apply LLM-based data augmentation on all of the available training dataset, without considering the LLM’s prediction accuracy on individual training data points. We have observed that for various reasoning tasks such as arithmetic and reading comprehension, the LLM correctly solves simpler examples in the fine-tuning dataset, but may struggle with harder examples. It will be sub-optimal to keep augmenting data points for which the LLM is already achieving high accuracy on.
To address these challenges, we introduce LLM2LLM, a new targeted and iterative data augmentation framework that uses a teacher LLM to expand the training dataset, with a targeted and iterative approach. In more detail, we make the following contributions:
• We propose LLM2LLM, a targeted and iterative LLM-based data augmentation technique that efficiently and effectively augments small task-specific datasets. LLM2LLM achieves this by (1) fine-tuning a student LLM on the initial dataset, (2) evaluating on the training data and extracting data points which the model got incorrect after training, and (3) using a Self-Instruct (Wang et al., 2023) style data augmentation to augment these data points, which are then added back into the training data (Section 3.1).
• We benchmark LLM2LLM on randomly sampled subsets of GSM8K (Cobbe et al., 2021), CaseHOLD (Zheng et al., 2021), SNIPS (Coucke et al., 2018), TREC (Li and Roth, 2002) and SST-2 (Socher et al., 2013) in order to evaluate the effectiveness of our approach in the low-data regime (Section 4.2). Here, we get up to a 24.2% improvement on GSM8K, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC, and 39.8% on SST-2 (Table 1).
• We conduct a series of ablations studies comparing LLM2LLM to several existing baselines as well as to variants of LLM2LLM to evaluate the effectiveness of our design decisions (Section 4.5). We observe that both the iterative and targeted nature of LLM2LLM are critical to improving model performance.

heading: 2 Background and Related Work

heading: 2.1 Instruction Following LLMs
text: The earliest works (Wei et al., 2021; Longpre et al., 2023; Chung et al., 2022; Aribandi et al., 2021; Sanh et al., 2021; Muennighoff et al., 2023; Wang et al., 2022b; Mishra et al., 2022; Wang et al., 2022a; Xu et al., 2022) in instruction fine-tuning involved gathering and processing different existing NLP datasets in order to improve the performance of LLMs on a wide range of tasks. SelfInstruct (Wang et al., 2023) removed the reliance on existing datasets by introducing a framework for bootstrapping instruction datasets with the outputs of the model itself. Follow-up work (Ouyang et al., 2022; Taori et al., 2023; Geng et al., 2023; Chiang et al., 2023; Xu et al., 2023; Mukherjee et al., 2023; Mitra et al., 2023; Kang et al., 2023; Nori et al., 2023) took advantage of stronger models (Achiam et al., 2023; Touvron et al., 2023a,b) in order to fine-tune stronger general-purpose instructionfollowing models.

heading: 2.2 Self-Improving LLMs
text: Various early works (Zelikman et al., 2023; Haluptzok et al., 2023; Zelikman et al., 2022; Madaan et al., 2023; Gulcehre et al., 2023; Singh et al., 2023) explore using self-improvement for finetuning LLMs. These works generally filtered the outputs of the model before fine-tuning it on its own outputs. LLM2LLM differs from these methods, as we do not directly fine-tune on the outputs of our own model, and we employ a teacher model to provide feedback in the form of synthetic data.
Concurrent with our work, several papers have been recently published that use an iterative approach to improving LLMs (Chen et al., 2024; Anil et al., 2023; Burns et al., 2023; Li et al., 2023b; Yuan et al., 2024). These works combine ideas from Reinforcement Learning (RL) and SelfPlay (Samuel, 2000; Tesauro et al., 1995) in order to iteratively build stronger LLMs by fine-tuning on the outputs of the model itself. LLM2LLM is distinguished by how it focuses on the lowdata regime for task-specific fine-tuning of LLMs whereas others are attempting to create stronger general-purpose LLMs. In addition, our technique exclusively uses the data points that the model got incorrect during training, and it uses the teacher model to augment these data points. Instead of providing feedback in the form of a critique or rationale, the teacher model’s feedback is only in the form of synthetic data points, which simplifies the training pipeline.

heading: 2.3 Data Augmentation
text: Data augmentation has been long studied in NLP. Early work augmented at the character (Belinkov and Bisk, 2017; Coulombe, 2018) and word (Wei and Zou, 2019) level. Notably, Easy Data Augmentation (EDA) (Wei and Zou, 2019) was a popular early method that used word level augmentations:
synonym replacement, random insertion, swap, and deletion to augment data for text classification. We refer the reader to (Feng et al., 2021) for a more complete summary of data augmentation in NLP.
A popular new approach is to use LLMs themselves to synthesize new training data (Deng et al., 2023a; Prasad et al., 2023; Fu et al., 2023b; Dai et al., 2023; Ubani et al., 2023; Fang et al., 2023; Liu et al., 2023a; Yu et al., 2023; Kumar et al., 2020; Yoo et al., 2021; Wang et al., 2021; Ding et al., 2023; Li et al., 2023a; Liang et al., 2023). A noteworthy example is AugGPT (Dai et al., 2023), which used ChatGPT to rephrase text to augment text classification tasks.
Many of these techniques generate very large amounts of synthetic data. Recent work (Chen et al., 2023; Cao et al., 2023; Wei et al., 2023; Zhou et al., 2023) found that one could replicate the results of fine-tuning on these large datasets with significantly smaller subsets.

heading: 3 Methodology
text: We assume that we are given an LLM model M (e.g., GPT-3.5 or Llama-2-7B) that is pre-trained on some source dataset (e.g., Common Crawl). The goal is to adapt M (hereon called the student model) to a new target domain by using a small
Algorithm 1 LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement. Given a seed dataset D0, we finetune the model Mistudent, evaluate, and extract training set data points that the model gets wrong. These are used to generate new training data points using the teacher model Mteacher for the next step. 1: procedure LLM2LLM(M0student,Mteacher, D0) 2: i← 0 3: while i < n do 4: Mistudent ← Finetune(M0student, Di) 5: Ei ← Evaluate(Mistudent, D0) ▷ Evaluate on seed data 6: W i ← Filter(Ei,D0) ▷ Keep wrong answers 7: Ai ← Generate(Mteacher, W i) ▷ Augment using teacher 8: Di+1 ←Di +Ai ▷ Append to data 9: i← i+ 1 10: end while 11: Evaluate M∗student 12: end procedure
seed dataset D, where D potentially has unseen characteristics, compared to the pre-trained dataset (e.g., a medical dataset with specific terminology, or a private database with specific characteristics). In this case, the model’s zero-shot or fine-tuned performance is likely to be unsatisfactory. While strategies to address this challenge have been explored, e.g., through enhanced few-shot learning methods as discussed in Section 2, here we strictly focus on enriching the provided target dataset D with an LLM. This method is orthogonal to the aforementioned techniques, offering a complementary solution that can be applied alongside them.
To enrich D, AugGPT (Dai et al., 2023) has introduced a promising approach that generates additional augmented data by applying a prompted LLM to all available data points in the target training dataset. However, this method falls short by indiscriminately augmenting data without considering the student model’s varying performance across different data points. For instance, the model may easily solve the majority of the dataset, but it may struggle with a small subset of more challenging examples. In this case, rather than indiscriminately expanding the dataset by replicating simpler cases, a better augmentation strategy would be to generate more data points that align conceptually with these challenging examples. This is because the former approach could lead to longer training time without noticeable performance improvement.
Here, we propose a more general formulation of an LLM-based data augmentation pipeline that addresses the aforementioned limitation. To do so,
we consider the following iterative process:
Dn+1 = f(Mteacher,Mstudent, Dn, · · · , D0). (1)
In Equation (1), Mteacher is the teacher model, Mstudent is the student model (potentially being fine-tuned in many iterations), n refers to the nth step of data augmentation, Dn+1 is the new training dataset at the next iteration, and f is the datageneration algorithm. At each step, the teacher model has access to how the student model performs at the nth step (e.g., correct/incorrect labels, or possibly prediction distributions for white-box models), and based on that it can edit training data points for the next iteration.
Note that LLM2LLM is different from knowledge distillation (Hinton et al., 2015). Knowledge distillation is generally applicable to cases where the teacher model has high accuracy on the target data. In contrast, in this case, it is possible that the teacher model also performs sub-optimally on the target data (e.g., in the private database case, where the teacher lacks domain-specific knowledge). However, if the teacher model has enough reasoning capability to produce conceptually similar but semantically different examples when it is given both the prompt and answer, then our framework can improve performance.
In LLM2LLM, we consider a specific instantiation of Equation (1), as discussed next.

heading: 3.1 LLM2LLM
text: The end-to-end algorithm of LLM2LLM is presented in Algorithm 1. Inspired by Self-Instruct (Wang et al., 2023), we use the teacher model Mteacher to generate synthetic data from the data points that the model got incorrect during training in order to target these deficiencies in the student model. In more detail, we first train the baseline student model Mstudent on the provided target data D0, and we evaluate its performance (lines 4-5 of Algorithm 1). We then filter the results and keep the incorrect training examples that the student model struggled to answer correctly (Ei in line 6). Then the teacher model is prompted to create additional training data points that are conceptually aligned but semantically different (line 7, see Section B.4 for specifics on the prompt). The teacher model does not necessarily need to be bigger, although that could potentially improve performance. The primary requirement for the teacher model is to have reasoning capability to be able to follow the
data augmentation instruction, and the ability to create data points similar to the incorrect examples. This process is schematically illustrated in Figure 1.
A subtle but important design decision in LLM2LLM is that we only use examples from the seed data when prompting the teacher model to generate additional data points. This is similar to Alpaca (Taori et al., 2023), but unlike EvolInstruct (Xu et al., 2023). There are two main reasons for this. First, our approach prevents data degradation from multiple augmentation iterations. Early experiments revealed that while the teacher model could generate high-quality augmentations, some examples contained logical errors. Therefore,
1The 0% here is because our prompting does not include the formatting that we used to extract the answer with. Adding some instructions to make sure the output format is correct results in a 0.30% test accuracy, which is in line with the rest of the results in Table 1.
2SST-2 has no test set, therefore we evaluate on the dev set instead, see Section A.3.
further augmentation applied to these examples could potentially propagate the error, degrading the quality of the dataset over time. This is highlighted in our ablation studies in Table 4, where using both seed and synthetic data for data augmentation leads to an accuracy drop.
Second, this approach limits the amount of new data being generated overall. Suppose that the original seed dataset is of size n, and at each iteration, the student model gets pi proportion of the training dataset Di wrong, where 0 < pi < 1. If we include the augmented data into the seed data for data generation, then the size of the dataset Dj at step j will be
|Dj | = n j∏
i=0
(1 + pi) ≥ n(1 + pmin)j .
This has a lower bound that grows exponentially with each step. Limiting the input wrong answers
W i during dataset generation to only data from the original seed data allows us to bound the total number of training data points to
|Dj | = n+ j∑
i=0
npi ≤ n(1 + jpmax),
which has an upper-bound that grows linearly with the number of steps. The empirical evaluations shown in Section 4.5.2 (Table 4) corroborates this.

heading: 4 Results

heading: 4.1 Experimental Setup
text: To evaluate the performance of LLM2LLM, we applied our framework to fine-tune Llama-2-7B on various target datasets, including GSM8K (Cobbe et al., 2021), CaseHOLD (Nori et al., 2023), SNIPS (Coucke et al., 2018), TREC (Li and Roth, 2002) and SST-2 (Socher et al., 2013). We subsampled these datasets with different sampling rates from 0.02% to 50% to evaluate performance across different low-data regimes. Our teacher model for these results is GPT-3.5 (1106 release) unless otherwise specified. We considered several other teacher models, including GPT-3.5, GPT-4-Turbo, Llama2-70B (Touvron et al., 2023b), and Airoboros-L270B (Durbin, 2023) in Section 4.4. We include a more detailed experimental setup in Section A. Additionally, we conducted additional experiments (Section B.6) to ensure that the augmented data from the teacher models differs from the test dataset
used to evaluate final model accuracy. This addresses the issue of potential test data leakage that could have happened if the teacher model had been trained on similar data.

heading: 4.2 Main Results
text: Here, we discuss LLM2LLM’s performance with varying amount of training data by presenting results for fine-tuning Llama-2-7B on GSM8K using GPT-3.5 as the teacher model. We then discuss how these trends extend to different datasets (Table 1).
The final model accuracy after applying 10 iterations of LLM2LLM is given in Table 1. For a low-data regime with 74 available examples (i.e., 1% of the GSM8K training dataset), vanilla finetuning achieves only 0.99% test accuracy. However, LLM2LLM boosts the accuracy to 19.56% by iteratively generating 391 additional examples based on data points where the model makes mistakes. With slightly more available data of 149 seed examples (i.e., 2% of the training dataset) we can achieve 25.70% accuracy. As shown in the baseline accuracy with 20% data in Table 2, we would need over 10× more training data points to match this accuracy if we only rely on vanilla fine-tuning. We also highlight that LLM2LLM can lead to noticeable gains with data-sufficient regimes (e.g., 100% data), albeit at a smaller improvement over the baseline compared to lower-data regimes.
We observe a similar trend for CaseHOLD, SNIPS, TREC, and SST-2, where LLM2LLM helps improve performance in the low-data regime.
Interestingly, LLM2LLM generally generates proportionally more augmented data for GSM8K than other datasets. This is because the baseline accuracy is lower for GSM8K overall, suggesting that it is a more difficult dataset compared to the others. However, in all cases, we find that LLM2LLM helps recover a high-performing model.
In Figure 2, we also illustrate how the baseline accuracy improves on GSM8K and CaseHOLD with each iteration of applying LLM2LLM. We can observe a rapid increase in test accuracy in the first few iterations of LLM2LLM, especially in lower-data regimes.

heading: 4.3 Comparison with Other Augmentation Methods
text: In Table 2, we compare our method against other augmentation techniques, including EDA (Wei and
Zou, 2019) and AugGPT (Dai et al., 2023). We also compare against adding more data from the unseen training set. The details of all augmentation methods we used in our comparison are provided in Section A.3.
On GSM8K, LLM2LLM outperforms naive fine-tuning by over 20%, EDA by over 8%, and AugGPT by over 5%. Similarly, on CaseHOLD, LLM2LLM outperforms the fine-tuning baseline by approximately 35%, EDA by 2.3%, and AugGPT by 1.1%. These improvements, particularly in comparison to AugGPT, can be attributed to LLM2LLM’s capability to generate more targeted examples based on where the model struggles, as opposed to AugGPT which augments data indiscriminately. This allows for more effective and targeted use of the augmented data budget.

heading: 4.4 Choice of Teacher Model
text: Thus far, we have illustrated LLM2LLM’s performance with GPT-3.5 as the teacher model, but other LLMs can serve this role as well. A stronger teacher model is expected to yield higher-quality augmentation and, consequently, higher accuracy. Table A.1 demonstrates the LLM2LLM’s accuracy with GPT-4-Turbo, Llama-2-70B, and AiroborosL2-70B as the teacher model on GSM8K. With 74 seed data examples, LLM2LLM only achieves 11.8% accuracy with Llama-2-70B, which can be contrasted with 15.0% with Airoboros and 19.8% with GPT-4-Turbo. This aligns with our expectation, as GPT-4-Turbo’s mathematical reasoning is known to be better than the other models, being generally on par with that of GPT-4 (Fu et al., 2023a; Deng et al., 2023b). The qualitative analysis of augmented data using different models (Figure B.16) further supports this, showing that Llama and Airoboros models produce less varied data than GPT-3.5 or GPT-4-Turbo.

heading: 4.5 Ablation Studies
text: Here, we provide ablation studies to justify the design decisions we made in LLM2LLM.

heading: 4.5.1 Iterative Augmentation vs One-Shot Augmentation
text: We first evaluate the efficacy of iterative augmentation versus adding all augmented data at once. To evaluate this, we compare the final accuracy achieved by augmenting data over 10 iterations against adding the equivalent amount of data in
a single iteration, for both the GSM8K and CaseHOLD datasets. As shown in Table 3, using a single augmentation step with a larger amount of augmented data significantly underperforms the alternative of executing 10 iterative steps of LLM2LLM with a smaller number of augmentations per iteration. In particular, on GSM8K, augmenting one data point per example over 10 steps yields a 7.4% higher accuracy than augmenting five data points per example in a single step. Similarly, on CaseHOLD, iterative augmentation of one data points per example over 10 steps results in a 4.6% improvement over a one-shot augmentation with four data points per example. This justifies the LLM2LLM’s iterative augmentation approach that generates one data point per each incorrectly answered example.

heading: 4.5.2 Data Augmentation with Seed Data vs Augmented Data
text: In each iteration, LLM2LLM evaluates the student model’s performance only on the original seed dataset and generates augmented data from incorrect seed examples. However, a possible alternative is performing evaluation and data augmentation using both seed and previously augmented data. The latter often leads to sub-optimal performance as
well as excessive amounts of total augmented data points, as we demonstrate in Table 4. On GSM8K, generating augmented data from the previous iteration’s augmented data yields 18.3% accuracy, while using the seed data for further augmentation improves the accuracy to 23.75%. We observe a similar trend for CaseHOLD. As discussed in Section 3.1, a potential reason for the performance drop, when using augmented data for further augmentation, has to do with a deviation from the original data distribution.

heading: 4.5.3 From-scratch Fine-tuning vs Continuous Fine-tuning
text: Another key decision for LLM2LLM is whether to continue fine-tuning from the last iteration’s checkpoint (i.e. continuous fine-tuning) or to restart finetuning from the pre-trained model at each iteration (i.e. from-scratch fine-tuning). Considering the non-convex nature of the optimization target and complex loss landscapes, this decision is not necessarily obvious. Nevertheless, as shown in Table 5, we observe that from-scratch fine-tuning consistently and significantly outperforms continuous fine-tuning, with up to 9% accuracy improvement. The inferior performance of continuous fine-tuning can be attributed to a potential overfitting to small seed data over multiple iterations of fine-tuning, especially in lower-data regimes where the seed data is small. This can be alleviated by restarting fine-tuning from scratch in each iteration with sufficient augmented data appended to the seed data to form the training dataset.

heading: 5 Conclusion
text: We have introduced LLM2LLM, an adaptive and iterative LLM-based data augmentation framework that uses LLMs to scale up smaller fine-tuning datasets in lieu of manually generating more data. This framework substantially reduces the amount
of real data needed, and it allows us to efficiently scale the dataset with synthetic data that can match or even exceed the effect of hand-collecting more data. The method is effective because of the iterative and targeted nature of the process, which allows us to boost the signal from data points that the LLM gets wrong. As a result, we were able to achieve a 24.2% improvement on GSM8K, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC, and 39.8% on the SST-2 dataset in the low-data regime using a Llama-2-7B student model. Future work can focus on tuning the hyperparameters of our framework as well as incorporating our approach with other LLM techniques such as prompt tuning and few-shot learning.

heading: Limitations
text: Our results primarily reflect improvements that occur in a low training data regime, from tens of examples to a couple thousand. However, practitioners may deal with larger datasets from time to time, in which our method may be out of scope.
Furthermore, there could be other factors that help explain the disparity in performance between different teacher models. Also, we have analyzed the generated data for differences in quality, but there may be other ways to close the gap between open-source models and the GPT models as a teacher model. This warrants further investigation.
Our focus primarily reflects a specific use case where there is low training data available due to difficulty in data collection such as labor or resource constraints. Exploring the effects of using synthetic data to further eke out performance when there is abundant data is a promising research direction.

heading: Ethics Statement
text: LLM2LLM relies on using LLMs to augment a training dataset in order to train another student LLM more efficiently. This can reduce the energy and monetary cost of experimentation and machine learning research, as it enables those with smaller datasets to achieve better performance on a domainspecific task. Of course, misuse of this method may lead to unethical data being generated, which can lead to societal harm. This is not a concern specific to this work, but to LLM research in general. Furthermore, there are still open questions about latent implicit biases and ethical issues surrounding the generated output of LLMs that the authors and prac-
titioners of this method are aware of and continue to consider throughout the whole process.

heading: Acknowledgements
text: We appreciate the valuable feedback from Andrew Aikawa. We acknowledge gracious support from Furiosa team. We also appreciate the support from Microsoft through their Accelerating Foundation Model Research, including great support from Sean Kuno. Furthermore, we appreciate support from Google Cloud, the Google TRC team, and specifically Jonathan Caton, and Prof. David Patterson. Prof. Keutzer’s lab is sponsored by the Intel corporation, Intel One-API, Intel VLAB team, the Intel One-API center of excellence, Apple, Samsung, Panasonic, as well as funding through BDD and BAIR. We appreciate great feedback and support from Ellick Chan, Saurabh Tangri, Andres Rodriguez, and Kittur Ganesh. Sehoon Kim would like to acknowledge the support from the Korea Foundation for Advanced Studies (KFAS). Amir Gholami was supported through funding from Samsung SAIT. Michael W. Mahoney would also like to acknowledge a J. P. Morgan Chase Faculty Research Award as well as the DOE, NSF, and ONR. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.

heading: A Experimental Setup

heading: A.1 Datasets
text: We evaluate LLM2LLM on five different datasets that are either multiple-choice or classification tasks and were widely adopted in prior works including (Ubani et al., 2023). Our datasets are as follows:
1. GSM8K: a grade school math work multiplechoice dataset that consists of 7.5K train problems and 1K test problems (Cobbe et al., 2021). Figure B.1 shows an example.
2. CaseHOLD: a multiple-choice law dataset that requires one to choose the relevant holding (i.e. the court’s determination) of a cited case which backs up the proceeding argument (Zheng et al., 2021). Figure B.2 shows an example.
3. SNIPS: a 7-way classification dataset to determine the correct user intent for a voice assistant. (Coucke et al., 2018). Figure B.3 shows an example.
4. TREC: a 6-way classification dataset where one must classify the type of text into a category e.g., abbreviation, location, or numeric value (Li and Roth, 2002). Figure B.4 shows an example.
5. SST-2: a binary classification dataset to decide whether a sentence of positive or negative sentiment (Socher et al., 2013). Figure B.5 shows an example.
For each dataset, we sample between 0.02% to 50% of the total training data and use this as the seed data for each experiment. This allows us to measure how effectively LLM2LLM scales up small task-specific datasets. For consistency, we use identical samples of seed data across different experiments (e.g. 1% on GSM8K) to avoid introducing new randomness with different samples.
In particular, for SNIPS, TREC, and SST-2, we always uniformly sample the same number of examples per class, similar to (Dai et al., 2023; Ubani et al., 2023). In Table 1, we sample 10, 15, and 20 examples per class to measure the efficacy of LLM2LLM in the extreme low-data regime. These three tasks are relatively simpler than GSM8K and CaseHOLD, and therefore using an extremely small amount of training data is sufficient to achieve exemplary performance.

heading: A.2 Models
text: For all of our experiments, we use Llama-2-7B (Touvron et al., 2023b) as the student model. We perform minimal prompt tuning for each task, only formatting the data as necessary and not employing many few-shot examples, which can be seen in Figure B.1 and Figure B.2. Using excessive prompting would undermine the benefits of fine-tuning and would muddle the evaluation of the effectiveness of LLM2LLM. Our fine-tuning settings are described in Section A.4.
For our main experiments, we use GPT-3.5 (1106 release) as the teacher model for data generation. In Section 4.4 and Table A.1, we show that our framework can be extended to different teacher models such as the more powerful GPT-4-Turbo (1106 release) model as well as open source LLMs such as Llama-2-70B-chat (Touvron et al., 2023b) and Airoboros-l2-70b-3.1.2 (Durbin, 2023).

heading: A.3 Baselines and Evaluation
text: To measure the efficacy of LLM2LLM, we finetune the student model using samples of different sizes from each dataset and evaluate on the validation sets of each of these datasets. We then run 10 steps of LLM2LLM, and use the validation sets to select the best performing model. In Section 4.2, we compare these results against basic fine-tuning on just the seed data.
For GSM8K and TREC, since there is no development set, we choose the best checkpoint’s test set results to be representative for the overall improvement. Similarly, for SST-2, since the test set labels are not public, we use the development set results. For all other datasets, we record the test set performance of the checkpoint that performs best on the development set.
For TREC, SST-2, and SNIPS, since these are simple classification tasks, we perform string matching between the generated output and the ground truth after some cleanup. For CaseHOLD, which is a multiple choice task, we extract the letter of the answer that the model generates. For GSM8K, we use a regular expression extraction based on the answer format that GSM8K provides. Specifically, we extract the number after the #### token.
In Section 4.3, we only sample 100 examples from GSM8K and CaseHOLD. For SNIPS, TREC, and SST-2, we sample 10 examples per class. We
run an extensive ablation study against several different baselines:
• Fine-tuning: Standard fine-tuning on the initial seed data.
• EDA: EDA (Wei and Zou, 2019) which uses synonym replacement plus random swap, insert, and deletion to augment text with 10% probability. Note that we take care not to augment any special formatting or structural elements for each dataset.
• AugGPT: Augment our seed data using the teacher prompts from Section 3.1 with no filtering, similar to (Dai et al., 2023; Ubani et al., 2023; Yoo et al., 2021).
• More Data: Randomly sampling new data from unseen train data and adding to the training data.

heading: A.4 Fine-tuning Settings
text: Following the example from Alpaca (Taori et al., 2023), all models are fine-tuned with a maximum sequence length of 1024 for 3 epochs with a batch size of 128 examples. We use a learning rate of 2 × 10−5 with 0 weight decay and a warmup ratio of 0.03 with a cosine learning rate scheduler. These models were trained using either 4 NVIDIA A100-80GB or 8 NVIDIA A6000s. We do full finetuning for simplicity and to reduce the complexity of our experiments, but in practice, one can also use some form of parameter-efficient fine-tuning method such as LoRA (Hu et al., 2021).

heading: B LLM2LLM Details

heading: B.1 Fine-tuning
text: The fine-tuning step trains a small student model using seed data and previously generated LLM2LLM data, if any. This LLM2LLM data is generated by a process further detailed in Section 3.1 that generates synthetic data targeted toward data points the model got wrong. We always fine-tune the original student model on the full dataset (seed data + LLM2LLM data) at each step. Our ablation study in Section 4.5.3 shows that finetuning the original baseline model from scratch on the full dataset always outperformed re-using the already-fine-tuned model. We hypothesize that this is because fine-tuned models have already seen most of the data, causing them to overfit and fail to converge to a better optimum.

heading: B.2 Evaluation
text: After fine-tuning, the model needs to be evaluated on the original (training) seed data to identify the examples that the model gets wrong. This allows the LLM2LLM framework to use those failed examples to generate targeted synthetic data for the model to train on. For example, if the model was unable to solve problems involving the Pythagorean theorem as in Figure 1, these examples will be used to generate more problems with this concept.
For many datasets, this evaluation step can be extremely costly, as traditional NLP datasets can have more than thousands of data points. However, this evaluation step is cost-effective and relatively quick in the low-data regime where the seed dataset size is small, thereby not slowing down the LLM2LLM process.

heading: B.3 Filtering Generated Dataset
text: Once the teacher model generates the synthetic data, we need to apply simple filtering of the output for quality insurance. Like in previous work (Wang et al., 2023; Taori et al., 2023), we use regex filters to ensure that the basic format of the output is aligned with our expectations. We also use a ROUGE (Lin, 2004) filter in order to enforce that the augmented data points are sufficiently different from previous samples. However, we use a weaker ROUGE filter of 0.95 to filter out similar instructions, rather than the score of 0.85 used in other works like Alpaca (Taori et al., 2023) and Selfinstruct (Wang et al., 2023). The reason we can do this is because unlike Alpaca and Self-Instruct, we do not require as much diversity in the generations, as we are not targeting general-purpose instructionfollowing. In fact, we would like to constrain the generated data points to the task and domain of the datasets as much as we can. Thus, we are able to use a weaker filter so that we can simply filter out exact matches during generation. In addition, since we are augmenting each sample individually, there is already enough inherent diversity in the generation process.

heading: B.4 Prompting Details
text: We devised simple but thorough prompts for each task that the teacher model uses while augmenting the dataset. Previous work in open-domain dataset generation such as (Wang et al., 2023; Mukherjee et al., 2023; Xu et al., 2023) used generic system
Dataset # Seed Teacher Total # Aug Accuracy (%)
GSM8K
74 (1%) Llama-2-70B 333 11.83 Airoboros 345 15.01 GPT-3.5 391 19.56
GPT-4-Turbo 388 19.79
149 (2%) Llama-2-70B 661 17.59 Airoboros 671 19.33 GPT-3.5 802 25.70
GPT-4-Turbo 805 25.78
343 (5%) Llama-2-70B 1308 19.33 Airoboros 1286 21.76 GPT-3.5 1641 27.07
GPT-4-Turbo 1739 28.43
Table A.1: Experiments on how the quality of teacher model affects the performance of LLM2LLM. For each of these experiments, we only change the teacher model to measure the effect of the teacher model on the final outcome.
instructions for generating new data points from stronger models such as GPT-3.5 and GPT-4. This was necessary as these approaches targeted improving the LLM over a wide range of different tasks. However, for LLM2LLM, we are trying to improve the LLM at domain-specific tasks. Thus, for each task, the system prompt that we give to the teacher-model differs on a per-task basis. This allows the user to inject and leverage domain-specific knowledge about the task into the dataset generation procedure, creating higher quality fine-tuning data. In practice, we also use in-context learning with few-shot prompting to bootstrap the teacher model’s ability to generate relevant questions.
The detailed system prompt and in-context examples for each dataset are provided below:
1. GSM8K: System (Figure B.6) and In-Context Examples (Figure B.7)
2. CaseHOLD: System (Figure B.8) and InContext Examples (Figure B.9)
3. SNIPS: System (Figure B.10) and In-Context Examples (Figure B.11)
4. TREC: System (Figure B.12) and In-Context Examples (Figure B.13)
5. SST-2: System (Figure B.14) and In-Context Examples (Figure B.15)

heading: B.5 Training and Data Generation Costs
text: In Table B.2, we report the training and data generation costs to perform LLM2LLM. This includes the cost of generating new data from OpenAI as well as the amount of GPU hours required to train and evaluate the student models. We measured

heading: Dataset % Data Cost ($) Time (Hours)
text: these numbers using 4xA100-80GB PCIe based NVIDIA GPUs. As we can see, generating data for LLM2LLM costs relatively little compared to the cost of collecting new data points manually. Furthermore, the process of fine-tuning the student model also finishes in a reasonable amount of time.

heading: B.6 Decontamination Experiments
text: When using an LLM to generate data, there are potential concerns of data contamination i.e., when
the teacher LLM has been trained on the test data and thus leaks it into the student model’s training data which would artificially inflate the student model’s scores. To test for this we measure the word overlap percentage between examples generated by the teacher model and examples from the train and test set for each dataset in the same manner as (Ubani et al., 2023).
Word overlap is computed by first removing stop words and punctuation from each example. Then, for each example pair, we count the number of words that are common between the two, and divide by the number of words in longer example. This provides a metric for measuring how many words are similar between the two examples.
For each dataset, we run LLM2LLM for 10 steps using the same amount of seed data as in Section 4.3. Then, we take all the synthetic data generated for the final step and calculate our word overlap metric per example. We then calculate word overlap summary statistics similarly to (Ubani et al., 2023) which are the percentage of examples above 66% similarity, the maximum overlap, and the number of examples at the maximum overlap.
Looking at Table B.3, we can see varying levels of overlap between the synthetic data and the training data. This is to be expected, as LLM2LLM uses training data as seed data in order to generate new synthetic examples based on that training data.
Regarding the test/dev set results, we see that for GSM8K, CaseHOLD, and SST-2, 0% of the generated examples have above 66% word overlap with the test set. This indicates that the model is not leaking test set data into the student model’s training data. For SNIPS and TREC, we see that the percent of examples with above 66% overlap is still well below 10% thus indicating no large-scale overlap.
For SNIPS and TREC specifically the maximum overlap percentage is 80%. Upon closer inspection this is because these two datasets have very short examples. Given the small number of such examples we don’t believe this poses a high risk of data contamination. (Ubani et al., 2023) came to a similar conclusion regarding these two datasets.

heading: CaseHOLD Example:

heading: GSM8K Example:

heading: SST-2 Example:

heading: TREC Example:

heading: SNIPS Example:

heading: System:

heading: Assistant:

heading: User:

heading: E. [OPTION 5]

heading: D. [OPTION 4]

heading: C. [OPTION 3]

heading: B. [OPTION 2]

heading: A. [OPTION 1]

heading: System:

heading: Assistant:

heading: User:

heading: System:

heading: Assistant:

heading: User:

heading: System:

heading: Assistant:

heading: User:

heading: Assistant:

heading: User:

heading: Future Work
text: Future work can focus on tuning the hyperparameters of our framework as well as incorporating our approach with other LLM techniques such as prompt tuning and few-shot learning