LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs
Authors: Chansung Park, Juyong Jiang, Fan Wang, Sayak Paul, Jing Tang
Accepted at: acl25

LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs

Chansung Park*◊, Juyong Jiang*◊, Fan Wang*◊, Sayak Paul♣, Jing Tang†◊♣
◊Electronics and Telecommunications Research Institute
◊The Hong Kong University of Science and Technology (Guangzhou)
♣The Hong Kong University of Science and Technology
♣Hugging Face
{deep.diver.csp,csjuyongjiang,csfanwang,spsayakpaul}@gmail.com
jingtang@ust.hk

Abstract

The widespread adoption of cloud-based proprietary large language models (LLMs) has introduced significant challenges, including operational dependencies, privacy concerns, and the necessity of continuous internet connectivity. In this work, we introduce an LLMOps pipeline, “LlamaDuo”, for the seamless migration of knowledge and abilities from service-oriented LLMs to smaller, locally manageable models. This pipeline is crucial for ensuring service continuity in the presence of operational failures, strict privacy policies, or offline requirements. Our LlamaDuo involves fine-tuning a small language model against the service LLM using a synthetic dataset generated by the latter. If the performance of the fine-tuned model falls short of expectations, it is automatically improved through additional fine-tuning using extra similar data generated by the service LLM. This multi-turn process guarantees that the smaller model can eventually match or even surpass the service LLM’s capabilities in specific downstream tasks, offering a practical and scalable solution for managing AI deployments in constrained environments. Extensive experiments with leading-edge LLMs are conducted to demonstrate the effectiveness, adaptability, and affordability of LlamaDuo across various downstream tasks. Our pipeline implementation is available at https://github.com/deep-diver/llamaduo.

1 Introduction

The emergence of LLMs has significantly transformed a myriad of tasks and domains (Chowdhery et al., 2023; Gemini Team, 2023; Achiam et al., 2023; Touvron et al., 2023; Zhao et al., 2023; Jiang et al., 2024a,b). In particular, cloud-based proprietary LLMs, referred to as service models, such as GPT-4 (Achiam et al., 2023), Gemini 1.5 (Gemini Team, 2023), and Claude 3 (Anthropic, 2024), have exhibited exceptional capabilities when compared to their smaller, open-source counterparts (Chang et al., 2024). A notable survey involving 70 AI industry leaders from diverse enterprises reveals that approximately 80% of the enterprise market share is dominated by closed-source platforms, with a significant portion of this share attributed to OpenAI (Wang and Xu, 2024).

However, the increasing reliance on cloud-based service models presents significant challenges in terms of operational dependencies (Achiam et al., 2023), privacy concerns (Wu et al., 2024), and accessibility challenges (Ray, 2023). These challenges manifest in various ways, including potential service disruptions, heightened risks to data privacy due to the transmission of sensitive information to external providers, mandatory internet connectivity for utilization, and inconsistencies stemming from updates to service providers’ LLMs (Hadi et al., 2023; Zhao et al., 2023). Additionally, the transition from proof-of-concept (PoC) development utilizing service LLMs to deployment with local models frequently leads to diminished prompt effectiveness owing to differences between models, subsequently resulting in a suboptimal experience for end-users (Naveed et al., 2023; Lyu et al., 2024). To address these concerns and ensure consistent service delivery, it is imperative to develop smaller, locally manageable LLMs that can operate independently of cloud-based infrastructures.

Recent studies have demonstrated that the strategic fine-tuning of smaller and open-source LLMs with high-quality synthetic data (Wang et al., 2023b; Xu et al., 2023) generated by service LLMs can achieve performances that are on par with, or even surpass, those of proprietary LLMs in specific downstream tasks (Chiang et al., 2023; Taori et al., 2023a; Luo et al., 2023; Abdin et al., 2024; Zhou et al., 2024). Motivated by these findings, we introduce an LLMOps pipeline namely LlamaDuo
designed to automatically facilitate the seamless migration from service-oriented LLMs to smaller, locally manageable models without the need for human intervention. Our pipeline begins with utilizing a task-specific initial dataset, referred to as the coverage dataset, to fine-tune a smaller open-source LLM. The performance of fine-tuned local LLMs is evaluated using a service LLMs-as-a-Judge strategy (Zheng et al., 2024). If the performance of the fine-tuned model falls short of expectations, we improve it by iteratively fine-tuning on additional synthetic data generated by the service LLM. LlamaDuo ensures that the smaller model is capable of eventually matching or even surpassing the service LLM’s performance in specific downstream tasks, offering superior long-term economic advantages. Therefore, it presents a practical and scalable solution for managing AI deployments in environments where resources are limited. We conduct extensive experiments and analyses across a range of typical tasks, using popular service LLMs such as GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash, as well as local LLMs, including Gemma 2B and 7B, Mistral 7B, and LLaMA3 8B, to demonstrate that our LlamaDuo guarantees the smaller local LLMs possess the potential to eventually match or even exceed the performance of service LLMs in specific downstream tasks. To summarize, our contributions are as follows:

• We introduce LlamaDuo, an efficient and affordable LLMOps pipeline designed to facilitate seamless migration from service-oriented LLMs to smaller, locally manageable models without human intervention, ensuring service continuity in constrained environments.

• We employ a multi-turn approach using task-specific synthetic data generated by service LLMs to ensure that LlamaDuo empowers the smaller model to eventually match or even exceed the performance of the service LLM in specific downstream tasks.

• We substantiate the pipeline’s robust performance and adaptability in real-world context through comprehensive experiments across a range of typical tasks, employing popular service LLMs as synthetic data generators and judges for well-known small local LLMs.

• We emphasize the significant economic advantages of LlamaDuo for investing in smaller, locally manageable LLMs and their deployment for sustained use, as opposed to the transient benefits derived from the token-based API usage of service LLMs.

2 Related Work

2.1 Alignment with Instruction Tuning

LLMs pretrained on massive corpora demonstrate remarkable capabilities across a wide range of tasks (Zhao et al., 2023; Cai et al., 2024; Yoo et al., 2024; Wang et al., 2024a). Despite their capabilities, a notable challenge with LLMs is their misalignment with user instructions, which limits their practical applications in real-world scenarios (Xu et al., 2023; Wang et al., 2023b). The misalignment stems from the initial pretraining objective of LLMs, which focuses on minimizing generation errors rather than adhering to human instructions (Ouyang et al., 2022; Chung et al., 2024). To solve the mismatch, instruction tuning is proposed, which enables LLMs to complete diverse tasks from instructions without significant computational resources or alterations to the model’s architecture(Longpre et al., 2023; Muenninghoff et al., 2023; Taori et al., 2023b). Specifically, instruction tuning involves supplementary training of pretrained LLMs with datasets structured as instruction-output pairs (Zhang et al., 2023). The efficacy of instruction tuning is largely contingent upon the quality and diversity of the instruction datasets employed (Wang et al., 2024b). However, the process of curating high-quality, diversified data is fraught with challenges, including the extensive time required for creation, privacy concerns, high costs, and the need for substantial human labor (Xu et al., 2023). In response to these challenges, recent studies have explored innovative methods for constructing instruction datasets, notably the utilization of LLMs for data synthesis (Liu et al., 2024).

2.2 LLM-synthetic Instruction Data

LLMs have demonstrated an unprecedented ability to comprehend and execute natural language instructions (Ouyang et al., 2022; Chung et al., 2024; Touvron et al., 2023). This ability is attributed to the process of training LLMs using substantial instruction datasets (Wang et al., 2023b). However, acquiring massive instruction datasets is challenging due to data scarcity, privacy issues, low data quality, and prohibitive costs associated with manual data curation (Abay et al., 2019; Xu et al., 2023;
Liu et al., 2024). Given these constraints, recent studies probe into utilizing LLMs to automatically generate synthetic instruction data (Whitehouse et al., 2023; Dai et al., 2023; Taori et al., 2023b). Specifically, these approaches involve prompting powerful LLMs with limited seed data to generate additional synthetic data. These data are subsequently employed to fine-tune smaller models, aiming to transfer knowledge to small LLMs and enhance their performance (Wang et al., 2023a). Leveraging LLMs to generate data can significantly reduce the costs and time for data curation (Liu et al., 2024), while simultaneously improving the efficacy of the fine-tuned models for designated downstream tasks (Yang et al., 2020; Puri et al., 2020; Guo et al., 2023; Samuel et al., 2023; Schlegel et al., 2023).

3 LLMOps Pipeline: LlamaDuo

In this section, we elaborate on the details of the proposed LlamaDuo, which are depicted in Figure 1. This LLMOps pipeline aims to ensure service LLMs continuity by transitioning knowledge and abilities from service-oriented LLMs to smaller, locally manageable LLMs without the need for human intervention.

3.1 Coverage Dataset

Users interact with service LLMs through prompt engineering efforts. The historical trials composed of the user input prompt and the responses of service LLMs, and potential errors will be recorded and saved in local storage. Subsequently, users annotate and collect the most satisfied prompt and response pairs conformed with their real-world use cases. The resulting instruction dataset is termed as coverage dataset, denoted as \( \mathcal{D}^{(0)} := \{(\mathcal{I}_i^{(0)}, \mathcal{R}_i^{(0)})\}_{i=1}^N \), and split as train and test subsets by ratio \( \Phi \). Here, \( \mathcal{I}_i^{(0)} \) denotes the \( i \)-th instruction (prompt) in \( \mathcal{D}^{(0)} \), \( \mathcal{R}_i^{(0)} \) is the corresponding response for the \( i \)-th instruction, and \( N \) is the number of samples in \( \mathcal{D}^{(0)} \). Since coverage dataset is of high quality and satisfying the user’s intent in real-world context, the train subsets \( |\mathcal{D}^{(0)}_{train}| = \Phi \cdot N \) will be served as seeds for synthetic datasets generation, while the test subset \( |\mathcal{D}^{(0)}_{test}| = (1 - \Phi) \cdot N \) is reserved for performance evaluation of the fine-tuned local LLMs.

3.2 Fine-tuning

To efficiently and effectively adapt the local model to specific downstream task(s), we finetune the local LLM with the supervised learning paradigm on high-quality instruction data. At the initial cyclicality of the pipeline, the selected local LLM is fine-tuned on the train subsets \( \mathcal{D}^{(0)}_{train} \) of the coverage dataset, obtaining the fine-tuned model \( \pi^{(0)} \). At subsequent cyclicality \( t \), if the performance of fine-tuned model does not reach or surpass the predetermined evaluation threshold \( \varepsilon \) of specific tasks, the local LLM \( \pi^{(t)} \) will be continuously fine-tuned on the increasing number of synthetic data \( \{\mathcal{D}^{(1)}_{synth}, \mathcal{D}^{(2)}_{synth}, \ldots, \mathcal{D}^{(t-1)}_{synth}\} \) generated from service LLMs with \( \mathcal{D}^{(0)}_{train} \) as seed dataset. Consequently, when \( t \geq 1 \), the objective of the fine-tuning phase can be formulated as

\[
\mathcal{L}_{SFT}(\pi^{(t)}, \mathcal{D}^{(t)}) = -\mathbb{E} \left[ \log P_{\pi^{(t-1)}}(\mathcal{R}^{(t)} \mid \mathcal{I}^{(t)}) \right],
\]

where \( \mathcal{R}^{(t)} \sim \{ \mathcal{D}^{(0)}_{train}; \{ \mathcal{D}^{(\tau)}_{synth} \}_{\tau=1}^{t-1} \} \) and \( \mathcal{I}^{(t)} \sim \mathcal{D}^{(0)}_{train} \).

3.3 Batch Inference

After the fine-tuning stage, the fine-tuned local model is prompted with prompts \( \mathcal{I}^{(0)} \) sampled from the test subsets \( \mathcal{D}^{(0)}_{test} \) of the coverage dataset to produce corresponding response \( \hat{\mathcal{R}} \sim \pi^{(t)}(\mathcal{R}^{(0)} \mid \mathcal{I}^{(0)}) \). To improve the diversity and robustness of responses, the local model generates a batch of \( K \) responses \( \{ \hat{\mathcal{R}}_1, \hat{\mathcal{R}}_2, \ldots, \hat{\mathcal{R}}_K \} \) for each given prompt \( \mathcal{I}^{(0)} \). Totally, it will construct prompt and responses pairs \( \{ (\mathcal{I}_i^{(0)}, \hat{\mathcal{R}}_i ) \}_{i=1}^{(1-\Phi)\cdot N \cdot K} \). Formally,

\[
\hat{\mathcal{R}}_k \sim \pi^{(t)}(\mathcal{R}^{(0)} \mid \mathcal{I}^{(0)}),
\]

where \( k \in \{1, 2, \ldots, K\} \), \( \mathcal{I}^{(0)} \sim \mathcal{D}^{(0)}_{test} \).

3.4 Evaluation

In the evaluation stage, we employ “service LLMs-as-judge”, denoted as \( \mathcal{E}_{LLM}(\cdot) \), to conduct performance evaluation of local model on \( \{ (\mathcal{I}_i^{(0)}, \hat{\mathcal{R}}_i ) \}_{i=1}^{(1-\Phi)\cdot N \cdot K} \). Following the works (Zheng et al., 2024; Yuan et al., 2024), the service LLMs evaluate each response triple \( (\mathcal{I}^{(0)}, \hat{\mathcal{R}}, \mathcal{R}^{(0)}) \), comprising prompt, the corresponding generated response, and the ground truth, by \( M \) times with pairwise comparison and single answer grading strategies. This evaluation process guarantees the trustworthy and reduces the inherent bias of results. Moreover, when employing
Figure 1: The LLMOps pipeline namely LlamaDuo for migrating from service LLMs to small-scale local LLMs involves three phases. In the Development/PoC phase, (1) users manually engineer prompts to interact with service LLMs and (2) collect satisfying (prompt, response) pairs into train and test datasets. In the Alignment phase, (3) local LLMs are aligned with the train dataset, (4) tested on the test dataset, and (5) evaluated by service LLMs. (6) Synthetic data is generated iteratively until the performance of the aligned model meets a threshold. In the Deployment phase, (7) the satisfactory model is deployed in constrained environments.

LLMs as evaluators, the evaluation metrics can be more flexibly adapted to specific tasks, along with a thorough evaluation guide. In this paper, we measure the similarity between \( \hat{\mathcal{R}} \) and \( \mathcal{R}^{(0)} \), and how precise \( (\mathcal{T}^{(0)}, \hat{\mathcal{R}}) \) the responses generated by the local LLM answer the given instructions. These two metrics are provided simultaneously through a prompt, as shown in Figure 5 of Appendix A. Therefore, \( \{(\mathcal{T}_i^{(0)}, \hat{\mathcal{R}}_i, \mathcal{R}_i^{(0)})\}_{i=1}^{(1-\Phi)\cdot N\cdot K} \) invokes service LLMs to perform evaluation by \( (1-\Phi) \cdot N \cdot K \cdot M \) times. Subsequently, the evaluation results can be leveraged according to the intention of the operator performing this LLMOps pipeline. For example, actions can be taken to increase the reliability of service LLM as an evaluator by calculating the mean or median. In this study, we adopt the mean score \( V_{\pi^{(t)}} \) and coverage percentage \( C_{\pi^{(t)}} \) with \( \zeta \) score as evaluation results. Here, the coverage percentage \( C_{\pi^{(t)}} \) indicates the proportion of responses that have met or exceeded the quality benchmark. Formally,
\[
V_{\pi^{(t)}} = \frac{1}{(1-\Phi) \cdot N \cdot K} \sum_{j=1}^{(1-\Phi)\cdot N \cdot K} V_{\pi^{(t)}}^j, \tag{3}
\]
\[
C_{\pi^{(t)}} = \frac{1}{(1-\Phi) \cdot N \cdot K} \sum_{j=1}^{(1-\Phi)\cdot N \cdot K} \mathbb{1}(V_{\pi^{(t)}}^j \geq \zeta), \tag{4}
\]
\[
V_{\pi^{(t)}}^j = \frac{1}{M} \sum_{m=1}^M \mathcal{E}_{\text{LLM}}(\text{prompt}^{(eval)}, d_j), \tag{5}
\]
\[
d_j \sim \{(\mathcal{T}_i^{(0)}, \hat{\mathcal{R}}_i, \mathcal{R}_i^{(0)})\}_{i=1}^{(1-\Phi)\cdot N \cdot K}, \tag{6}
\]
where \( V_{\pi^{(t)}} \) and \( C_{\pi^{(t)}} \) denote the performance of local LLM at \( t \)-th cyclicality, \( \mathbb{1}(\cdot) \) is an indicator function, \( \zeta \) denotes a threshold score of \( C_{\pi^{(t)}} \), \( \text{prompt}^{(eval)} \) is the system prompt used for LLM-as-a-Judge.

3.5 Data Synthesis

If the performance of fine-tuned local LLM \( V_{\pi^{(t)}} \) or \( C_{\pi^{(t)}} \) fails to reach or surpass the predetermined evaluation threshold \( \varepsilon \) of specific tasks, it indicates that fine-tuned local LLM’s capabilities are insufficient for the tasks at hand. Thus, the local LLM cannot yet serve as a substitute for the service LLM and necessitates further refinement. To achieve this, we utilize service LLMs to generate additional synthetic datasets for fine-tuning local LLM in the next cyclicality. To maintain the consistency of data distribution of coverage dataset \( \mathcal{D}^{(0)} \) constructed from real-world scenarios, we employ the train subsets \( \mathcal{D}_{train}^{(0)} \) as seeds and apply the same framework (Wang et al., 2023b; Taori et al., 2023a) for synthetic dataset generation. During synthetic dataset generation, we perform data deduplication to exclude identical samples from \( \mathcal{D}' = \{\mathcal{D}_{train}^{(0)}, \{\mathcal{D}_1^{synth}, \mathcal{D}_2^{synth}, \ldots, \mathcal{D}_l^{synth}\}\} \) and filter out low-quality samples based on carefully designed rules. Finally, we conduct rigorous data decontamination for the synthetic dataset to remove samples that closely resemble those in the test subset \( \mathcal{D}_{test}^{(0)} \) of the coverage dataset. Formally, the data synthesis stage can be formulated as
\[
\mathcal{D}_j^{synth} \leftarrow \bigcup \psi(\mathcal{D}_j^{synth}, \mathcal{D}', \mathcal{D}_{test}^{(0)}), \tag{7}
\]
\[
\mathcal{D}_j^{synth} \sim S_{\text{LLM}}(\text{prompt}^{(synth)}, seed), \tag{8}
\]
\[
seed \sim \mathcal{D}_{train}^{(0)}, \text{for } V_{\pi^{(t)}} < \varepsilon \text{ or } C_{\pi^{(t)}} < \varepsilon. \tag{9}
\]
where \( \bigcup \psi(\cdot, \cdot, \cdot) \) represent a series of data post-processing operations, \( D_{synth}^{(t)} \) denotes synthetic data generated from service LLMs at \( t \)-th cyclicality, \( S_{LLM} \) and prompt\(^{(synth)}\) are the service LLM and system prompt used for the data synthesis, respectively.

4 Experiments

In this section, we present a comprehensive evaluation of our LlamaDuo across a series of settings, demonstrating its robust performance and adaptability in real-world scenarios.

4.1 Experimental Settings

Tasks and coverage dataset. We select four categories of downstream tasks—summarization, classification, coding, and closed QA—based on their prevalent use and relevance to the operational scope of service LLMs. We utilize the open-source “No Robots” (Rajani et al., 2023) dataset as the coverage dataset. This coverage dataset consists of 10K high-quality prompt and response pairs across 10 categories, crafted by expert annotators. Specifically, we utilize four subsets of the coverage dataset, each corresponding to our targeted tasks. These subsets serve as seeds for generating synthetic data that can closely align with user expectations for LLM interactions.

Service and local LLMs. Considering the API cost effectiveness, rate limit, and model utility, we select popular service LLMs including GPT4o by OpenAI, Claude 3 Sonnet by Anthropic, and Gemini 1.5 Flash by Google to serve as synthetic data generators and judges. As for the small-scale local LLMs to be fine-tuned, we opt for the open-source Gemma 2B and 7B (Gemma Team, 2024), Mistral 7B (Jiang et al., 2023), and LLaMA3 8B (Meta, 2024) as the base models. This selection is motivated by our aim to rigorously evaluate the efficacy and adaptability of our proposed pipeline across diverse settings. The varying scales of base models facilitate a nuanced comparison, allowing us to assess the impact of model scale on performance improvements. However, as a model-agnostic LL-MOps pipeline, our LlamaDuo can be generalized to various forms of service and local LLMs beyond the aforementioned models.

4.2 Implementation Details

We implement LlamaDuo using PyTorch and conduct experiments on 8 × A100 (80GB) GPUs.

Synthetic dataset by service LLMs. We utilize the seeds selected from the train subset of the coverage dataset to prompt service LLMs to generate datasets, each comprising 300k samples. The specific prompt for data generation is presented in Figure 6 of Appendix A. Subsequently, we employ Locality-Sensitive Hashing (LSH) with MinHash and Rouge scoring mechanisms for data deduplication. Specifically, the LSH MinHash can efficiently identify and remove duplicate data samples, while the Rouge scoring mechanism ensures that the curated data exhibits high-quality and meaningful variations. After that, we acquire 256k samples for summarization tasks and 128k for other tasks.

Fine-tuning Local LLMs. We proceed to fine-tune the small local LLMs with \( 2^n k, n \in \{0, 1, \ldots, 8\} \) volumes of the synthetic dataset. To efficiently customize local LLM for a specific downstream task within constrained environments, we leverage QLoRA (Dettmers et al., 2024) for parameter-efficient fine-tuning with superior cost-effectiveness. The detailed configurations, which are tailored according to dataset sizes and tasks, can be found in Appendix B.

Batch inference. Each fine-tuned local model is prompted to generate \( K = 4 \) distinct responses, with each prompt sampled from the test subsets of the coverage dataset. To ensure fair comparisons, we maintain a consistent batch inference configuration across all fine-tuned models. The detailed configuration is depicted in Appendix B.

Service LLMs as judges. Following (Zheng et al., 2024), we employ pairwise comparison and single answer grading strategies to evaluate the response quality of the fine-tuned local LLMs. The corresponding prompts are given in Figure 5 of Appendix A. We utilize similarity and precision metrics. The similarity metric assesses the degree of correspondence between the generated responses and the ground truth, while the precision metric evaluates the accuracy of the match between the input prompts and their corresponding responses. To ensure reliability and mitigate inherent biases in the results, both metrics are quantified on a 0 to 100 scale, with each sample undergoing evaluation \( M = 10 \) times. The score of coverage percentage is set to \( \zeta \in \{50, 70\} \).

4.3 Experimental Results

This section delves into the effectiveness and adaptability of the LlamaDuo pipeline, spanning different tasks with varying degrees of complexity, in-
Table 1: Performance of the service LLMs and local LLMs fine-tuned on 128K synthetic dataset produced by GPT4o, evaluated by GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash as judges on test subsets of coverage dataset. Each entry is presented as mean score / coverage percentage (%) with 50 score / coverage percentage (%) with 70 score. The best results from service and local LLMs are highlighted in bold. “P-Match” represents performance matching, which is defined as the best performance of the local LLM divided by the best performance of the service LLM, with the best results highlighted in bold across different judges.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Model</th>
    <th colspan="2">GPT4o</th>
    <th colspan="2">Claude 3 Sonnet</th>
    <th colspan="2">Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision*</th>
    <th>Similarity†</th>
    <th>Precision*</th>
    <th>Similarity†</th>
    <th>Precision*</th>
    <th>Similarity†</th>
  </tr>
  <tr>
    <td rowspan="6">Summarization</td>
    <td>GPT4o</td>
    <td>90.71 / 97% / 96%</td>
    <td>82.00 / 95% / 89%</td>
    <td>83.25 / 100% / 100%</td>
    <td>86.60 / 100% / 95%</td>
    <td>87.10 / 100% / 92%</td>
    <td>67.45 / 65% / 48%</td>
  </tr>
  <tr>
    <td>Claude 3 Sonnet</td>
    <td>88.04 / 97% / 92%</td>
    <td>78.18 / 95% / 78%</td>
    <td>83.39 / 100% / 99%</td>
    <td>85.55 / 100% / 95%</td>
    <td>86.70 / 100% / 92%</td>
    <td>64.10 / 80% / 56%</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>87.90 / 96% / 96%</td>
    <td>79.14 / 95% / 88%</td>
    <td>81.95 / 100% / 96%</td>
    <td>85.05 / 100% / 95%</td>
    <td>85.65 / 100% / 96%</td>
    <td>66.45 / 89% / 40%</td>
  </tr>
  <tr>
    <td>Gemma 2B</td>
    <td>87.54 / 96% / 96%</td>
    <td>78.54 / 95% / 78%</td>
    <td>81.95 / 100% / 96%</td>
    <td>85.05 / 100% / 95%</td>
    <td>85.65 / 100% / 96%</td>
    <td>66.45 / 89% / 40%</td>
  </tr>
  <tr>
    <td>Gemma 7B</td>
    <td>73.54 / 85% / 65%</td>
    <td>68.55 / 85% / 59%</td>
    <td>86.19 / 99% / 93%</td>
    <td>77.41 / 94% / 77%</td>
    <td>74.59 / 95% / 69%</td>
    <td>53.92 / 65% / 22%</td>
  </tr>
  <tr>
    <td>Mistral 7B</td>
    <td>76.38 / 93% / 70%</td>
    <td>69.65 / 88% / 56%</td>
    <td>86.20 / 99% / 92%</td>
    <td>78.44 / 96% / 80%</td>
    <td>72.74 / 95% / 62%</td>
    <td>50.15 / 54% / 14%</td>
  </tr>
  <tr>
    <td>LLaMA3 SB</td>
    <td>75.67 / 88% / 75%</td>
    <td>68.54 / 86% / 69%</td>
    <td>87.82 / 99% / 94%</td>
    <td>84.72 / 97% / 86%</td>
    <td>72.74 / 95% / 62%</td>
    <td>52.23 / 64% / 25%</td>
  </tr>
  <tr>
    <td>P-Match*</td>
    <td>84.70 / 93% / 83%</td>
    <td>77.00 / 95% / 83%</td>
    <td>85.05 / 100% / 95%</td>
    <td>87.00 / 100% / 95%</td>
    <td>79.35 / 100% / 95%</td>
    <td>79.35 / 100% / 95%</td>
  </tr>
  <tr>
    <td rowspan="6">Classification</td>
    <td>GPT4o</td>
    <td>83.62 / 94% / 81%</td>
    <td>74.45 / 80% / 66%</td>
    <td>87.50 / 92% / 92%</td>
    <td>72.28 / 77% / 66%</td>
    <td>82.66 / 94% / 80%</td>
    <td>63.06 / 67% / 44%</td>
  </tr>
  <tr>
    <td>Claude 3 Sonnet</td>
    <td>82.32 / 92% / 78%</td>
    <td>71.51 / 81% / 70%</td>
    <td>82.89 / 100% / 100%</td>
    <td>75.07 / 81% / 73%</td>
    <td>87.34 / 97% / 97%</td>
    <td>67.18 / 80% / 45%</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>85.43 / 94% / 91%</td>
    <td>72.71 / 81% / 75%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 2B</td>
    <td>87.54 / 96% / 96%</td>
    <td>78.54 / 95% / 78%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 7B</td>
    <td>70.73 / 69% / 55%</td>
    <td>64.67 / 62% / 53%</td>
    <td>78.78 / 81% / 75%</td>
    <td>67.66 / 69% / 62%</td>
    <td>70.73 / 75% / 61%</td>
    <td>59.77 / 59% / 52%</td>
  </tr>
  <tr>
    <td>Mistral 7B</td>
    <td>67.53 / 70% / 53%</td>
    <td>61.65 / 67% / 47%</td>
    <td>76.01 / 80% / 72%</td>
    <td>64.43 / 70% / 52%</td>
    <td>67.90 / 73% / 53%</td>
    <td>54.27 / 43% / 25%</td>
  </tr>
  <tr>
    <td>LLaMA3 SB</td>
    <td>68.41 / 70% / 52%</td>
    <td>61.65 / 67% / 47%</td>
    <td>76.01 / 80% / 72%</td>
    <td>64.43 / 70% / 52%</td>
    <td>67.90 / 73% / 53%</td>
    <td>54.27 / 43% / 25%</td>
  </tr>
  <tr>
    <td>P-Match*</td>
    <td>95.50% / 93% / 80%</td>
    <td>82.20% / 89% / 95%</td>
    <td>89.33% / 94% / 94%</td>
    <td>101.45% / 100% / 92.59%</td>
    <td>95.79% / 96.15% / 79.38%</td>
    <td>107.88% / 91.25% / 106.17%</td>
  </tr>
  <tr>
    <td rowspan="6">Coding</td>
    <td>GPT4o</td>
    <td>90.31 / 100% / 98%</td>
    <td>75.18 / 92% / 70%</td>
    <td>94.57 / 100% / 100%</td>
    <td>86.32 / 100% / 91%</td>
    <td>90.78 / 100% / 100%</td>
    <td>58.43 / 62% / 25%</td>
  </tr>
  <tr>
    <td>Claude 3 Sonnet</td>
    <td>88.76 / 100% / 92%</td>
    <td>73.25 / 94% / 67%</td>
    <td>93.82 / 100% / 100%</td>
    <td>87.42 / 100% / 100%</td>
    <td>89.84 / 100% / 100%</td>
    <td>60.46 / 69% / 31%</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>85.43 / 94% / 91%</td>
    <td>72.71 / 81% / 75%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 2B</td>
    <td>87.54 / 96% / 96%</td>
    <td>78.54 / 95% / 78%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 7B</td>
    <td>80.56 / 92% / 80%</td>
    <td>71.92 / 89% / 70%</td>
    <td>80.26 / 92% / 84%</td>
    <td>84.66 / 100% / 88%</td>
    <td>61.23 / 72% / 36%</td>
    <td>45.25 / 30% / 18%</td>
  </tr>
  <tr>
    <td>Mistral 7B</td>
    <td>68.32 / 70% / 56%</td>
    <td>61.01 / 69% / 45%</td>
    <td>81.25 / 92% / 81%</td>
    <td>69.07 / 88% / 55%</td>
    <td>72.99 / 88% / 69%</td>
    <td>31.25 / 20% / 18%</td>
  </tr>
  <tr>
    <td>LLaMA3 SB</td>
    <td>77.48 / 82% / 70%</td>
    <td>67.82 / 70% / 56%</td>
    <td>83.31 / 92% / 84%</td>
    <td>75.31 / 88% / 75%</td>
    <td>72.99 / 88% / 69%</td>
    <td>31.25 / 20% / 18%</td>
  </tr>
  <tr>
    <td>P-Match*</td>
    <td>89.20% / 92% / 81%</td>
    <td>83.11% / 94% / 85%</td>
    <td>95.66% / 100% / 98%</td>
    <td>91.81% / 92% / 84%</td>
    <td>93.26% / 100% / 88%</td>
    <td>95.36% / 85% / 97.80%</td>
  </tr>
  <tr>
    <td rowspan="6">Closed QA</td>
    <td>GPT4o</td>
    <td>95.45 / 100% / 100%</td>
    <td>84.23 / 93% / 80%</td>
    <td>97.21 / 100% / 100%</td>
    <td>92.56 / 100% / 97%</td>
    <td>93.58 / 100% / 100%</td>
    <td>75.58 / 83% / 63%</td>
  </tr>
  <tr>
    <td>Claude 3 Sonnet</td>
    <td>94.03 / 100% / 98%</td>
    <td>85.10 / 100% / 82%</td>
    <td>97.40 / 100% / 100%</td>
    <td>93.95 / 100% / 100%</td>
    <td>93.66 / 100% / 100%</td>
    <td>76.33 / 82% / 63%</td>
  </tr>
  <tr>
    <td>Gemini 1.5 Flash</td>
    <td>87.54 / 96% / 96%</td>
    <td>78.54 / 95% / 78%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 2B</td>
    <td>87.54 / 96% / 96%</td>
    <td>78.54 / 95% / 78%</td>
    <td>89.01 / 98% / 98%</td>
    <td>77.96 / 81% / 81%</td>
    <td>83.55 / 94% / 84%</td>
    <td>62.23 / 78% / 47%</td>
  </tr>
  <tr>
    <td>Gemma 7B</td>
    <td>81.85 / 88% / 83%</td>
    <td>79.02 / 85% / 78%</td>
    <td>88.83 / 93% / 93%</td>
    <td>83.95 / 87% / 83%</td>
    <td>82.51 / 93% / 80%</td>
    <td>72.24 / 73% / 56%</td>
  </tr>
  <tr>
    <td>Mistral 7B</td>
    <td>68.32 / 70% / 56%</td>
    <td>61.01 / 69% / 45%</td>
    <td>81.25 / 92% / 81%</td>
    <td>69.07 / 88% / 55%</td>
    <td>72.99 / 88% / 69%</td>
    <td>31.25 / 20% / 18%</td>
  </tr>
  <tr>
    <td>LLaMA3 SB</td>
    <td>75.55 / 78% / 75%</td>
    <td>72.62 / 77% / 72%</td>
    <td>86.03 / 88% / 85%</td>
    <td>77.64 / 88% / 80%</td>
    <td>79.09 / 85% / 77%</td>
    <td>68.78 / 75% / 65%</td>
  </tr>
  <tr>
    <td>P-Match*</td>
    <td>87.62% / 88% / 83%</td>
    <td>93.06% / 85% / 94%</td>
    <td>90.41% / 93% / 93%</td>
    <td>87.02% / 88% / 83%</td>
    <td>86.85% / 93% / 80%</td>
    <td>86.33% / 80% / 84.34%</td>
  </tr>
</table>

cluding summarization, classification, coding, and closed QA. We utilize GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash as judges to evaluate the fine-tuned model performance on test subsets of the coverage dataset. As demonstrated in Table 1, the fine-tuned local LLMs, despite their significantly smaller scale, achieve comparable performance on diverse tasks compared to much larger service LLMs. For instance, in the summarization task, LLaMA3 SB achieved a comparable precision score of 87.02 / 99% / 94%, compared to GPT4o’s score of 93.25 / 100% / 100%, Claude 3 Sonnet’s score of 93.39 / 100% / 99%, and Gemini 1.5 Flash’s score of 91.95 / 100% / 98%, with Claude 3 Sonnet serving as judge. These results underscore the efficacy of LlamaDuo in seamlessly transferring knowledge and capabilities from service LLMs to smaller local LLMs without a substantial decrease in performance.

In Table 1, we observe distinct performance across four fine-tuned models when applied to different tasks. Specifically, Mistral 7B stands out in summarization tasks, achieving the best performance in 7 out of 12 cases. Moreover, LLaMA3 SB consistently outperforms competitors across all metrics and evaluators in the classification task. Conversely, in coding tasks, Gemma 7B is identified as the leading model, excelling across all metrics and evaluations. Mistral 7B shows superior performance in the closed QA task, leading in 8 out of 12 cases. Within the realm of service LLMs, Claude 3 Sonnet and Gemini 1.5 Flash demonstrate exceptional performance in classification and closed QA tasks, securing the best results in 8 and 10 out of 12 cases, respectively. Lastly, GPT4o emerges as the leading model in summarization and coding tasks, achieving the best performance in 10 and 7 out of 12 cases, respectively. Notably, although Gemma 2B exhibits inferior performance compared to larger 7B models overall, the disparity in results is not markedly substantial, with Gemma 2B attaining closely comparable performance in certain tasks. For example, in closed QA tasks, Gemma 2B secures a mean precision score of 80.22, while Gemma 7B achieves 88.83, Mistral 7B reaches 88.25, and LLaMA3 SB obtains 86.03, as evaluated by Claude 3 Sonnet. This observation lends further support to the notion that through the strategic fine-tuning of smaller local LLMs on synthetic datasets via the LlamaDuo, it is possible to closely approximate the performance of their larger counterparts. Consequently, it offers increased flexibility and solutions for users and scenarios with budgetary considerations. More experimental results are presented in Appendix C.
Figure 2: Performance of Gemma 7B fine-tuned on varied volumes of synthetic dataset produced by various service LLMs including GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash. The first to third columns represent the performance of the model evaluated by GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash as judges, respectively. The first row show mean scores, while the second and third rows shows the coverage percentage with 50 and 70 scores, respectively.

4.4 In-depth LLMOps Pipeline Analysis

In this section, we conduct an in-depth analysis of LlamaDuo through summarization task. Notably, the experimental findings exhibit a consistent pattern across various tasks, underscoring the generalizability of LlamaDuo.

Impact of synthetic dataset volume. We explore how the volume of synthetic dataset influences the performances of fine-tuned local LLMs, aiming to elucidate a scaling law (Kaplan et al., 2020; Hoffmann et al., 2022) on how the performance of fine-tuned models changes as the number of synthetic dataset samples increases. Overall, the Gemma 7B model exhibits consistent performance improvements and comes closer to the performance of service LLMs with increasing volumes of synthetic data, as assessed through precision and similarity metrics by diverse evaluators, as depicted in Figure 2. This suggests that fine-tuning local LLMs with synthetic data, which mimics the characteristics and patterns of real-world data, can bring the same effect as actual data. Thus, it paves a new way to eliminate the challenges of data scarcity, privacy concerns, and high costs associated with crafting data (Liu et al., 2024). Notably, we observe that the synthetic data generated by Claude 3 Sonnet results in the highest-performing models, outperforming those fine-tuned with data produced by GPT4o and Gemini 1.5 Flash, in descending order. Moreover, when the synthetic dataset volume ranges from 64k to 256k, the Gemma 7B model reaches the performance saturation point and achieves performance that is much closer to, or equal to, that of service LLMs. This demonstrates the efficacy of our LlamaDuo in enabling smaller models to replicate or even surpass the performance of service LLMs in specific downstream tasks.

Impact of service LLMs as data generator and judge. As shown in Figure 2, we observe that the choice of service LLM for data generation does not significantly impact the performance of the fine-tuned models. Specifically, (1) a consistent trend of performance enhancement is observed with the increased volume of synthetic data, irrespective of the service LLM employed for data generation; (2) the local LLMs fine-tuned on synthetic data generated by GPT4o and Claude3 Sonnet typically lead to slightly better performance than those by Gemini 1.5 Flash. On the other hand, employing different service LLMs as judges manifests a more pronounced impact on the performance of the fine-tuned local LLMs. Overall, GPT4o and Gemini 1.5 Flash emerge as more rigorous judges compared to Claude 3 Sonnet, with Gemini 1.5 Flash assigning notably lower similarity scores. Moreover, we observe that in data sparsity scenarios (1k to 4k), the type of evaluators has minimal influ-

Figure 3: The KDE Plots of Precision v.s. Similarity by varied synthetic dataset volumes with \(2^k, n \in \{0, 1, \ldots, 8\}\) and various evaluators with GPT4o, Claude 3 Sonnet, Gemini 1.5 Flash as judges from first to third columns, while the first and second rows represent the results of Gemma 2B (first row) and Gemma 7B (second row), respectively.
Table 2: Monthly operational cost comparison between Gemma 7B and GPT4o under different workloads. For GPT4o, input and output token counts are represented in the format input/output.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">Light Workload</th>
    <th colspan="2">Heavy Workload</th>
  </tr>
  <tr>
    <th>Gemma 7B</th>
    <th>GPT4o</th>
    <th>Gemma 7B</th>
    <th>GPT4o</th>
  </tr>
  <tr>
    <td>Fine-tuning</td>
    <td>Cloud<br>$800</td>
    <td>-</td>
    <td>Cloud<br>$800</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Serving Specs.</td>
    <td>1 x L4<br>$2,539</td>
    <td>300M/30M<br>$1,950</td>
    <td>8 x L4<br>$20,312</td>
    <td>1500M/150M<br>$9,750</td>
  </tr>
  <tr>
    <td>Serving Elec.</td>
    <td>165 kWh<br>$30</td>
    <td>-</td>
    <td>1319 kWh<br>$240</td>
    <td>-</td>
  </tr>
  <tr>
    <td>2 Months</td>
    <td>$3,369</td>
    <td>$3,900</td>
    <td>$21,592</td>
    <td>$19,500</td>
  </tr>
  <tr>
    <td>12 Months</td>
    <td>$3,699</td>
    <td>$23,400</td>
    <td>$23,992</td>
    <td>$117,000</td>
  </tr>
</table>

ence on the performance of the Gemma 7B model, suggesting that larger local LLMs exhibit diminished sensitivity to the choice of service LLM as a judge. To qualitatively demonstrate the differences when using various types of service LLMs as evaluators, Figure 3 presents the results as KDE plots, characterized by the dataset volume. We observe that GPT4o maintains consistency in its evaluations across both similarity and precision metrics. In contrast, Claude 3 Sonnet is found to be more lenient in scoring, while Gemini 1.5 Flash assigns higher precision scores but significantly lower similarity scores. This underscores the importance of strategically aligning the selection of service LLMs with specific task requirements.

4.5 Cost of Long-term Deployment

We elucidate the cost-effectiveness of our proposed LlamaDuo pipeline, by conducting a long-term operational cost comparison between the fine-tuning of the small LLMs (Gemma 7B) and the token-based API usage of service LLMs (GPT4o). In the context of local LLM deployment, the QLoRA fine-tuning process of Gemma 7B, utilizing a dataset containing 256K samples, necessitates approximately one hour to complete a single experiment on \( 8 \times A100 \) GPUs. This process incurs an estimated cost of $50, based on the price provided by Google Cloud Platform. Accounting for multiple iterations of hyperparameter optimization, we estimate that the total fine-tuning cost remains below $800, which is deemed to be negligible. Deploying a single instance of the Gemma 7B model with support for a 1024 context length necessitates 24GB of GPU memory, making the L4 GPU an appropriate choice. Depending on the projected workload, the Gemma 7B model can be deployed either on a single server equipped with one L4 GPU ($2,539) or across eight servers, each with one L4 GPU, with each server hosting a replica of the model instance ($20,312). In addition, the power consumption for each server is approximately $30 per month. For GPT4o, as of August 2024, the pricing is $5 and $15 per million tokens for input and output, respectively. We estimate that a light workload, utilizing 10 million input tokens and 1 million output tokens per day, incurs a daily cost of $65. Conversely, a heavy workload, consuming 50 million input tokens and 10 million output tokens per day, is estimated to cost $325 daily. The monthly operational cost comparison between Gemma 7B and GPT4o under different workloads is summarized in Table 2, demonstrating a significant advantage in fine-tuning and deploying a local LLM. Moreover, as depicted in Figure 4, after the first two months, the cost of using GPT4o under both light and heavy workloads exceeds that of setting up and running a local model deployed on \( 1 \times L4 \) GPU and \( 8 \times L4 \) GPU, respectively, as indicated by markers A and B. After one year, GPT4o’s costs surpass those of deploying a local model in all scenarios, as denoted by marker C. These findings highlight the substantial economic benefits of investing in local LLM fine-tuning and deployment for long-term use. Avoiding recurring token-based charges and maintaining control over model customization further enhances the appeal of the LlamaDuo for cost-conscious users and scenarios.

Figure 4: Long-term operational cost comparison between fine-tuning a local LLM and API-based token usage of GPT4o.

![Long-term operational cost comparison between fine-tuning a local LLM and API-based token usage of GPT4o.](page_1012_180_410_312.png)

5 Conclusion

In this study, we introduce LlamaDuo, the first automatic LLMOps pipeline designed to facilitate the seamless migration from service-oriented LLMs to smaller, locally manageable models. We conduct extensive experiments and analysis across a range of tasks with popular service and local LLMs to substantiate that LlamaDuo guarantees smaller local LLMs possess the potential to match or even exceed the performance of service LLMs in specific
downstream tasks, providing a promising research direction to maintain cloud-based LLMs’ service continuity in constrained environments.

Limitations

While our LlamaDuo pipeline presents a promising solution for migrating capabilities from service-oriented LLMs to smaller local models, as depicted in Table 1, several limitations must be acknowledged. First, the reliance on synthetic datasets generated by the service LLM may introduce biases and safety issues inherent in the original model, potentially affecting the fine-tuned model’s performance on specific tasks or datasets (Liu et al., 2024). Additionally, the effectiveness of the pipeline in transferring knowledge is contingent upon the quality and diversity of the synthetic data generated. If the data does not adequately cover the necessary scope, the fine-tuned model may struggle with tasks outside of the provided examples (Razeghi et al., 2022; Kandpal et al., 2023). Furthermore, the iterative fine-tuning process, while beneficial for performance enhancement, can be computationally intensive and time-consuming, potentially offsetting some gains in model efficiency, cost, and affordability. Another limitation is the potential plateau in performance gains after several SFT iterations, which could necessitate alternative strategies for further improvement, e.g., reinforcement learning (RL) (Ouyang et al., 2022; Rafailov et al., 2023). Lastly, the pipeline assumes access to the service LLM for data generation, which may not always be feasible due to proprietary restrictions or API access limitations.

Ethical Considerations

Our work introduces several ethical considerations that require careful examination. Primarily, the process of generating synthetic datasets raises questions about data privacy and security, especially if the data contains sensitive or proprietary information. There is a risk that such data, if not properly anonymized and secured, could lead to privacy violations or unauthorized data exposure (Liu et al., 2024; Das et al., 2025). Moreover, the transfer of biases from the service LLM to the smaller model could perpetuate or even exacerbate existing biases, leading to unfair or discriminatory outcomes in certain applications. It is crucial to implement robust bias detection and mitigation strategies within the pipeline to safeguard against these risks. Additionally, the use of proprietary models for generating synthetic data necessitates transparency regarding data handling practices and the potential limitations of the resultant models (Wang et al., 2023b).

Broader Impact

Beyond the immediate focus of this paper, we believe that the introduction of the LlamaDuo pipeline has the potential to significantly impact the landscape of LLMs deployment, particularly in environments with constrained resources or stringent privacy requirements. By enabling the migration of capabilities from large service-oriented LLMs to smaller, locally manageable models, the pipeline empowers organizations to maintain LLMs functionalities independently of external service providers, enhancing operational resilience and reducing dependency. This can lead to increased accessibility to advanced LLMs for smaller entities or those operating in regions with limited internet connectivity.

Acknowledgements

Jing Tang’s work is partially supported by National Key R&D Program of China under Grant No. 2024YFA1012700 and No. 2023YFF0725100, by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project (No. 2023QN10X025), by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2023A1515110131, by Guangzhou Municipal Science and Technology Bureau under Grant No. 2024A04J4454, by Guangzhou Municipal Education Bureau (No. 2024312263), by Nansha District Project (No. 2023ZD022), and by Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628) and Guangzhou Municipal Key Laboratory of Financial Technology Cutting-Edge Research (No. 2024A03J0630). This work is supported by IITP grant funded by the Korea government(MSIT)[RS-2023-00215959, Development of Access Agnostic wired and wireless integrated optical access technology].

References

Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioğlu, Bhavani Thuraisingham, and Latanya Sweeney. 2019. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Databases: European Conference,
ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18, pages 510–526. Springer.

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harikirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altshenmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbaf5c3b80b7bbc618857627/Model_Card_Claude_3.pdf.

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1–45.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liannmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gautav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Hyung Won Chung, Le Hou, Shayne Longpre, Barrett Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. Aug-gpt: Leveraging chatgpt for text data augmentation. Preprint, arXiv:2302.13007.

Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Gemini Team. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Gemma Team. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

Z Guo, P Wang, Y Wang, and S Yu. 2023. Improving small language models on pubmedqa via generative data augmentation. arXiv, Jul, 12.

Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024a. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024b. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PMLR.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations.

Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540.

AI Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI.

Niklas Muenninghoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111.

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2020. Training question answering models from synthetic data. arXiv preprint arXiv:2002.09599.

Rafael Rafaïlov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741.

Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023. No robots. https://huggingface.co/datasets/HuggingFaceH4/no_robots.

Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3:121–154.

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:2202.07206.

Vinay Samuel, Houda Aynaou, Arijit Ghosh Chowdhury, Karthik Venkat Ramanan, and Aman Chadha. 2023. Can llms augment low-resource reading comprehension datasets? opportunities and challenges. arXiv preprint arXiv:2309.12426.

Viktor Schlegel, Hao Li, Yuping Wu, Anand Subramanian, Thanh-Tung Nguyen, Abhinav Ramesh Kashyap, Daniel Beck, Xiaojun Zeng, Riza Theresa Batista-Navarro, Stefan Winkler, et al. 2023. Pulsa at medica-sum 2023: Large language models augmented by synthetic dialogue convert patient dialogues to medical records. arXiv preprint arXiv:2307.02006.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023a. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023b. Stanford alpaca: An instruction-following llama model.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. 2023. The alignment handbook. https://github.com/huggingface/alignment-handbook. NA.

Fan Wang, Juyong Jiang, Chansung Park, Sunghun Kim, and Jing Tang. 2024a. Kasa: Knowledge-aware singular-value adaptation of large language models. arXiv preprint arXiv:2412.06071.

Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. 2024b. A survey on data selection for llm instruction tuning. arXiv preprint arXiv:2402.05123.

Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. 2023a. Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. arXiv preprint arXiv:2310.13671.

Sarah Wang and Shangda Xu. 2024. 16 changes to the way enterprises are building and buying generative ai. URL: https://a16z.com/generative-ai-enterprise-2024/.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508.

Chenxi Whitehouse, Monojit Choudhury, and Alham Fikri Aji. 2023. LLM-powered data augmentation for enhanced cross-lingual performance. In The 2023 Conference on Empirical Methods in Natural Language Processing.

Xiaodong Wu, Ran Duan, and Jianbing Ni. 2024. Unveiling security, privacy, and ethical concerns of chatgpt. Journal of Information and Intelligence, 2(2):102–115.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. arXiv preprint arXiv:2004.11546.

Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020.

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36.
A Prompt Templates

In the LlamaDuo pipeline, we employ two prompt templates that serve different purposes: one for the generation of synthetic datasets and another for the evaluation of the outputs produced by the fine-tuned LLMs.

Figure 5 illustrates the prompt template used to assess the precision and similarity of the response $lm_response generated by fine-tuned small-scale LLMs, based on the prompt $instruction and response $human_response from the test subset of the coverage dataset. It is important to note that the $ symbol indicates a placeholder, designed to be substituted with actual data during the runtime.

Given an instruction and two responses—one generated by a human and the other by a language model—I’m seeking to evaluate how closely the language model’s response mirrors the human-generated one. Additionally, I want to assess the accuracy and precision of the language model’s response to the original instruction.

Instruction:
...
$instruction

Human Response:
$human_response

Language Model Response:
$lm_response

You are quality assessor who analyzes the similarity between the Human Response and the Language Model Response on a scale of 1 to 100, where 1 indicates no similarity and 100 indicates identical responses.

Also you analyze the Language Model Response how it accurately answers the given Instruction on a scale of 1 to 100. Analysis MUST be rigorous and thorough. Provide the results in the following JSON format:

{
    "similarity_assessment": {
        "score": [Insert similarity score here]
    },
    "precision_assessment": {
        "score": [Insert precision score here]
    }
}

Figure 5: Prompt template to evaluate the fine-tuned model’s response.

Figure 6 shows the prompt template designed for the generation of synthetic data tailored to the summarization task while Figure 7 shows the prompt template for other tasks. Specifically, we use a prompt $instruction alongside its corresponding response $response, both sourced from the train subset of the coverage dataset, serving as an example pair. This example pair is utilized to instruct service LLMs to generate analogous data samples. In addition, our template is designed to generate multiple synthetic data samples through a singular request, thereby enhancing the efficiency of API utilization. Due to the unique features of different downstream tasks, there is no optimal prompt template that universally applies. The actual content of the prompt template is adjusted to align with the specific requirements of the task for which the synthetic dataset is being generated.

Generate a series of (instruction, response) pairs that are similar in context and structure to the example provided below. Each pair should consist of a concise instruction followed by an appropriate, detailed response. The instruction should pose a clear task or question, while the response should provide a comprehensive answer or solution that could be understood by someone with a basic understanding of the subject.

Example pair:
Instruction: $instruction
Response: $response

Your task is to generate more pairs that maintain this level of clarity and detail. The topic is $topic. Write a long text of instruction by yourself, then summarize the given instruction into a response. Ensure that the responses are informative and accurate, suitable for an educational context.

Store the generated pairs in JSON format, with each pair as an object within an array. Each object should have two key-value pairs: "instruction" and "response". For instance:

{
    "contents": [
        {
            "instruction": "text", "response": "text",
        },
        {
            "instruction": "text", "response": "text",
        },
        ...
    ]
}

Remember to maintain consistency in the format and ensure the generated pairs are diverse and cover a broad range of subjects. You must return the response in the asked format and you must not add any additional text in your response.

Figure 6: Prompt template of data synthesis for summarization tasks.

Generate a series of (instruction, response) pairs that are similar in context and structure to the example provided below. Each pair should consist of a concise instruction followed by an appropriate, detailed response. The instruction should pose a clear task or question, while the response should provide a comprehensive answer or solution that could be understood by someone with a basic understanding of the subject.

Example pair:
Instruction: $instruction
Response: $response

Your task is to generate more pairs that maintain this level of clarity and detail. The topic is $topic. Ensure that the responses are informative and accurate, suitable for an educational context.

Store the generated pairs in JSON format, with each pair as an object within an array. Each object should have two key-value pairs: "instruction" and "response". For instance:

{
    "contents": [
        {
            "instruction": "text", "response": "text",
        },
        {
            "instruction": "text", "response": "text",
        },
        ...
    ]
}

Remember to maintain consistency in the format and ensure the generated pairs are diverse and cover a broad range of subjects. You must return the response in the asked format and you must not add any additional text in your response.

Figure 7: Prompt template of data synthesis for classification, coding, and closed QA tasks.

B Implementation Configuration

This section delineates the statistical information of the coverage dataset and synthetic dataset generated by service LLMs. In addition, we present the details of the training configurations of our experiments. The detailed pipeline implementation of LlamaDuo is available at https://github.com/deep-diver/llamaduo.

B.1 Coverage Datasets

The foundational coverage dataset employed in our study is the “No Robots” dataset (Rajani et al., 2023). We leverage four subsets of the coverage dataset, namely summarization, classification, coding, and closed QA, for synthetic data generation.
Table 3: Volume of coverage dataset before and after LlamaDuo pipeline.

<table>
  <tr>
    <th>Task</th>
    <th>Split</th>
    <th>Before</th>
    <th>After</th>
  </tr>
  <tr>
    <td rowspan="2">Summarization(GPT4o)</td>
    <td>train</td>
    <td>395</td>
    <td>256K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>25</td>
    <td>100</td>
  </tr>
  <tr>
    <td rowspan="2">Summarization(Claude 3 Sonnet)</td>
    <td>train</td>
    <td>395</td>
    <td>256K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>25</td>
    <td>100</td>
  </tr>
  <tr>
    <td rowspan="2">Summarization(Gemini 1.5 Flash)</td>
    <td>train</td>
    <td>395</td>
    <td>256K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>25</td>
    <td>100</td>
  </tr>
  <tr>
    <td rowspan="2">Classification(GPT4o)</td>
    <td>train</td>
    <td>334</td>
    <td>128K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>16</td>
    <td>64</td>
  </tr>
  <tr>
    <td rowspan="2">Coding(GPT4o)</td>
    <td>train</td>
    <td>334</td>
    <td>128K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>16</td>
    <td>64</td>
  </tr>
  <tr>
    <td rowspan="2">Closed QA(GPT4o)</td>
    <td>train</td>
    <td>245</td>
    <td>128K</td>
  </tr>
  <tr>
    <td>test</td>
    <td>15</td>
    <td>60</td>
  </tr>
</table>

Table 4: Token-level statistics of the coverage and synthetic datasets.

<table>
  <tr>
    <th>Task</th>
    <th>Min</th>
    <th>Max</th>
    <th>Avg.</th>
    <th>Std.</th>
  </tr>
  <tr>
    <td>Summarization (Coverage-Train)</td>
    <td>85</td>
    <td>2386</td>
    <td>389</td>
    <td>256</td>
  </tr>
  <tr>
    <td>Summarization (Coverage-Test)</td>
    <td>148</td>
    <td>1150</td>
    <td>426</td>
    <td>245</td>
  </tr>
  <tr>
    <td>Summarization (GPT4o)</td>
    <td>10</td>
    <td>2386</td>
    <td>95</td>
    <td>53</td>
  </tr>
  <tr>
    <td>Summarization (Claude 3 Sonnet)</td>
    <td>10</td>
    <td>2386</td>
    <td>118</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Summarization (Gemini 1.5 Flash)</td>
    <td>10</td>
    <td>2386</td>
    <td>108</td>
    <td>62</td>
  </tr>
  <tr>
    <td>Classification (Coverage-Train)</td>
    <td>18</td>
    <td>2159</td>
    <td>207</td>
    <td>244</td>
  </tr>
  <tr>
    <td>Classification (Coverage-Test)</td>
    <td>46</td>
    <td>520</td>
    <td>119</td>
    <td>109</td>
  </tr>
  <tr>
    <td>Classification (GPT4o)</td>
    <td>6</td>
    <td>2159</td>
    <td>67</td>
    <td>37</td>
  </tr>
  <tr>
    <td>Coding (Coverage-Train)</td>
    <td>38</td>
    <td>6518</td>
    <td>350</td>
    <td>502</td>
  </tr>
  <tr>
    <td>Coding (Coverage-Test)</td>
    <td>49</td>
    <td>821</td>
    <td>317</td>
    <td>189</td>
  </tr>
  <tr>
    <td>Coding (GPT4o)</td>
    <td>9</td>
    <td>6518</td>
    <td>151</td>
    <td>84</td>
  </tr>
  <tr>
    <td>Closed QA (Coverage-Train)</td>
    <td>58</td>
    <td>1497</td>
    <td>320</td>
    <td>241</td>
  </tr>
  <tr>
    <td>Closed QA (Coverage-Test)</td>
    <td>126</td>
    <td>1578</td>
    <td>411</td>
    <td>378</td>
  </tr>
  <tr>
    <td>Closed QA (GPT4o)</td>
    <td>12</td>
    <td>1701</td>
    <td>135</td>
    <td>59</td>
  </tr>
</table>

Table 3 illustrates the initial composition of the task-specific subsets, with each initially containing approximately 300 original data points. These subsets are subsequently expanded to encompass more data points using the LlamaDuo framework. To perform an in-depth analysis of the behavior of different service LLMs, we create synthetic datasets for the summarization task by utilizing GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash. For all other tasks, we exclusively use GPT4o, owing to budget constraints.

Table 4 presents the statistical information of the token count across each dataset. We only use data from the coverage train set for data synthesis and alignment tasks. We observe a reduction in both the average number of tokens and the standard deviation across the synthetic datasets compared to the original dataset. This is due to that the data synthesis process generates multiple synthetic data samples within a single API request.

Table 5: Detailed configurations used in the experiments.

<table>
  <tr>
    <th>Configuration</th>
    <th>Value</th>
  </tr>
  <tr>
    <td colspan="2">Common</td>
  </tr>
  <tr>
    <td>Data Type</td>
    <td>bfloat16</td>
  </tr>
  <tr>
    <td>Learning Rate Scheduler</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Max Number of Tokens</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>LoRA Type</td>
    <td>QLoRA</td>
  </tr>
  <tr>
    <td>LoRA Dropout</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td colspan="2">1K~16K</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>8</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>16</td>
  </tr>
  <tr>
    <td colspan="2">32K</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>16</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>32</td>
  </tr>
  <tr>
    <td colspan="2">64K~256K</td>
  </tr>
  <tr>
    <td>LoRA Rank</td>
    <td>32</td>
  </tr>
  <tr>
    <td>LoRA Alpha</td>
    <td>64</td>
  </tr>
</table>

B.2 Training Configurations

We utilize Hugging Face’s “Alignment Handbook” (Tunstall et al., 2023) and the alignment recipes tailored for the Gemma models to streamline the fine-tuning process.

As outlined in Table 5, we employ QLoRA (Dettmers et al., 2024) to align the Gemma 2B and 7B, Mistral 7B, and LLaMA3 8B models efficiently. The QLoRA method leverages the advantages of low-rank adaptation, reducing the computational resources required for training. Throughout the alignment procedure, we incrementally adjust the rank and alpha values of LoRA, aiming to optimize the adaptation layer’s capacity to match the increasing complexity of the datasets.

We set the maximum token as 1024 for the training phase, notwithstanding the presence of data samples exceeding this threshold. This decision is made based on a comprehensive analysis of the dataset, which indicates that data samples surpassing the token limit constitute a negligible portion of the total dataset. By imposing this limitation, we can concentrate our computational efforts on the majority of the data, thereby enhancing the efficiency of training without significantly compromising the models’ ability to generalize to real-world scenarios.

The 1024-token limit, though seemingly restrictive, does not impede the performance of the aligned fine-tuned small-scale models. All fine-tuned models exhibit robust performances across the experiments, as they are trained and evaluated on data predominantly falling within the 1024-token boundary. This outcome corroborates our analysis of the data and demonstrates the efficacy of QLoRA, even within the constraints of our allo-
Figure 8: Performance of Gemma 2B fine-tuned on varied volumes of synthetic dataset produced by various service LLMs including GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash. The first to third columns represent the performance of the model evaluated by GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash as judges, respectively. The first row show mean scores, while the second and third rows show the coverage percentage with 50 and 70 scores, respectively.

cated computational budget.

C More Experimental Results

The performance of Gemma 2B fine-tuned on varied volumes of synthetic dataset produced by various service LLMs including GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash is shown in Figure 8.

D Case Study

This section delves into detailed case studies showcasing the enhanced capabilities of the aligned small-scale local LLMs. We use Gemma 2B and 7B models as examples to illustrate.

The cases (Figure 9-17) illustrate the performances of the aligned models across summarization, classification, coding, and closed QA tasks. Specifically, these models are tuned on distinct 128K datasets generated by GPT4o for each corresponding task. Each case provides evaluations by GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash, offering a comprehensive assessment of the precision and similarity of the models’ responses.

To expand the scope of our analysis, we include two additional cases (Figure 11 and 12) to explore the summarization capabilities of the Gemma 2B and 7B models tuned with 256K synthetic datasets. These datasets are generated by GPT4o, Claude 3 Sonnet, and Gemini 1.5 Flash respectively, providing valuable insights into the models’ adaptability to different training data sources.

The cases presented above demonstrate the capability of the aligned Gemma 2B and 7B models to produce high-quality responses. Additionally, the cases offer insight into how different service LLMs evaluate text. Through this comparative lens, we reveal discernible variances in judgment and assessment criteria, enriching our understanding of the models’ operational dynamics.
Prompt
Summarize this paragraph. Do not mention the Mosquirix vaccine.

The Matrix-M vaccine was approved in late 2022. It also targets the P. falciparum parasite. However, it contains a chemical substance that stimulates the body’s immune response. This makes it more effective than Mosquirix. The initial trial involved 450 children, aged 5-17 months. The children received three doses, followed by a booster a year later. The study found that the vaccine reduced the number of malaria cases by 80 percent.

Target response
In late 2022, the Matrix-M vaccine, which includes a chemical that stimulates the body’s immune response, was approved. An initial study found the vaccine reduced malaria cases by 80% after 450 children between 5-17 months received three doses and a booster shot during a trial.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>63.0</td>
    <td>95.1</td>
    <td>79.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>77.5</td>
    <td>89.8</td>
    <td>64.0</td>
  </tr>
</table>

Gemma 2B (aligned on GPT4o generated 128k dataset)
The vaccine, Matrix-M, targets P. falciparum and stimulates the immune system, making it more effective than Mosquirix. The initial trial with 450 children showed a 80% reduction in malaria cases.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>55.0</td>
    <td>90.0</td>
    <td>77.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>76.5</td>
    <td>87.5</td>
    <td>68.0</td>
  </tr>
</table>

Gemma 7B (aligned on GPT4o generated 128k dataset)
The Matrix-M vaccine, approved in late 2022, targets both malaria and the P. falciparum parasite. It uses a chemical substance to boost immunity, making it more effective than the Mosquirix vaccine. Tested on 450 children aged 5-17 months, the vaccine reduced malaria cases by 80 percent after three doses and a booster one year later.

Figure 9: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for summarization task.

Prompt
Read this article and give me a description of Eagleman.

Eagleman is thirty-nine now and an assistant professor of neuroscience at Baylor College of Medicine, in Houston. Physically, he seems no worse for the fall. He did a belly flop on the bricks, he says, and his nose took most of the impact. “He made a one-point landing,” as his father puts it. The cartilage was so badly smashed that an emergency-room surgeon had to remove it all, leaving Eagleman with a rubbery proboscis that he could bend in any direction. But it stiffened up eventually, and it’s hard to tell that it was ever injured. Eagleman has puckish, neatly carved features, with a lantern jaw and modish sideburns. In Baylor’s lab-coated corridors, he wears designer jeans and square-toed ankle boots, and walks with a bounce in his step that’s suspiciously close to a strut, like Pinocchio leaving off to Pleasure Island.

If Eagleman’s body bears no marks of his childhood accident, his mind has been deeply imprinted by it. He is a man obsessed by time. As the head of a lab at Baylor, Eagleman has spent the past decade tracing the neural and psychological circuitry of the brain’s biological clock, and he has the good fortune to arrive in his field at the same time as fMRI scanners, which allow neuroscientists to observe the brain at work, in the act of thinking. But his best results have often come through more inventive means: video games, optical illusions, physical challenges. Eagleman has a talent for testing the untestable, for taking seemingly sophomoric notions and using them to nail down the slippery stuff of consciousness. “There are an infinite number of boring things to do in science,” he told me. “But we live these short life spans. Why not do the thing that’s the coolest thing in the world to do?”

Target response
Eagleman is 39 with puckish features including a lantern jaw, styled sideburns, flashy jeans and ankle boots. Physically, he has completely recovered from his childhood fall that broke his nose. He is obsessed with time and is an assistant professor of neuroscience at Baylor College of Medicine.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>35.5</td>
    <td>42.5</td>
    <td>53.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>29.5</td>
    <td>26.5</td>
    <td>20.0</td>
  </tr>
</table>

Gemma 2B (aligned on GPT4o generated 128k dataset)
Eagleman is a 39-year-old neuro-sci professor with a rubber nose, who used to be a professional scuba diver. He has excellent recall of marine life, which is useful in his research. His goal is to understand the biological clocks (the brain’s internal clock) and how they were modified during evolution. He uses video games, physical challenges and video games to learn more about consciousness. His work aims to make the brain accessible to scientists who might not have the same training as neuro-sci professionals.

Gemma 7B (aligned on GPT4o generated 128k dataset)
Dr. Eagleman is a thirty-nine-year-old professor of neuroscience known for his innovative research on the brain’s biological clocks. Despite a severe nose injury as a child, Eagleman has made a remarkable recovery. His sharp features and interest in ‘cool’ experiments reflect his personality.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>82.5</td>
    <td>90.5</td>
    <td>74.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>71.7</td>
    <td>82.5</td>
    <td>49.5</td>
  </tr>
</table>

Figure 10: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for summarization task.
Prompt
Please summarize this post. Make it easier to read and much shorter. It should only be about a paragraph or two. Use the text following this request. After World War II, designing the start and end dates for daylight saving time fell to state governments. Because this created many railroad scheduling and safety problems, Congress passed the Uniform Time Act in 1966. This law set the nationwide dates of daylight saving time from the last Sunday in April to the last Sunday in October. In 2005, the Energy Policy Act extended the act’s reach, so that daylight saving time is in effect from the second Sunday in March to the first Sunday in November – dates that remain in effect today. The Uniform Time Act allows states and territories to opt out of daylight saving time, however, Arizona and Hawaii are on permanent standard time, along with Puerto Rico, some U.S. Virgin Islands, and the Mariana Islands. Each year, several states and territories have tried to pass laws to stop falling back and springing ahead. Several U.S. states have legislation and resolutions under consideration to support permanent standard time, while other states are considering making daylight saving time permanent. In 2022, the number of states and territories seeking permanent standard time increased from 15% in 2021 to 31% in 2023. In March 2022, the U.S. Senate passed the Sunshine Protection Act, a bill to make daylight saving time permanent. But the House did not move forward with this legislation. Florida Sen. Marco Rubio reintroduced the bill on March 1, 2023. The spike in 2023 among states seeking to break from these twice-yearly board meetings reflects how more people are recognizing the needlessness of this practice. Now, it’s up to legislators to decide whether or not the time shift altogether, and to choose permanent standard or daylight saving time.

Target response
After World War II, Congress made daylight saving time a national holiday from the last Sunday in April to the last Sunday in October. In 2007, the Uniform Time Act was amended to extend the daylight saving time period from the second Sunday in March to the first Sunday in November. Congress has also allow state governments to opt out. Several U.S. states are considering or have considered legislation for permanent standard time, driven by opposition to daylight saving time. A similar bill was passed by the Senate in March 2022 but did not advance in the House, prompting renewed interest in the issue.

Gemma 2B (aligned on GPT4o generated 256k dataset)
After World, states were given the right to choose their own start and end dates for daylight saving time, but this caused several problems. To try to address this, Congress passed the Uniform Time Act in 1966 which standardized the changes, but still allowed states and territories to opt out entirely. Arizona, Hawaii, Puerto Rico, and some U.S. territories decided to stay on permanent time. Many other states and territories have been considering abolishing daylight saving time, but no bill has passed. People are starting to recognize the problems caused by changing the time twice a year and want to move to a permanent system.

Gemma 2B (aligned on Claude 3 Sonnet generated 256k dataset)
The key points regarding the Uniform Time Act and Daylight Saving Time are:

1) In 1966, the Uniform Time Act established the dates of daylight saving time nationwide.

2) States can opt out of daylight saving time under this Act, and there are currently 31 states on permanent standard time or permanent daylight saving time.

3) Multiple states are considering legislation to remove the time shift from our annual cycle of timekeeping to permanent standard time or permanent daylight saving time, reflecting increasing public opposition to the current system.

4) The Sunshine Protection Act passed by the U.S. Senate in 2022 proposed continuing daylight saving time but sought to make it permanent.

Gemma 2B (aligned on Gemini 1.5 Flash generated 256k dataset)
Congress passed the Uniform Time Act in 1966 to standardize daylight saving time across the country. However, the law’s focus on railroad safety led to practical issues. The act was amended in 2007 to extend the dates of daylight saving time. While states can opt out of the rules, many are moving toward permanent standard time. The Sunshine Protection Act, which would make daylight saving time permanent, has been reintroduced in recent years. This shift toward a more standardized time system reflects growing awareness of its downsides.7

Gemma 7B (aligned on GPT4o generated 256k dataset)
World War II led to inconsistent dates for daylight saving time until Congress passed the Uniform Time Act in 1966, setting the dates from the last Sunday in April to the last Sunday in October. In 2007, the dates were expanded to the second Sunday in March and first Sunday in November. The Uniform Time Act also allows states to opt out of daylight saving time, with Arizona, Hawaii, Puerto Rico, U.S. Virgin Islands, Northern Mariana Islands, and American Samoa remaining on permanent standard time. Several states are considering making daylight saving time permanent. The U.S. Senate passed the Sunshine Protection Act in 2022 to make daylight saving time permanent, Sen. Marco Rubio reintroduced the bill in 2023. The growing number of states seeking to abolish daylight saving time highlights the growing awareness of daylight saving time’s drawbacks. Ultimately, it’s up to legislators to decide whether we should end the practice of changing time or make daylight saving time permanent.

Gemma 7B (aligned on Claude 3 Sonnet generated 256k dataset)
The Uniform Time Act of 1966 established the dates for daylight saving time nationwide. It currently runs from March to November. However, states and territories can opt out and permanently remain on standard time. Some states have passed laws to opt out of daylight saving time, but others are considering changing the dates to end the twice-yearly shifting. States like Connecticut and New Hampshire have passed resolutions supporting permanent standard time. Meanwhile, legislation has been introduced in Florida and other states to make daylight saving time permanent. The U.S. Senate passed the Sunshine Protection Act in 2022 to make daylight saving time permanent, but the House did not take action. This increasing momentum reflects growing opposition to permanent daylight saving time among states, businesses, and the general public. Ultimately, it’s up to legislators to decide whether we should choose permanent standard or daylight saving time, or if states will make the decision themselves.

Gemma 7B (aligned on Gemini 1.5 Flash generated 256k dataset)
The United States has a long and complicated history with Daylight Saving Time. Its origins trace back to World War I, when Germany implemented it as a way to conserve energy. The United States followed suit in 1918, but the practice was abandoned after the war. Daylight Saving Time was re-introduced during World War II, leading to inconsistencies. In 1966, Congress passed the Uniform Time Act, standardizing the dates for Daylight Saving Time. The act remains in effect today, although the dates have been expanded in 2007. Several states have passed laws to opt out of daylight saving time, but others are considering changing the dates to end the twice-yearly shifting. States like Connecticut and New Hampshire have passed resolutions supporting permanent standard time. Meanwhile, legislation has been introduced in Florida and other states to make daylight saving time permanent. The U.S. Senate passed the Sunshine Protection Act in 2022 to make daylight saving time permanent, but the House did not take action. This increasing momentum reflects growing opposition to permanent daylight saving time among states, businesses, and the general public. Ultimately, it’s up to legislators to decide whether we should choose permanent standard or daylight saving time, or if states will make the decision themselves.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>59.5</td>
    <td>84.0</td>
    <td>72.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>55.0</td>
    <td>73.8</td>
    <td>49.0</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>71.8</td>
    <td>90.5</td>
    <td>72.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>63.5</td>
    <td>83.0</td>
    <td>41.5</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>84.5</td>
    <td>86.5</td>
    <td>76.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>76.8</td>
    <td>76.0</td>
    <td>57.0</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>84.5</td>
    <td>93.5</td>
    <td>85.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>74.2</td>
    <td>86.7</td>
    <td>67.5</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>80.5</td>
    <td>88.5</td>
    <td>73.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>71.3</td>
    <td>78.5</td>
    <td>43.5</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>61.0</td>
    <td>88.5</td>
    <td>68.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>48.0</td>
    <td>77.0</td>
    <td>36.5</td>
  </tr>
</table>

Figure 11: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o, Claude 3 Sonnet, Gemini 1.5 Flash generated 256K of synthetic datasets for summarization task.
Prompt
Summarize the following story in a few sentences.

The Fox one day thought of a plan to amuse himself at the expense of the Stork, at whose odd appearance he was always laughing. "You must come and dine with me today," he said to the Stork, smiling to himself at the trick he was going to play. The Stork gladly accepted the invitation and arrived in good time and with a very good appetite. For dinner the Fox served soup. But it was set out in a very shallow dish, and all the Stork could do was to wet the very tip of his bill. Not a drop of soup could he get. But the Fox lapped it up easily, and, to increase the disappointment of the Stork, made a great show of enjoyment.
The hungry Stork was much displeased at the trick, but he was a calm, even-tempered fellow and saw no good in flying into a rage. Instead, not long afterward, he invited the Fox to dine with him in turn. The Fox arrived promptly at the time that had been set, and the Stork served a fish dinner that had a very appetizing smell. But it was served in a tall jar with a very narrow neck. The Stork could easily get at the food with his long bill, but all the Fox could do was to lick the outside of the jar, and sniff at the delicious odor. And when the Fox lost his temper, the Stork said calmly:
Do not play tricks on your neighbors unless you can stand the same treatment yourself.

Target response
The Fox always judged the Stork's appearance and one day decided to pull a prank on him. The Fox invited the Stork over for dinner and served him soup in a very shallow dish. The Stork was not able to get a single drop of the soup and, even though he was hungry, he stayed calm. Later, the Stork invited the Fox over for dinner and served him a delicious-smelling fish dinner, but it was in a tall jar with a very narrow neck. The Fox was only able to lick the outside of the jar, and he lost his temper.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <td>Gemma 2B (aligned on GPT4o generated 256k dataset)</td>
    <td>Precision scores<br>56.5<br><br>Similarity scores<br>42.5</td>
    <td>Precision scores<br>58.0<br><br>Similarity scores<br>33.5</td>
    <td>Precision scores<br>65.5<br><br>Similarity scores<br>32.0</td>
  </tr>
  <tr>
    <td>Gemma 2B (aligned on Claude 3 Sonnet generated 256k dataset)</td>
    <td>Precision scores<br>88.0<br><br>Similarity scores<br>83.5</td>
    <td>Precision scores<br>89.0<br><br>Similarity scores<br>80.0</td>
    <td>Precision scores<br>81.0<br><br>Similarity scores<br>64.5</td>
  </tr>
  <tr>
    <td>Gemma 2B (aligned on Gemini 1.5 Flash generated 256k dataset)</td>
    <td>Precision scores<br>66.5<br><br>Similarity scores<br>67.0</td>
    <td>Precision scores<br>78.5<br><br>Similarity scores<br>69.0</td>
    <td>Precision scores<br>74.5<br><br>Similarity scores<br>57.0</td>
  </tr>
  <tr>
    <td>Gemma 7B (aligned on GPT4o generated 256k dataset)</td>
    <td>Precision scores<br>84.5<br><br>Similarity scores<br>74.2</td>
    <td>Precision scores<br>93.5<br><br>Similarity scores<br>86.7</td>
    <td>Precision scores<br>85.5<br><br>Similarity scores<br>67.5</td>
  </tr>
  <tr>
    <td>Gemma 7B (aligned on Claude 3 Sonnet generated 256k dataset)</td>
    <td>Precision scores<br>80.5<br><br>Similarity scores<br>71.3</td>
    <td>Precision scores<br>88.5<br><br>Similarity scores<br>78.5</td>
    <td>Precision scores<br>73.5<br><br>Similarity scores<br>43.5</td>
  </tr>
  <tr>
    <td>Gemma 7B (aligned on Gemini 1.5 Flash generated 256k dataset)</td>
    <td>Precision scores<br>61.0<br><br>Similarity scores<br>48.0</td>
    <td>Precision scores<br>88.5<br><br>Similarity scores<br>77.0</td>
    <td>Precision scores<br>68.0<br><br>Similarity scores<br>36.5</td>
  </tr>
</table>

Figure 12: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o, Claude 3 Sonnet, Gemini 1.5 Flash generated 256K of synthetic datasets for summarization task.
Prompt
Would you classify this email as an advertisement or an account update?

Only 4 hours left in 2020 to make your support known for public radio in the Ozarks! As always, your year-end contribution is tax-deductible.

Don't wait until the ball drops to make your donation...give now, and enjoy the countdown to midnight knowing you've done your part to help the programs you rely on thrive in the year to come.

Target response
This email is an advertisement.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Gemma 2B (aligned on GPT4o generated 128k dataset)</th>
    <td colspan="3">
      This email is an advertisement. It is asking supporters to make a donation to public radio before midnight. This is intended to persuade supporters to make a donation which is the purpose of an advertisement.
    </td>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>95.5</td>
    <td>94.5</td>
    <td>93.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>87.5</td>
    <td>91.0</td>
    <td>81.0</td>
  </tr>
  <tr>
    <th>Gemma 7B (aligned on GPT4o generated 128k dataset)</th>
    <td colspan="3">This email is an advertisement.</td>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>99.5</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>100.0</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
</table>

Figure 13: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for classification task.

Prompt
which of these songs are titled in english and which are titled in italian?

<table>
  <tr>
    <th>No.</th>
    <th>Title</th>
    <th>Length</th>
    <th></th>
  </tr>
  <tr>
    <td>1.</td>
    <td>"Zitti e buoni"</td>
    <td>3:14</td>
    <td></td>
  </tr>
  <tr>
    <td>2.</td>
    <td>"Livi di sui gomiti"</td>
    <td>2:45</td>
    <td></td>
  </tr>
  <tr>
    <td>3.</td>
    <td>"I Wanna Be Your Slave"</td>
    <td>2:53</td>
    <td></td>
  </tr>
  <tr>
    <td>4.</td>
    <td>"In nome del padre"</td>
    <td>3:39</td>
    <td></td>
  </tr>
  <tr>
    <td>5.</td>
    <td>"For Your Love"</td>
    <td>3:50</td>
    <td></td>
  </tr>
  <tr>
    <td>6.</td>
    <td>"La paura del bulo"</td>
    <td>3:29</td>
    <td></td>
  </tr>
  <tr>
    <td>7.</td>
    <td>"Vent'anni"</td>
    <td>4:13</td>
    <td></td>
  </tr>
</table>

Target response
1. Italian
2. Italian
3. English
4. Italian
5. English
6. Italian
7. Italian

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Gemma 2B (aligned on GPT4o generated 128k dataset)</th>
    <td colspan="3">
      <ol>
        <li>"Zitti e buoni" 3 Italian</li>
        <li>"Livi di sui gomiti" 2 Italian</li>
        <li>"I Wanna Be Your Slave" 2 Italian</li>
        <li>"In nome del padre" Italian</li>
        <li>"For Your Love" 3 Italian</li>
        <li>"La paura del bulo" 3 Italian</li>
        <li>"Vent'anni" 4 Italian</li>
      </ol>
    </td>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>18.5</td>
    <td>82.0</td>
    <td>63.6</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>20.8</td>
    <td>67.0</td>
    <td>50.0</td>
  </tr>
  <tr>
    <th>Gemma 7B (aligned on GPT4o generated 128k dataset)</th>
    <td colspan="3">
      <ol>
        <li>Zitti e buoni - Italian</li>
        <li>Livi di sui gomiti - Italian</li>
        <li>I Wanna Be Your Slave - English</li>
        <li>In nome del padre - Italian</li>
        <li>For Your Love - English</li>
        <li>La paura del bulo - Italian</li>
        <li>Vent'anni - Italian</li>
      </ol>
    </td>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>98.9</td>
    <td>100.0</td>
    <td>99.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>96.7</td>
    <td>100.0</td>
    <td>93.3</td>
  </tr>
</table>

Figure 14: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for classification task.
Prompt
How do I listen for an event in Discord.js?

Target response
You can listen for an event in Discord.js by using the Client#on method:

const { Events, Client } = require("discord.js");
const client = new Client();
client.on(Events.ClientReady, c => {
    console.log("Client is ready.");
});

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>68.0</td>
    <td>59.5</td>
    <td>82.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>63.5</td>
    <td>38.0</td>
    <td>56.0</td>
  </tr>
</table>

Gemma 2B (aligned on GPT4o generated 128k dataset)
To listen for an event in Discord.js, you need to use the 'listen' method. For example, to listen for a 'messageCreate' event, you can do the following:

```javascript
const Discord = require('discord.js');
const bot = new Discord.Client();

bot.on('messageCreate', (message) => {
    if (message.content === 'Hello') {
        message.reply('Hello there!');
    }
});
bot.login('yourTokenHere');
```

Gemma 7B (aligned on GPT4o generated 128k dataset)
To listen for an event in Discord.js, you use the `client.on` method. Here's an example of how to listen for a message event:

```javascript
const Discord = require('discord.js');
const client = new Discord.Client();

client.on('message', (message) => {
    if (message.author.id === 'your_id_here') {
        console.log(`Message from ${message.author.tag}: ${message.content}`);
    }
});
client.login('your_token_here');
```

Figure 15: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for coding task.
Prompt
This Java function below is not working as I expected, it's supposed to go through an array of numbers and if it finds a number in the array that is divisible by 2 with a remainder of 0, then it adds it to a new array and returns the new array at the end. Can you help debug the function and explain how why it's not working as expected?

public static List checkForEvens(int[] nums) {
    ArrayList answer = new ArrayList<Integer>();
    for(int i = 0; i < nums.length; i++) {
        if(nums[i] / 2 == 0) {
            answer.add(nums[i])
        }
    }
}

Target response
Certainly! The function 'checkForEvens' iterates through an array and if a number in the array is divisible by 2, then it gets added to 'answer' which is later returned.

The reason the function may not be returning the expected result is because of the if statement, 'if (nums[i] / 2 == 0)'. The statement is looking for numbers that get divided by 2 in the array and equal 0, which doesn't align with what you expect the function to do.

Instead, you want to use the modulo operator to check for the remainder of a number that is being divided by 2. If a number is divided by 2 and returns a remainder of 0, then we can assume that it's an even number. So, the small change of 'if (nums[i] % 2 == 0)' should solve your issue.

I've rewritten the function below with the correction.

public static List checkForEvens(int[] nums) {
    ArrayList answer = new ArrayList<Integer>();
    for(int i = 0; i < nums.length; i++) {
        if(nums[i] % 2 == 0) {
            answer.add(nums[i])
        }
    }
}

Gemma 2B (aligned on GPT4o generated 128k dataset)
Sure, I can help with that. The provided Java function is nearly correct but it does not handle cases where the number is negative. Here's a more robust version that includes handling edge cases.

```java
public class CheckForEvens {
    public static List<Integer> checkForEvens(int[] nums) {
        List<Integer> answer = new ArrayList<>();
        for (int num : nums) {
            if ((num / 2 - 1) % 2 == 0) {
                answer.add(num);
            }
        }
        return answer;
    }
}
```

This function first creates an answer list, then iterates through each number in the array. It checks if the number divided by 2 (with an exclusive or of 1) is even. If it is, it adds the number to the answer list.

Gemma 7B (aligned on GPT4o generated 128k dataset)
The issue with your function is that the modulus operator (%) is missing in the condition to check if the number is even. You should use (nums[i] % 2 == 0) instead of (nums[i] / 2 == 0).

Here's the corrected function:

public static List checkForEvens(int[] nums) {
    ArrayList answer = new ArrayList<>();
    for (int i = 0; i < nums.length; i++) {
        if (nums[i] % 2 == 0) {
            answer.add(nums[i]);
        }
    }
    return answer;
}

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>19.0</td>
    <td>45.0</td>
    <td>35.0</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>27.0</td>
    <td>25.0</td>
    <td>19.0</td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Precision scores</th>
    <td>92.8</td>
    <td>99.5</td>
    <td>95.5</td>
  </tr>
  <tr>
    <th>Similarity scores</th>
    <td>85.0</td>
    <td>95.0</td>
    <td>89.0</td>
  </tr>
</table>

Figure 16: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for coding task.
Prompt
Is the following statement true or false? Keeping a plant healthy reduces the risk of pest infestations.

Use the following passage to answer the question.

How to Treat Yellowing Leaves on Snake Plant
Once you narrow down the reason your snake plant has yellow leaves, you’ll want to address the issue quickly to keep your plant healthy. However, yellow leaves won’t turn green again even if the issue is remedied, so it’s best to cut them off with sharp, sanitized pruning shears at the base of the leaf.

Check your plant’s soil moisture. If the soil feels soggy and you’ve been watering frequently, overwatering could be to blame. Let the soil dry out completely before watering again. In extreme cases, you may need to repot the plant in fresh soil and inspect the roots for signs of rot.

If your snake plant’s leaves look sunburned and the plant is getting direct sunlight, move it further away from the light source or use a sheer curtain to filter the light and make it less intense. Address inconsistent temperature by keeping the plant away from drafty areas and putting it in a space with a more consistent temperature.

Inspect your snake plant regularly for signs of pests. Remove insects with blasts of water or a cotton swab dipped in rubbing alcohol, then use neem oil or horticultural soap to prevent them from coming back. Keep your plant healthy going forward to lessen the likelihood that pests will attack.

If you haven’t fertilized or refreshed your snake plant’s soil in a few years and other causes don’t apply, repot it with fresh soil. If the plant seems potbound, use a pot one size larger or divide the plant and repot it in its original container. If you suspect overfertilizing is the issue, flush your plant’s soil several times with water to remove excess nutrients. Going forward, only fertilize your snake plant twice per year in spring and summer.

Target response
The statement is true.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Gemma 2B (aligned on GPT4o generated 128k dataset)</th>
    <td>Precision scores<br>11.5</td>
    <td>68.0</td>
    <td>10.0</td>
  </tr>
  <tr>
    <td></td>
    <td>Similarity scores<br>3.6</td>
    <td>0.1</td>
    <td>1.0</td>
  </tr>
  <tr>
    <th>Gemma 7B (aligned on GPT4o generated 128k dataset)</th>
    <td>Precision scores<br>98.8</td>
    <td>99.5</td>
    <td>95.9</td>
  </tr>
  <tr>
    <td></td>
    <td>Similarity scores<br>95.0</td>
    <td>99.5</td>
    <td>90.0</td>
  </tr>
</table>

Figure 17: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for closed QA task.

Prompt
Who did Prince William marry?

The queen’s popularity, and that of the entire royal family, rebounded during the first decade of the 21st century. Though 2002 marked Queen Elizabeth’s Golden Jubilee—50 years on the throne—the death of her mother (the beloved Queen Mum) and sister early that year cast a pall on the celebrations.

In 2005, the queen enjoyed public support when she gave her assent to Prince Charles’ once-unthinkable marriage to his longtime love Camilla Parker Bowles.

In her seventh decade on the throne, Queen Elizabeth presided over the pomp and circumstance of another royal wedding at Westminster Abbey, that of Prince William to Catherine Middleton in April 2011. The Duke and Duchess of Cambridge, who are in line to become Britain’s next king and queen, continued the line of succession with their children, Prince George (born 2013), Princess Charlotte (born 2015) and Prince Louis (born 2018).

Target response
Prince William married Catherine Middleton.

<table>
  <tr>
    <th></th>
    <th>GPT4o</th>
    <th>Claude 3 Sonnet</th>
    <th>Gemini 1.5 Flash</th>
  </tr>
  <tr>
    <th>Gemma 2B (aligned on GPT4o generated 128k dataset)</th>
    <td>Precision scores<br>95.4</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td></td>
    <td>Similarity scores<br>96.1</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <th>Gemma 7B (aligned on GPT4o generated 128k dataset)</th>
    <td>Precision scores<br>98.8</td>
    <td>99.5</td>
    <td>95.9</td>
  </tr>
  <tr>
    <td></td>
    <td>Similarity scores<br>95.0</td>
    <td>99.5</td>
    <td>90.0</td>
  </tr>
</table>

Figure 18: Responses by Gemma 2B and Gemma 7B models fine-tuned on GPT4o generated 128K synthetic dataset for closed QA task.

Reviewer paper summaries: This work introduces an automated LLMOps pipeline called "LlamaDuo" that allows organizations and teams to migrate away from cloud-based LLM providers to smaller, locally deployed models. To achieve this, they utilize synthetic data generated by service providers like OpenAI, Google, and Anthropic to fine-tune local models and evaluate performance using LLMs as judges. If performance is not upto expectations, an iterative process begins where models are finetuned till they match or exceed the service LLM capability for pre-decided downstream tasks.

This paper introduces **LlamaDuo**, a pipeline that addresses the challenges of relying on cloud-based closed-source Large Language Models (LLMs). LlamaDuo enables the transfer of knowledge from service-based models like ChatGPT to smaller and local Language Models like Gemma by fine-tuning with synthetic data generated by the web LLM.

This paper proposes an automatic pipeline to migrate service-oriented LLM to smaller, locally manageable models for downstream tasks, in order to avoid privacy, security, and high cost. The authors first collect user input prompt and satisfied responses of service LLMs as coverage dataset, then fine-tune the local LLM based on the coverage dataset, then local model inference several responses to the test-set from the coverage dataset, afterward, use service LLM again to evaluate these responses to get average evaluation scores, then use service LLM to synthesize new datasets and fine-tune the local LLM on consequently synthetic dataset until it reaches evaluation requirement.
Summary of strengths: 1. **Comprehensive experimentation:** The authors conduct extensive experiments across multiple service LLMs (GPT4o, Claude 3 Sonnet, Gemini 1.5 Flash), local LLMs (Gemma 2B/7B, Mistral 7B, LLaMA3 8B), and diverse tasks (summarization, classification, coding, closed QA). They also observe the efficacy of using different models as judges and the difference in scoring. 

2. **Real-life relevance:** The paper addresses real-world challenges of dependency on cloud LLMs (privacy concerns, operational dependencies, costs) with a solution that has practical applications and an easy-to-use deployment pipeline.

3. **Multiple examples in the appendix:** The work showcases multiple models trained on different datasets for different tasks in the appendix. This helps the reader notice the differences and make an informed decision on the efficacy of this method for various downstream tasks.

This paper has the following strengths:
- The paper is well-written and very easy to understand.
- The proposed framework is very effective and easy to use.

1. The pipeline sounds simple and effective, and helps the migration of service LLM to local LLM.
2. Methods are stated clearly.
3. Experiments are conducted thoroughly.
Summary of weaknesses: 1. **Limited methodological novelty**: The core technical components (fine-tuning on synthetic data, using LLMs as judges) have been explored in prior work. The paper's main contribution is combining these into a unified pipeline rather than introducing fundamentally new techniques or iterating upon previous work. 

2. **Circular dependencies:** The pipeline depends on service LLMs for both synthetic data generation and evaluation, creating a circular dependency on the very systems it aims to replace (at least during development). LLM judges are also biased towards their own generation. The work does not explore this i.e. what happens when Gemini scores models trained on Gemini's synthetic dataset. The local model is also heavily dependent on synthetic data, it would be more prudent to train the models on a mixture of data i.e. 240K randomly selected data points from Claude, Gemini, and GPT to further mitigate biases. 

3. **Knowledge cutoff issues:** A local model was pre-trained on data up to a certain date but the service LLM has more recent knowledge, how can you ensure the fine-tuned model doesn't generate outdated/hallucinated information while still matching the service LLM's capabilities? Is catastrophic forgetting avoided? [1]

**References**
[1] Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., & Zhang, Y. (2025, January 5). An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv.org. https://arxiv.org/abs/2308.08747

One major weakness is the multiple supervised fine-tuning rounds, which could make the training time a little longer.

1. The downstream tasks are limited. The authors use an existing dataset as a coverage dataset and focus on four general downstream tasks. Realistic tasks should be included.
Comments, suggestions and typos: A much-needed contribution of this work is the massive synthetically generated dataset. I'm unable to see if there is any inclusion or discussion of releasing the dataset post-anonymity period. I would urge the authors to add the dataset as a valuable contribution to the open-source community.

I have a few suggestions:
- Include a table or figure illustrating the running time needed for different scale models.
- Try the lasted small-scale models like LLaMA3.2 1B and 3B.
- It would be better for the authors to try some reinforcement learning techniques like GRPO. And to observe the difference between using SFT and RL. It is also worthing exploring combing SFT and RL together.

1. Even though the local model can reduce the cost, it also reduces its capability to handle other tasks. What would happen when more than one downstream tasks are needed? I doubt that it will take even longer for the benefits of deploying an LLM locally to outweigh its costs.
2. Does synthesis datasets bring more hallucination?
Overall assessment scores: 3

4

3.5
Metareview score: 3
Metareview: This paper presents a simulation of fine-tuning small open-weight language models (i.e., with 7–8B parameters) using synthetic data generated by commercial models. The primary motivation is to reduce both the cost and dependency on closed-weight models. To enhance pretraining, the authors extend the synthetic data by using part of it as seed data.

While the proposed method is not particularly novel from a technical standpoint, the task it addresses is significant. The authors apply LoRA-based pretraining and demonstrate that smaller models can perform on par with—or even outperform—larger models in certain tasks. The spirit of this work is reminiscent of the original Alpaca dataset released alongside ChatGPT.

The experiments are thorough, covering multiple models and tasks, which makes the results both convincing and promising. Based on the current reviews and my understanding of the paper, I believe it is a good fit for Findings.
Summary of reasons to publish: * The paper addresses an important and timely issue—reducing reliance on closed-weight models by fine-tuning smaller, open-weight models. It proposes a practical approach to reducing the cost of large language model development by using synthetic data.

* The experiments are comprehensive, involving various models and tasks, which strengthens the credibility of the results. The smaller model, when fine-tuned using the proposed method, performs comparably or even better than larger models.