Generating Synthetic Datasets for Few-shot Prompt Tuning

Authors: xu  guo
zilin  du
boyang  li
chunyan  miao
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 17:26:27

Abstract: A major limitation of prompt tuning is its dependence on large labeled training datasets. Under few-shot learning settings, prompt tuning lags far behind full-model fine-tuning, limiting its scope of application. In this paper, we leverage the powerful LLMs to synthesize task-specific labeled data for training the soft prompts. We first introduce a distribution-aligned weighted generator tuning (DawGen) method to encourage generating in-distribution data that aligns with the few-shot real data. Then, we train soft prompts on both synthetic and real datasets using a gradient surgery approach, which eliminates the conflicting gradients from different data sources. Experiments on seven sentence-pair classification datasets demonstrate the effectiveness of our proposed method for boosting prompt tuning in few-shot learning settings. Results on QQP, MRPC, and SICK datasets are even comparable to the performance of transfer learning from large real-world datasets, showing the promise of synthetic data as an alternative for enhancing soft prompt tuning.

Main paper: 
 introduction: 
 As Large Language Models (LLMs) increase in size, adapting them to downstream tasks by fine-tuning (FT) a separate copy for each task becomes unfeasible. Prompt Tuning (Lester et al., 2021) (PT) emerges as a solution to this challenge by freezing the LLM and instead training a set of soft prompts pre-pended to the input data in an end-to-end manner. Compared with other parameter-efficient learning methods such as adapter tuning (Houlsby et al., 2019) and LoRA (Hu et al., 2022) , PT makes no changes to the model architecture and can be applied to a frozen model with a static computational graph, enabling fast and flexible deployment. On a wide range of downstream tasks, PT has shown comparable performance as FT (Lester et al., 2021; Liu et al., 2022) . However, recent studies indicate that PT requires sufficient labeled training data to achieve competitive performance as FT, yet in few-shot settings, PT significantly underperforms FT (Gu et al., 2022; Guo et al., 2022) .To boost PT in few-shot learning tasks, previous methods mainly focus on finding a better initialization for soft prompts (Gu et al., 2022; Guo et al., 2022) . This is achieved by pretraining the soft prompts on a large-scale real-world corpus such as OpenWebText (Gokaslan & Cohen, 2019a) or similar source-domain datasets (Gu et al., 2022; Vu et al., 2022; Guo et al., 2022) . However, these approaches bear a common limitation: the dependence on large real-world datasets. On the one hand, online text corpora often exhibit substantial domain discrepancies varying across different downstream tasks. On the other hand, source-domain datasets are often not readily available, particularly for low-resource and emerging domains.Recently, there has been a growing interest in generating training data with LLMs (Ye et al., 2022a; Meng et al., 2022; 2023; Yu et al., 2024) . This paper extends this line of research to tackle the limitation of prompt tuning in few-shot learning settings, aiming to bypass the need for large-scale labeled training data. Specifically, we employ a source LLM to generate a synthetic training set for the task at hand, which can be treated as a medium for carrying the pre-learned knowledge from a source LLM, to train soft prompts for a target LLM to achieve enhanced few-shot learning performance. Step 1: traing a soft prompt Q for a frozen LLM g ϕ on the few-shot training set D real .Step 2: apply g ϕ and Q to generate a synthetic labeled training set D syn .Step 3: With D real and D syn , train a soft prompt P for a discriminative LLM f θ to perform downstream tasks, while tackling issues like gradient conflicts.We show an overview diagram of our framework in Figure 1 . We divide the learning procedure into three steps. First, to encourage the source LLM g ϕ to generate in-domain data, we train label-conditional soft prompts Q using a few real-world samples, D real , to adapt the frozen g ϕ to the task domain. However, given the limited number of real-world samples, Q can easily overfit shortcut tokens (Du et al., 2021; Tang et al., 2023) and mislead g ϕ to generate texts that are not relevant to the task label, resulting in a poor distribution for D syn . Hence, we introduce Distribution-Aligned Weighted GENerator tuning (DawGen1 ) which encourages the generated texts under the same class to be semantically close while those under different classes to be semantically distant. Second, we apply the adapted generator g ϕ,Q to generate a synthetic training set, D syn , for the task at hand. However, directly combining synthetic data with the few-shot real data for training can lead to an interference effect, due to their potential distribution gap. To better exploit the synthetic data while making the best use of few-shot real data, we employ gradient surgery (Yu et al., 2020) to modify the gradients computed on the synthetic data by subtracting the components that oppose the direction of gradients from the real data. The trained soft prompts followed by an input example are used to prompt the target LLM for label prediction. They can also be combined with hard prompts to confer dual benefits (Gao et al., 2021; Gu et al., 2022) .Extensive experiments on seven sentence-pair classification tasks and two LLM backbones demonstrate the effectiveness of our proposed framework for enhancing PT in few-shot settings. Notably, PT with 102K parameters outperforms FT with 770M parameters by a large margin in few-shot settings and can even achieve comparable performance to transfer learning using extensive real-world datasets on QQP, MRPC, and SICK. 
few-shot learning with pre-trained language models: 
 Fine-tuning pre-trained language models has been the standard practice in few-shot learning, where a language model and a task-specific head are tuned together for a given task (Zhang et al., 2021; Gao et al., 2021; Liu et al., 2022; Zhang et al., 2022) . However, fine-tuning the entire model on a few training samples (e.g., 16 samples per class) often leads to overfitting. One possible remedy is to manually craft hard prompts, consisting of natural language instructions and demonstrations (Brown et al., 2020; Mishra et al., 2022) for LLMs to perform in-context learning without updating any model parameters. Nevertheless, their effectiveness greatly relies on the skills of the prompt engineer and the prompts they write.Instead of manually crafted hard prompts, prompt tuning (Lester et al., 2021; Zhang et al., 2022) learns soft prompt vectors from training data and PT on T5-large can outperform the results of manual prompting on GPT-3. Still, PT performance greatly depends on the availability of substantial training data (Gu et al., 2022; Guo et al., 2022) . In few-shot settings, PT significantly underperforms FT. To address this limitation, researchers have studied utilizing online corpora to pre-train soft prompts under self-supervised learning objectives (Gu et al., 2022) , such as next sentence prediction (Devlin et al., 2019) , or applying transfer learning from datasets in similar domains (Guo et al., 2022) . This paper provides an alternative solution by employing a more powerful LLM to generate synthetic training data for training soft prompts, thus circumventing the demand for extensive real-world data.Early efforts in generating synthetic data with language models were to augment the existing dataset with generic texts (Kumar et al., 2020; Puri et al., 2020; Anaby-Tavor et al., 2020) . E.g., Puri et al. (2020) use GPT-2 to generate a synthetic corpus to substitute the Wikipedia corpus to improve the performance of QA tasks. These texts are not tailored to any downstream task and therefore can not be used to train task-specific models. As LLMs continue to grow in size, recent works have shifted towards creating a new paradigm for playing with LLMs, i.e., distilling task-specific training data directly from a frozen LLM to enhance downstream tasks (Schick & Sch ütze, 2021b; Ye et al., 2022a; Meng et al., 2022; Gao et al., 2023; Meng et al., 2023; Yu et al., 2024) . These synthetic datasets are then used to boost downstream models such as DistillBERT and RoBERTa (Meng et al., 2022; 2023) .However, when prompted with a simple label-conditional natural language prompt, the LLM generator can easily forget label information when generating long sequences (Li et al., 2022; Zhong et al., 2024) and therefore may generate samples that are not associated with the given labels. Moreover, the LLM generator, when frozen for inference, tends to generate texts that follow its pertaining data distribution, which often exhibits a domain gap with the task at hand (Guo & Yu, 2022) . This paper provides a distribution-aligned weighted generator tuning method to mitigate these issues.The existence of low-quality samples can be detrimental to model training. Synthetic datasets are often generated at scale, which is inevitable to contain low-quality data (Gao et al., 2023) . Recent works on synthetic text generation with LLMs (Ye et al., 2022a; Meng et al., 2022; 2023) adopt various training strategies to exploit the synthetic training data. For example, ZeroGen (Ye et al., 2022a; Gao et al., 2022) employs a noise-robust loss function (Ghosh et al., 2017) for learning with synthetic training data. FewGen (Meng et al., 2023) adopts label smoothing and temporal ensembling (Laine & Aila, 2016) to degrade the confidence level of model prediction during training. ProGen (Ye et al., 2022b) incorporates a quality estimation module to select the synthetic dataset. These methods are primarily employed to work with synthetic data; however, in our paper, where we use a few real samples for supervision, they cannot address the disparity between real and synthetic data. Hence, we propose to use the gradient surgery method (Yu et al., 2020) to directly alter the conflicting gradients on the synthetic data, thereby enhancing the learning performance. 
preliminaries: 
 Prompt Tuning. Lester et al. (2021) converts all downstream tasks into a text-to-text generation format (Raffel et al., 2020) in order to reduce the gap between pre-training and downstream tasks. Taking sentence-pair classification as an example, given a labeled training example (X, y) ∈ D, |D| = N, where X = [S 1 , S 2 ] represents a sentence pair and y ∈ Y denotes a class label. For example, in paraphrase detection, the label space Y may include two labels, yes and no, indicating if two sentences are paraphrases. In order to fully utilize the pretrained model, we devise a task-specific natural language prompt H and a template T (•) that reformat the original task as a cloze-style task. T (X) = {H, X, [MASK]}. One example for paraphase detection is "Are ⟨S 1 ⟩ and ⟨S 2 ⟩ equivalent? [MASK] ". An LLM, f θ , predicts the label y at the position of the [MASK] token.Prompt tuning is done by prepending n tokens with trainable prompt embeddings, P ∈ R n×d , to the template T . Throughout the training, θ remains unchanged and P is optimized by minimizing the cross-entropy loss on the training set D:EQUATIONSynthetic Text Generation. Taking sentence-pair classification as an example. Given the label space Y = {Y l } L l=1 , we compose a label-conditional prompt for each Y l using the template T c (Y l ) = {H, Y l , X}, e.g., "Sentence 1 and sentence 2 are equivalent. Sentence 1: ⟨S 1 ⟩. Sentence 2: ". Then, an autoregressive LLM, g ϕ , takes T c (Y l ) as the context and generates a subsequent sequence of tokens x 1:K that maximize the joint likelihood:K ∏ j=1 Pr ϕ (x j |x <j ; T c (Y l )).(2)The decoding process stops when the end-of-sentence token is predicted or the maximum sequence length, K, is reached. The generated sentence for S 2 is expected to satisfy the relationship Y l with S 1 . To encourage generating diverse X syn for the same label Y l , we employ a stochastic decoding algorithm (e.g., top-k and nucleus sampling). The generated sentence pairs and the given label y l establish a synthetic dataset D syn = {(X syn , y syn )}.For every downstream task, we use the few-shot real dataset, D real , to perform domain adaptation for the generator g ϕ to improve the quality of the synthetic dataset D syn .Parameter-Efficient Generator Tuning. Recent LLMs for text generation often have billions of parameters. As such, tuning the entire model ϕ for domain adaptation is impractical.Parameter-efficient methods, such as prompt tuning (Lester et al., 2021) and prefix tuning (Li & Liang, 2021) , arise as an alternative to full-model fine-tuning by pre-pending a few external prompt embeddings, Q, to the input (or every transformer layer's output as done by prefix tuning), and training Q on the domain-specific data while keeping ϕ unchanged.For a downstream task with L classes, we can train one soft prompt Q l ∈ R n×d for each class label Y l . The training objective for tuning Q l in this setting is the standard language modeling objective. The generator parameters ϕ are frozen and only the soft prompt Q l is optimized on the real training set D real :L gen (Q l ) = - 1 |D real | ∑ X∈D real ,y=Y l ∑ x j ∈X log Pr ϕ (x j |x <j ; Q l ).(3)Weighted Generator Tuning. The above language modeling objective treats all tokens equally. To encourage the data generator network to generate label-discriminative texts, FewGen (Meng et al., 2023) propose to train a weight net, Φ W : R d → R, which learns to assign higher weights to those generated tokens that are more likely to discriminate the ground-truth label Y l from other labels Y l ′ ∈ Y by minimizing the weighted generation loss,EQUATIONHere, Q and W are optimized under the bi-level optimization framework. That is, first optimizing the loss L wGen produces a function Q of W: Q(W), which is then applied a second time to optimize the weight net, W, by minimizing the following loss:EQUATION)By optimizing Q and W iteratively, the generator g ϕ,Q learns to generate tokens that are more related to the given label than other labels.Algorithm 1: Generator Tuning. Distribution-Aligned Regularization. However, given the few-shot training set, it is easy for the weight net to overfit shortcut tokens, which are not robust tokens for the given task. For example, the token "not" can discriminate the positive sentence "It is a good movie" from the negative sentence "It is not a good movie", but is obviously not a generalizable label-discriminative token. Hence, enforcing the generator to solely rely on the weights may lead to generating sentences that are irrelevant to the given label. We propose to regularize the generator tuning objective by adding a sentence-level distribution constraint to encourage the generated sentence to align with the in-distribution data:EQUATIONwhereEQUATIONHere, W ∈ R 1×K indicates the weights for a sequence of K tokens. Z i,l = g ϕ,Q (X i ), Z i,l ∈ R K×d , denotes the last-layer hidden states output from the generator g ϕ,Q (•) for i-th instance X i of class Y l , and Z j,l ′ represents that from a different class Y l ′ . D(•, •) measures the cosine similarity between two vectors. By minimizing L dist (Q), we encourage the generated texts to stay close to the ones in the same class while being pulled away from those that belong to other classes. We present the whole procedures in Algorithm 1.Despite the domain adaptation procedure employed, the resulting synthetic training set can inevitably contain low-quality data. Training soft prompts with a naive combination of synthetic and few-shot real data can result in the optimization process being dominated by gradients from the synthetic data. Hence, we up-sample the few-shot data by pairing each batch of synthetic data with a corresponding batch from the few-shot real data. Then, we employ gradient surgery to these paired batches to resolve conflicting gradients from different data sources in prompt tuning.Gradient Surgery. It was first proposed to de-conflict gradients in a multi-task learning setting, where a model θ is trained on a set of M tasks (Yu et al., 2020) . Let δ i = ∂L i (θ) ∂θ denote the gradients of i-th task loss L i (θ) with respect to the model θ. ∀δ i ∈ {δ i } M i=1 , δ i is iteratively altered across all the other tasks by subtracting the component δ i •δ j ∥δ j ∥ 2 δ j , which is its projection to the plane of j-th task' gradient δ j , where j ̸ = i. This step is applied when δ i • δ j < 0, which indicates the two tasks have interference in driving the optimization path.In this paper, we treat the gradients from real data, δ real , as the positive gradients for the task and always project the gradients of the synthetic data, δ syn , to the direction of δ real :EQUATION)where δ real ∥δ real ∥ is the normal plane of the gradients δ real from real data, andδ syn •δ real ∥δ real ∥is the magnitude of the projection of δ syn onto this normal plane. If δ syn • δ real < 0, then the projected gradients will be dropped and the gradients of synthetic data will be modified as:EQUATIONBy removing the conflicting gradients of the synthetic data, we train soft prompts using the loss function in Equation 1 and update the weights with a gradient descent approach:EQUATION)where η is the learning rate and ϵ is a factor for controlling the strength of synthetic knowledge guidance, which was studied in a similar work in computer vision (Zhu et al., 2023) . The whole algorithm is presented in Algorithm 2. 
datasets, metrics, and settings: 
 We conduct evaluations on seven sentence-pair classification datasets in two tasks. In the paraphrase detection task, we use MRPC (Dolan & Brockett, 2005) and QQP2 . In the natural language inference task, we use MNLI (Williams et al., 2018) , SNLI (Bowman et al., 2015) , QNLI (Rajpurkar et al., 2016) , RTE (Dagan et al., 2005) , and SICK (Marelli et al., 2014) . We follow LM-BFF (Gao et al., 2021) to prepare the few-shot learning setting: both D train and D dev contain 16 samples per class, which are sampled from the original training set using 5 random seeds, and the original development set is used as the test set. We adopt Accuracy for all the classification tasks and report the average test accuracy over 5 seeds. We compare methods using the average performance across the seven datasets. More details about the datasets, models, and training settings can be found in the Appendix.We consider the following zero-shot and few-shot baseline methods. We also compared with two transfer learning methods, SPOT (Vu et al., 2022) and OPTIMA (Guo et al., 2022) , where a large-scale real-world source-domain dataset is available.(Zero-shot) Prompting. We prompt the frozen T5-large and Flan-T5-large with only taskspecific natural language prompts (i.e., hard prompts) and treat it as the zero-shot learning baseline. For fair comparisons, we apply the same hard prompts for all baselines where applicable. Prompt-based templates are presented in the Appendix.Following GPT-3 (Brown et al., 2020) , we incorporate the acquired few-shot examples as demonstrations in the hard prompt templates, which is also called few-shot prompting (in contrast to zero-shot prompting). The role of few-shot examples helps the frozen T5-large and Flan-T5-large models better understand the task by exemplifying the task instruction with real-world examples. The order of few-shot samples is randomly determined and we report the average performance across five runs.We feed the few-shot data without any hard prompts into T5-large and Flan-T5-large and fine-tune the entire networks. Different from traditional fine-tuning that trains an additional classification layer from scratch, here, we apply the label verbalizer and tune the language modeling head instead. Using hard prompts for model tuning is represented by LM-BFF (Gao et al., 2021) and PET (Schick & Sch ütze, 2021a) . Here, we apply the same hard prompts as other baselines to wrap every training sample for tuning T5-large and Flan-T5-large. We also consider incorporating soft prompts for model tuning as P-Tuning (Liu et al., 2022) and DART (Zhang et al., 2022) , denoted as PFT + soft prompt.Proposed by Gu et al. (2022) to pre-train soft prompts on text corpus from OpenWebText (Gokaslan & Cohen, 2019b) using the next sentence prediction objective (Devlin et al., 2019) . We download the pre-trained checkpoint for T5-xxl and fine-tune them on the sentence-pair classification tasks.FewGen. Proposed by Meng et al. (2023) , where authors first use the few-shot data to adapt CTRL (Keskar et al., 2019) to every task with prefix tuning and then generate synthetic datasets using the adapted generator. We use their code3 to produce the synthetic datasets (denoted as FewGen) for all tasks and train soft prompts for T5-large or Flan-T5-large. It can be treated as the ablation for DawGen where only L wGen is applied.Our main results are presented in Table 1 . Overall, compared with the naive prompt tuning method, our approach yields an average improvement of approximately 18% across all datasets when applied to T5-large, and about 15% for Flan-T5-large. In particular, PT (using 102K parameters) under our framework outperforms FT (using 770M parameters) by an average improvement of 3.8% across all datasets based on T5-large. When compared to SPOT and OPTIMA, which use large real-world datasets for transfer learning, our approach exhibits competitive performance on QQP, MRPC, and SICK, though its performance on MNLI and SNLI remains a big challenge, highlighting the possibility of using synthetic training data as an alternative to enhance few-shot prompt tuning.When comparing Prompt Tuning (PT) with both Prompting and In-Context Learning baselines, we observed that PT's performance enhancement is marginal, despite leveraging 102K parameters to learn from data. This suggests that PT can be prone to overfitting when applied to few-shot datasets. On the contrary, FT employs 770M parameters and, despite its greater tendency to overfit the data, surpasses PT by an average of 14-18%, indicating that a better pre-trained initialization can significantly enhance the learning outcomes.Previous research (Vu et al., 2022; Gu et al., 2022; Guo et al., 2022) indicates that data-driven initialization significantly improves PT. Here, we study how the order in which synthetic data and real data appear affects PT. We experimented with several strategies: Real+Syn 2 .We observed that a naive combination of Syn + Real data performs the worst, where the few-shot real data could be overwhelmed by the larger synthetic data. Simply applying a label smoothing regularization, denoted as Syn + Real + LS, generally does not help. In contrast, (Real, Syn), which up-samples the few-shot real data when paired with either FewGen or DawGen data, confers an obvious improvement for both T5-large and Flan-T5large. We also observed an interesting phenomenon, where T5-large prefers Real → Syn while Flan-T5-large prefers Syn → Real. This may suggest that training a model initially on the few-shot real data is not always an advantage and the order in which real and synthetic data are presented impacts differently for different LLM backbones. Table 3 : Test accuracy of ablation studies. "GS" stands for "Gradient Surgery" in prompt tuning.We evaluate every component in our approach and present the results in Table 3 . Specifically, we observe that: 1) the distribution-aligned regularization term L dist is effective -Comparing DawGen against FewGen, the soft prompts trained exclusively on DawGen results in an average improvement of 4% for T5-large and 1.6% for Flan-T5-large across all datasets compared to the one trained on FewGen; 2) utilizing a few real samples to augment synthetic datasets is beneficial, and this benefit is more pronounced on FewGen than DawGen, indicating that the higher the quality of a synthetic dataset, the less it requires supplementation with real data supervision. 3) the gradient surgery technique effectively mitigates the conflict between synthetic and real data sources -applying gradient surgery further improves the performance by an average of approximately 2% across the datasets.Recent works (Varia et al., 2023; Aly et al., 2023) suggested that the advantage of instruction tuning (Ouyang et al., 2022; Chung et al., 2022) for language models can extend to few-shot learning settings. This is further corroborated in our experiments. Across the tables, Flan-T5-large excels T5-large on all experiments by a large margin. In this paper, we provide a few new findings to this avenue: 1) instruction-tuned models can follow soft prompts better, as shown by PFT + soft prompt versus PFT, where an average improvement of 2% is spotted on Flan-T5-large while no increase is observed on T5-large. 2) Prompt Tuning on top of instruction-tuned models are less prone to overfitting the few-shot data. It is shown that Flan-T5-large elevates the PT performance of T5-large by around 20% on average. 3) instruction-tuned models prefer using synthetic data to warm up the learning, as shown in Table 2 . Moreover, when using Flan-T5-large as the backbone, Syn → Real strategy leads to an average improvement of about 3 ∼ 4%, as shown by comparing either FewGen or DawGen from Table 3 with Syn → Real from Table 2 . While synthetic data is often used as a form of regularization for learning on real datasets, in our study, presenting a few real samples after training on synthetic data produces a regularization effect. We suspect that the instruction-following capability of Flan-T5-large may play an important role. 
conclusion: 
 This paper presents a framework for generating synthetic training data with LLMs to boost prompt tuning in few-shot settings. We introduce Distribution-Aligned Weighted GENerator tuning (DawGen) to encourage the generation of label-relevant text samples to improve the quality of the synthetic dataset. To better exploit the synthetic data while making the best use of the few-shot real data set, we employ the gradient surgery technique during prompt tuning to eliminate the conflicting gradients from the synthetic data batches. Experiments on seven sentence-pair classification datasets and two LLM backbones demonstrated the effectiveness of our method for boosting prompt tuning in few-shot settings with synthetic data. We show that Prompt Tuning augmented with DawGen can surpass Full-Model Fine-Tuning in few-shot settings by a large margin, highlighting the feasibility of using LLM-generated data as an alternative. 
limitations and discussions: 
 Currently, there are a few limitations of this study that may limit the impact scope of the insights conveyed by this paper.• The gap between the evaluation metric and the quality of the synthetic data. Specifically, the downstream few-shot learning performance of prompt tuning, which is often measured by accuracy, may only reflect the preferences of the model rather than aligning with human judgment in terms of the quality of the synthetic data.There should be intuitive methods to explain to humans why one synthetic sample is superior to another.• The few-shot learning performance on these public benchmarks reported in this study does not stand for the state-of-the-art. They only reflect the performance of prompt tuning. More advanced parameter-efficient learning methods like LoRA could bring higher performance than prompt tuning. Nevertheless, the relevant improvements in this study can still support the effectiveness of a specific strategy.• Synthetic data generation cost can be a concern. Current data generators rely on large language models, which have a deep stack of transformer layers. Feedforward computations and autoregressive generations involve non-trivial GPU, memory, and time costs. A few research efforts, such as FastGen Ge et al. (2024) , have been devoted to accelerating LLM inference costs. 
future directions: 
 In Table 4 , we compare the modern augmentation paradigm, which leverages Generative AI, specifically LLMs, with the traditional paradigm that relies on transfer learning. Together with the discussions in the Limitation section, we propose the following directions: 1) employ explainable approaches to highlight the influential elements in the synthetic data and then devise quantitative measures to assess the data quality; and 2) develop inference acceleration algorithms tailored specifically for batch generation. Prompt Tuning. Following previous works (Lester et al., 2021; Gu et al., 2022; Guo et al., 2022) , we conduct prompt tuning on T5 (Raffel et al., 2020) and use the lm-adapted version of T5-large (770M parameters) for the main experiments. We also experimented with Flan-T5-large (Chung et al., 2022) , which further trained T5-large on instruction tuning datasets to improve its instruction following capability (Ouyang et al., 2022) . We fix the prompt length n = 100 and learning rate η = 0.3 as suggested by Lester et al. (2021) . We use the cosine learning rate scheduler for all methods. We set the maximum number of training steps to 1, 000 and evaluate models on the development set every 4 steps. We set batch size to 4 for MRPC and QQP, and 6 for the other NLI datasets. For all the prompt tuning experiments, the T5-large and Flan-T5-large backbone models are fixed and only soft prompt embeddings are updated. All the training experiments are done on NVIDIA A6000 with 49 GB. 
 
 Appendix: We conducted a significance test for Table 1 on results where our method was superior. Table 8 shows the p-values. Our method shows statistically significant improvement over the second-best baseline, full-model fine-tuning (FT). Although not superior in every task, our method shows higher average improvements while using only 102K parameters, compared to 770M in full-model fine-tuning.Comparing the computational costs of different training paradigms directly is challenging due to variations in training data, pretraining strategies, and stopping conditions. There could be a trade-off between performance and computational cost.• PPT: It involves a one-time pre-training of prompts on 10GB of OpenWebText data using next sentence prediction. While these prompts are reusable across tasks, they may underperform in tasks that are underrepresented in the OpenWebText corpora.• SPoT: Prompts are pre-trained on one or more source tasks, and then adapted for a target task. However, selecting appropriate source tasks requires an extensive search and may struggle with large domain gaps.• DawGen: It involves a one-time adaptation of the generator to the target task using few-shot examples. The adapted generator can then produce unlimited synthetic data, enhancing the flexibility of prompt training. However, the generation cost depends on the LLM backbone and the advanced inference acceleration methods such as FastGen.https://github.com/guoxuxu/soft-prompt-transfer/tree/main/DawGenhttps://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairshttps://github.com/yumeng5/FewGenhttps://github.com/yumeng5/FewGen 


Decision: The paper studies few-shot prompt tuning enhanced with data augmentation. The general idea is to use LLMs to generate synthetic data for training the soft prompts. The proposed method, Distribution-Aligned Weighted Generator Tuning (DawGen), aims to produce in-distribution synthetic data that are aligned with few-shot samples. The training procedure consists of a gradient surgery method to mitigate gradient conflicts when training the model with both real and synthetic data. The reviewers generally believe that the paper is well-motivated and clearly-written, and find the proposed method to be novel and effective. 

While the AC believes that the paper has enough merits to warrant acceptance, some of the authors' responses appear unprofessional and disrespectful toward the reviewers. One of the reviewers expressed concerns about the tone and content of the authors' response. It is natural that authors/reviewers have different opinions, but please note that the reviewers have dedicated their time and efforts to reviewing the manuscript, and it is crucial that all interactions remain respectful and professional.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper addresses the topic of prompt tuning, which typically requires large quantities of labelled training data and underperforms full-model fine-tuning in few-shot settings (by which the authors refer to low-resource settings rather than few-shot prompting). The authors present a method to create synthetic data using an LLM to do prompt tuning on and a method to fine-tune on the data, consisting of using gradient surgery when fine-tuning on a mix of the synthetic and real data. 

Their method consists in (i) training a soft prompt using a few-shot dataset to adapt an LLM to a given domain, (ii) using their method Distribution-Aligned Weighted GENerator tuning (DawGen) to align generated examples to those of the same class, and (iii) using label-conditional prompts and (iv) using gradient surgery when fine-tuning on a combination of synthetic and real data to account for possibly low quality data.

They test their method using T5-large and FlanT5-large for 2 tasks (paraphrase detection and NLI) over 7 datasets, comparing to multiple other methods (fine-tuning, zero-shot, in-context learning, etc.). Their method gives results better than fine-tuning for some tasks and worse for others, and 18% improvement on average compared to basic prompt tuning for T5-large.

In its current state, I found the paper unclear with respect to its main aim (more with respect to the writing of the paper - providing motivation for working with so few training examples for example) and would also have appreciated some more details concerning the method (see below). The paper is also lacking examples allowing me to review the quality of the produced synthetic data.
strength_weakness: strength_weakness: - Lack of general clarity in the paper about the main aim of the article. This is in my opinion linked to several things, starting from an unclear motivation concerning the use of “few-shot”, which in the context of prompting often refers to in-context learning, but here appears to refer to the case where we have few training examples (this could be made much clearer and non-ambiguous right from the start). On first reading, I was not sure why the authors were only using 16 examples per class of real data, which is so few. Also, why only choose 16 examples? If the main aim of the paper is to study whether prompt tuning can rival fine-tuning in low-resource settings, I think it would have been more informative to have the comparison at different numbers of training data examples in order to see the progression of the 2 approaches. With so few examples, fine-tuning must be hugely overfitting too. Is this a realistic setting?
- In the main method, it was unclear to me what the input to the LLM looks like in order to produce synthetic training examples. Does it just consist of the label-conditional soft prompt and one of the 16 examples (for each class)? Also, I may have missed it, but how many examples were generated?
- A major weakness in my opinion is the fact that no examples of generation data or predictions are given in the example. As a reviewer, this makes it impossible for me to judge the performance of the generation method and also whether the base LLM used is up to the job at all. The very low accuracy scores of the baseline on the binary tasks (below random) appear to show that the base model used may be inadequate.
questions: questions: 1. The meaning of “few-shot learning” was initially unclear to me (between using few-shot examples and using a very small set of  training examples), particularly as the article is on the topic of LLMs, which typically use few-shot examples. I assume that the article wants to deal with very few training examples, but I only got this after rereading several times, and putting the article in the context of (Gao et al., 2022).
2. Synthetic data generation method: 
 
    a) It is Unclear to me what the input to the LLM is other than the label-conditioned soft prompt and the current example. 

     b) What is the quality of the generated data like? There are currently no examples of generated examples in the article, making it impossible for me as a reviewer to judge the performance of the generation method. This is particularly relevant as the authors cite the inevitability of low quality data.
3. Missing details:

    a) Why was so little data used? 16 examples for each class? Is this a realistic setting? What about other amounts of data?

    b) Amount of synthetic data generated. How diverse is the data? Are there duplicates?

    c) In-context learning -> how many examples were used?

    d) What does the “number params” correspond to in Table 1? The models are the same, so does this refer to the number of tuned parameters?

    e) PPT appears to use T5-xxl and not T5. If so these results are not comparable.

4. In the related work, the authors mention that fine-tuning on a model on few examples (e.g. 16 samples per class) can lead to overfitting. What is the significance of 16 samples? i.e. why choose this number, which appears to me to be far too low to be realistic.
5. Verbaliser -> it would be good to make it clear what you mean by this in the heart of the text (it only became clear to me from the Appendix)
6. Very low scores of the baselines “Prompting” and “In-context”, especially given the binary nature of the tasks. Why do they do so poorly (Under random for some tasks)? Could this mean that the base model being used is not good enough and perhaps a better model should be used?

## Language, grammar, etc.

- Few-shot real data -> real few-shot data
- Comparable performance as -> comparable performance to
- E.g., -> it is good practice to use a non-breaking space after E.g. to avoid linebreaks
- Pertaining data -> Pretraining data
- Which is inevitable to contain low-quality data -> which inevitably results in low quality data
- Lester et al. (2021) converts -> Lester et al. (2021) convert
- Taking sentence-pair classification as an example. Given -> Taking sentence-pair classification as an example, given
paper_summary: paper_summary: The paper introduces a novel approach to enhance prompt tuning (PT) in few-shot learning settings by synthesizing task-specific labeled datasets using large language models (LLMs). The authors propose a method called Distribution-Aligned Weighted Generator Tuning (DawGen) to create in-distribution synthetic data, and they apply gradient surgery to manage gradient conflicts when training with both real and synthetic data. Their experiments across various datasets demonstrate that this method can outperform traditional fine-tuning techniques, presenting an efficient alternative when large labeled datasets are not available.
strength_weakness: strength_weakness: 1. While the method shows interesting results, there is not a unique method that has the best results. The best results are scattered between different methods.

2.Some parts of the methodology, particularly around the implementation of gradient surgery and the generation of synthetic data, could be more detailed to ensure reproducibility.

3. Are the superior results statistically significant.
questions: questions: 1.How robust is the proposed method to variations in the quality of synthetic data? Can the model's performance degrade significantly with lower-quality synthetic datasets?

2. How does the performance of DawGen scale with the size of the LLMs used for generating synthetic data? Would larger or smaller models significantly affect the outcomes?
paper_summary: paper_summary: This work tackles the problem that prompt tuning performs poorly in low-resource settings. It proposes a DawGen framework to generate synthetic data to augment prompt tuning and apply a gradient surgery approach to avoid conflicts from different data sources. Experiments shows that DawGen outperforms various baselines in low-resource prompt tuning.
strength_weakness: strength_weakness: 1. More parameter efficient methods like LoRA are more commonly used in current LLMs and usually achieves better performance than prompt tuning. It would be better to examine the method's effectiveness on LoRA [1].
2. Compared to the baselines (PPT [2], SPoT [3]), DawGen seems to introduce additional computation cost because it requires data generation for each downstream task while PPT and SPoT pre-train the prompt and applies it to various end tasks. This computational cost should be addressed.

[1] LoRA: Low-Rank Adaptation of Large Language Models.
[2] PPT: Pre-trained Prompt Tuning for Few-shot Learning.
[3] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer.
questions: questions: Gradient surgery seems to be orthogonal to the prompt pre-training stages. How does it perform when combined with the baselines like PPT and SPoT?

Comments: title: Regarding few-shot learning and increasing the amount of shots
Last Day of Discussion
Response to the Authors' rebuttal
Author-Reviewer Discussions
Response to the Authors' rebuttal
comment: I want to clarify that **“with a bit of annotation …”** is impossible for many real-world tasks especially for rare diseases. The assumption of the availabity of sufficient labeled data is unrealistic. Many machine learning fields have been dedicated to solve this problem including transfer learning, zero-shot and few-shot learning (FSL). FSL focuses on enabling models to learn new tasks from a very limited amount of data – typically only a few examples, hence the term "few-shot". I won’t repeat the facts and significance about FSL but refer you to the survey [1]. 

* Scale is the power of today’s LLMs! Preceding research established many neural scaling laws for deep learning models [2,3]. It is a known fact that as the amount of training data decreases, the model’s performance degrades significantly. If annotated data is not a problem, maybe one panacea can cure all diseases. Unfortunately, data is a big problem! **Few-shot setting is at the tail of the scaling law** and **abundant research have been done to elevate the performance of this extreme case at the tail**. These methods predominantly leverage external knowledge, such as pretraining and transfer learning, to guide the model in learning effectively from a few examples. 

Based on these findings and limitations, we **focus on pushing the boundary of prompt tuning in few-shot settings with the help of generative AI.** 

* Studying scaling laws involves non-trivial computational costs. This is not the focus of this paper (as we presented many other interesting findings if you read the paper), therefore we referred you to previous studies conducted **exactly for comparing prompt tuning and full-model tuning given different amounts of real data**. Scaling up in our problem setting is **extremely resource-intensive** and deviates from our main objective. *It involves generator tuning, data generation, and prompt tuning for every random seed and every hyperparameter setting*. For example, suppose we fix 5 random seeds and 3 hyperprameter settings, and want to plot 10 points for drawing the scaling law for a single NLI task, it would approximately take about (2h + 3h + 2h) * 5 * 3 * 10 = 1050 hrs = 43 days on the A6000 workstation for both training and evaluation. This study obviously does not worth the costs compared to other more important studies. Please allow this paper to present limited (but nontrivial) findings as we can only push forward the science step by step. 

The opportunity for us to conduct rebuttal and discussion is to solve your questions to avoid biased scores, encourage insightful discussions and for us to receive constructive suggestions to improve the paper. It is not meant for debating on some already established facts and problems, for which we should remain honest and scientific. Since I have addressed most of your questions, I would expect you to justify your score with sound reasons and facts, presented professionally. I really hope you can spend a little more time to reread and understand this paper's contributions, and also want to refer you to other reviewers’ justifications. We’re actively working on improving the paper and hope to receive your constructive suggestions and more importantly, a fairer score!


## A take-away summary of the key contributions

Although it has been clearly stated at the beginning of this paper, we want to clarify in short what our contributions are for your reference.

Prompt tuning, conditioned on LLMs, has demonstrated competitive performance as full-model tuning under full-data settings. However, recent studies uncovered that prompt tuning significantly underpeforms full-model tuning in few-shot settings, limiting its application scope (Problem). Conventional methods generally rely on transfer learning from real-world datasets, which assumes the availability of large-scale, real-world, task-relevant datasets (Limitations of existing solutions). This paper builds on recent research for “LLMs as training data generators” to provide an alternative solution for advancing prompt tuning in few-shot settings (Motivation). We identified the problem of potential ill distributions in synthetic datasets and proposed DawGen to promote distribution alignment for synthetic data generation (Contribution 1). We want to make the best use of few-shot and synthetic data to further enhance downstream learning performance. Hence, we employ gradient surgery to modify the gradients from synthetic data by always projecting them onto the direction of real-data gradients and subtracting conflicting components (Contribution 2). 


References

[1] A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities

[2] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022

[3] Revisiting Neural Scaling Laws in Language and Vision.  NeurIPS 2022.

[4] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. NeurIPS 2022.
Thank you for your detailed response. I do believe that the aim/motivation of the paper could be made clearer with some reformulations in the text, along the lines of what you write in your response. I think that the examples of generated synthetic data - I think this would be a useful contribution to the supplementary material, although it would have been good to have a more detailed analysis of the generated data.

I maintain that it would be more interesting to test different numbers of training examples other than 16 per class, which is very few (even if this is what some other works have done, I am not convinced that it that realistic). In reality, with a bit of annotation it is often possible to have more examples than 16 per class. I would therefore find it realistic to test the method with varying amounts of data to ascertain what the threshold is for prompt tuning still lagging behind FT.

I therefore choose to keep the same score.
Dear Reviewer 1fLs,

Thanks for your review! The discussion is ending tomorrow, and it'd be greatly appreciated if you could acknowledge the author's rebuttal and update your review if necessary.

Thank you!  
AC
Thanks for your response. After reading your response, I think my rating is ok and I'm not change it.
Dear reviewers,

Thank you for your review of this submission. The authors have provided their rebuttal, and it is time to engage in discussions with them if you have additional questions. Please take the time to read the other reviews and update your own review as necessary. 

Reviewers 1fLs/T1e9: Even if you do not have further questions and/or choose to maintain your original score, please acknowledge that you have read the authors' rebuttal.

Thank you!  
AC
Thanks for your response. After reading your response, I think my rating is ok and I'm not change it.