Self-Guide: Better Task-Specific Instruction Following via Self-Synthetic Finetuning
Authors: Chenyang Zhao
Xueying Jia
Vijay Viswanathan
Graham Neubig
Tongshuang Wu
Venue: COLM (2024)
Keywords: instruction following, synthetic data, self-training
TL;DR: We improve instruction-following ability in language models via self-distillation (finetuning on data generated by themselves).

Abstract: Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. However, prompting often leads models to make predictions with lower accuracy compared to finetuning a model with ample training data. On the other hand, while finetuning LLMs on task-specific data generally improves their performance, abundant annotated datasets are not available for all tasks. Previous work has explored generating task-specific data from state-of-the-art LLMs and using this data to finetune smaller models, but this approach requires access to a language model other than the one being trained, which introduces cost, scalability challenges, and legal hurdles associated with continuously relying on more powerful LLMs. In response to these, we propose Self-Guide, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM, then use these input-output pairs to finetune the student LLM itself. In our empirical evaluation of the Natural Instructions V2 benchmark, we find that Self-Guide improves the performance of LLM by a substantial margin. Specifically, we report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics. This sheds light on the promise of self-synthesized data guiding LLMs towards becoming task-specific experts without any external learning signals.

Published as a conference paper at COLM 2024
SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning
Chenyang Zhao'", Xueying Jia"" Vijay Viswanathan', Graham Neubig,, Tongshuang Wu?
!Department of Computer Science and Technology, Tsinghua University, Beijing, China LLanguage Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA zhaochenyang22@gmail com {xjia2, vijayv, gneubig, vijayv.gneubig. sherryw sheooby
@andrew. cmu, edu
Abstract
Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. However, prompting often leads models make predictions with lower accuracy compared to finetuning a model with ample training data. On the other hand, while finetuning LLMs on task-specific data generally improves their performance, abundant annotated datasets are not available for all tasks. Previous work has explored generating task-specific data from state- of-the-art LLMs and using this data to finetune smaller models, but this approach requires access to a language model other than the one being trained, which introduces cost, scalability challenges, and legal hurdles associated with continuously relying on more powerful LLMs. In response to these, we propose SELF-GUIDE, multi-stage mechanism in which synthesize task-specific input-output pairs from the student LLM, then these input-output pairs to finetune the student LLM itself. In empirical evaluation of the Natural Instructions V2 benchmark, we find that SELF-GUIDE improves the performance of LLM by substantial margin. Specifically, we report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics. This sheds light on the promise of self-synthesized data guiding LLMs towards becoming task-specific experts without any external learning signals
1 Introduction
One of the traits of large language models (LLMs) that has captured the imagination of model developers their potential automate broad range of highly complex tasks with relative ease (Brown et al., 2020a). Where previous generations of models required vast task-specific training sets, LLMS now offer the promise of enabling similar accuracy by simply writing prompt and supplying few examples. However, this may be false promise. Prompting typically leads models make predictions with lower accuracy compared to finetuning model with ample training data [Gao et al. 2021: Zhang et al. 2023). Moreover, LLMs' performance crucially depends on their ability to follow instructions outlined the prompts and even minor alterations to these prompts can result in notable performance decline (Sclar et al. 2023a).
On the other hand, in data-abundant tasks, finetuning pre-trained language model through supervised finetuning (Ouyang et al. 2022 Wei et al. 2022) and reinforcement learning from human feedback (Lambert al. 2022) has proven successful strategy. However, the effectiveness of this approach diminishes notably for underrepresented tasks suffering from
'qqual contribution
All code and data necessary to reproduce experiments are released publicly: http:///ddttbooooo///oomm//////o

---

Published as a conference paper at COLM 2024
Instruction: "Given sentences (a premise and hypothesis), classify their relationship."
Examples:
*Premiss:: five guests that answered might go. Hypothesic: are exactly five guests that answered.
*Premise: The four peppers that fall might have gone bad. Hypothasis: Thare are exactly four apples that fall.
Label: Neutral
Baseline (Prompting)
Label: I'm sorry. but the given input doas not contain premise and hypothesis.
Input:
"Promise: The five guests that answered don't die.
Hypathesis: There exactly six guests answered."
Ours (Self-Guide)
Label: Noutral
Generate Data
Finetune Model
Figure 1: SELF-GUIDE uses model's ability to generate synthetic data as a vehicle to improve the model's ability to execute task as specified by an instruction.
data scarcity (Li et al. 2022), highlighting the critical need for high-quality training data. To this end, recent studies have explored the potential of utilizing potent "teacher" LLMs to create task-specific training data, thereby enhancing the performance of comparatively less advanced "student" LLMs (Khattab et al. 2023 Viswanathan et al. 2023). While this strategy is effective, its feasibility is hampered by the cost, scalability, and legal hurdles associated with the continuous reliance on more powerful teacher LLMs. And when there isn't a stronger model available to provide learning signals, this approach is fundamentally infeasible.
In this paper, we instead propose to finetune language models on self-synthesized data to improve their ability to follow instructions on specific tasks with minimal annotated data. Concretely, we ask the question: can enhance model's performance for arbitrary tasks without external training signals, such as labeled data or another teacher model? In our affirmative answer to this question, we introduce SELF-GUIDE, a novel methodology that enables LLMs to better execute task-specific instructions without requiring additional data or training signals. SELF-GUIDE operates in the few-shot setting, where we are given only task instruction and up to three examples of task demonstrations. SELF-E works by first employing the target model to generate a synthetic dataset for given task. The model is then finetuned on this "self-generated" data.
This approach differs from previous methods for bootstrapping a model from its own outputs. Self-Instruct (Wang et al., 2023 is the most similar work; here, they use base dataset to finetune the same
LLM (GPT-3) to generate a synthetic instruction-followii LLM. In contrast to their method, which aims to improve general-purpose LLM capabilities, our method aims to optimize an LLM for specific task instruction. Methodologically Self-Instruct generates large set of instructions and demonstrations to use for instruction- finetuning. While Self-Instruct asks an LLM to self-generate synthetic demonstrations for each synthetic instruction, their method only generates a single demonstration for each instruction, while our method can effectively self-generate hundreds of examples for given instruction. Our methods are complementary. We find that our method can perform target tasks with significantly greater reliability when applied on top of gerersl- instruction finetuned models (e.g. the kind of model resulting from Self-Instruct). We provide more details regarding other related methods in Section
In our empirical evaluation of SELF-GUIDE using on multiple tasks from Super- Naturallnstruuciins V2 (Wang et al.) 2022), applying SELF-GUIDE to already instruction- tuned model (Vicuna-1.5-7B, (Zeng et al. 2023 yields an absolute performance improve- ment of 17.9 points of ROUGE-L for open-ended generation tasks and 14.5 points of accuracy on classification tasks, compared against the baseline of prompting the same model with the same prompt.
2

---

Published as a conference paper at COLM 2024
"Write topic word from the given fact."
Examples:
"pesticides pollution" - "pallution harms"
"running takes energy" + "marathon running"
[#1, #4]
Input Generation
Input Filter
Output Generation
Input-Outpus Pair Filter
#1. "plants require pollination"
"bee pollination"
#2 "light"
X
#3 "erosion causes deposition"
"1"
#4. "tadpole changes into frog"
"frog"
Figure 2: At the heart of SELF-GUIDE lies an efficient and effective multi-stage generation mechanism, where the LM generates input-output pairs step by step. After the generation and filtering, the self-generated data are further used to finetune the LM itself. This figure describes the process for the generation tasks.
"whether the two sentences clearly entall, contradict, neutral with each other"
Examples:
"vertence, dancer dancing stage."
"sontence_ B. gir is giving dance performance on the das.." "ontal"
[H3]
Pseudo Label Sampling
Input Generation
Inputs Filter
Output Generation
Input- Oupptttu Pairs Filter
#1. "ontall
"sentence A: The crowd is cheering at her dance performance.
"sentence A: Two girls are not playnng inside
"contradcctonn
house sentence Br Two girls are playing inside a jumper house"
"sentence A boy playing guitar. sentence B. A lttle girl is playing
"noutral"
gutter.
Figure 3: This figure describes the process for the classification tasks; we use a slightly modified procedure for self-generating data for classification tasks. Put simply, we first generate pseudo-labels, then generate corresponding diverse inputs, and finally generate true labels. Regarding the Input-Output Pairs Filter, a set of labels will be provided filter out labels. Further details will be described in Section 2.1
SELF-GUIDE
In the following section, we outline the proposed SELF-GUIDE framework (illustrated in Figure and and Figure 3. As input, SELF-GUIDE takes an instruction (e.g. "Write topic word from a given fact."), and a few example inputs and outputs (e.g. "pesticides cause pollution" " as input fact, "pollution harms" output topic word). Our method proceeds in multiple stages, where each stage performs procedures to improve quality, either through rule-based filters on data quality, hyperparameter search. Below, describe the key design points.
3

---

Published as a conference paper at COLM 2024
2.1
Data Generation
The input generation process starts by extracting inputs from provided example pairs and combining them with the instruction to populate a prompt template. The constructed prompts are then forwarded to the LLM to obtain modee-genertted inputs. After each round of prompting, the newly generated inputs are added to an input repository. A subset of inputs is randomly sampled from this repository and merged with inputs from the initial examples to formulate a new prompt, aiming to incrementally expand the set of LLM-generated inputs. We only go through one round of input generation and subsequently, in the quality refinement stage, rule-based filters are applied to remove low-quality inputs. Detailed descriptions of this quality refinement process are provided in Section 2.2
Input Generation
The prompt templates employed vary based on the task type-generation or classification. For generation tasks, a simple prompt template instructs the model to generate new input following the provided examples. In contrast, in classification tasks, the model is directed to generate an input corresponding to a randomly chosen label from the available set, allowing for more balanced label distribution in the generated examples. Further details on the prompt templates are provided in Appendix A.111 A.1.1
Output Generation The output generation employs conventional in-context learning techniques (Min et al. 2022a Brown et al. 2020b). It provides the model with instructions and original examples, allowing it to annotate every single input that was generated earlier in the input generation stage. After obtaining all annotated outputs for the previously generated inputs, another round of rule-based filtering takes place to select the final synthetic dataset, the details of which are described in Section 2.2 The comprehensive prompt templates used for output generation are provided in Appendix A.1.2
2.2 Quality Optimization: Temperature and Rule-based Filters
The quality of generated data is essential for the success of the downstream training. Our approach takes two-fold strategy of adjusting generation parameters to improve quality and filtering out low quality samples.
Temperature:
Intuitively, adjusting the temperature is a common strategy to balance di- versity and quality. Our framework also leverages this approach, using a relatively higher temperature during input generation to encourage diversity, and a lower temperature in other stages to promote quality. However, solely relying on temperature adjustment is insufficient to achieve the desired balance. SELF-GUIDE further employs two rounds of rule-based data filtering, one after the inputs are generated, and the other after the outputs are annotated. For clarity, we describe these two rounds together.
We manually curate a list of noise terms, such as common salutations, greetings, and noise characters (e-g.. \ in the generated contents). If any noise term from this curated list appears in either the input or the output of a generated example, we discard the entire example. This filtering step ensures that the generated examples remain concise, focused, and free from irrelevant content or artifact patterns.
Noise Filter:
Length Filter: While the lengths of demonstrative examples may exhibit bias, we assume these examples already possess decent representativeness for a specific task in terms of length distribution. Based on this assumption, we further assume that the lengths of demon- strative examples follow a normal distribution for specific task, and since most data points of a normal distribution fall within two standard deviations from the mean, we stipulate that the lengths of generated examples' inputs and outputs should also approximate nor- mal distribution with mean и and variance 2² calculated from the demonstrative examples. Specifically, the lengths are required to be within the range (μ - 20,, μ + 20r). Other attributes like semantic similarity of generated examples are computationally expensive and lack clear, intuitive definitions. Hence, we opt for the efficient and tractable length attribute under this normality assumption.
4

---

Published as a conference paper at COLM 2024
Prompting (Baseline)
Few-Shot Finetuning
SELF-GUIDE
Task ID
Task Description
A
(Ours)
Classification Tasks
task1516
NLI (IMPPRES)
17.6
32.2
35.2
17.6
task1529
NLI (SeiTail)
8.5
48,9
54.5
46.0
task1612
Sentiment Class. (SICK)
51.3
33.3
33.3
-18.0
task1615
NLI (SICK)
0.5
33.3
33.1
32.6
task284
Sentiment Class. (IMDB)
90.0
71.9
82.2
-7.8
task329
Coreferent Class.
29.1
45,4
44.7
15.6
task346
Word POS Class.
35.1
49.9
50.7
15.6
Avg
Metric: Exact Match
33.2
45.0
47.7
14.5
Generation Tasks
task1345
Question Paraphrasing
40.7
50.5
9.8
task281
Find Common Entity
46.8
40.7
49.3
2.5
task1562
Question l'rraprassigg
29.5
48.6
59.3
29.8
task1622
Fluency Correction
49.2
86.2
78.5
29.3
Avg
Metric: ROUGE-L
41.6
52.9
59.4
17.9
Table 1: For each task category (classification and generation), we randomly split the tasks into two halves, one half to tune the parameters for the "One Parameter Fits All" strategy, and the other half to test SELF-GUIDE's performance using this tuned set of parameters, We use the same decoding parameters and prompt template to evaluate the performance of the base model and the Self-Guided expert model the held-out tasks. A is the performance difference between SELF-GUIDE and the Prompting baseline.
2.3 Quality Optimization: Parameter Tuning
We want SELF-GUIDE to generate training data matching to the desired distribution specified by an instruction and examples. This requires optimizing various labeled data points, including the initial number of generated inputs, the temperature for input generation, the temperature for output generation, finetuning parameters like training epochs, and on. To achieve this, tune the parameters on set of existing tasks and their corresponding instructions, and then evaluate the model on held-out dataset. We do so by searching parameters that maximize worst task performance, in order to identify parameters that are likely to be "good enough" for broad set of tasks (Michel et al. 2021
on
max min (performance IEtasks
ICL(())
In Section 3below, we demonstrate that default set of parameters generally performs well with an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
Experimental Setup
3.1 Datasets
To evaluate the effectiveness of SELF-GUIDE, we selected 14 classification tasks and generation tasks from the SuperrNaturalllnsttuttion V2 benchmark. We randomly chose half of these tasks (7 classification and 4 generation) for hyper-parameter search, and the remaining half for evaluation (referred to held-out tasks). Table Tists the held-out tasks, To find the optimal set of parameters as described in Section 2.3 we performed random search over the temperature for input generation and output generation, as well as the number of generated inputs before filtering.
3.2
Base Model
We selected Vicuna-7b-1.5 [Zheng et al.) 2023 Wan et al. 2024 Fan & Tao 2024) as the foundational model for input generation, output generation, and fine-tuning for two primary

---

Published as a conference paper at COLM 2024
reasons: Firstly, having undergone fine-tuning on extensive and diverse corpus of user conversations from ShareGPT, this model demonstrates robust cross-task instruction- following capabilities. However, despite its extensive training, its performance is not deemed satisfactory, as evidenced by its relatively low ranking of 73rd on the LMSYS Chatbot Arena Leaderboard (Chiang et al. 2024), prompting to investigate opportunities for improvement.
We employ the same evaluation metrics in the Naturartttttllllsstuutionssssss V2 benchmark, namely Exact Match for classification tasks and ROUGE-L for generation tasks. Exact Match measures the exact string match between the predicted and ground truth labels, while ROUGE-L computes the longest common subsequence between the generated and reference texts. For fair comparison, we use identical settings for our base model and Self-Guided model, including model size, batch size, decoding strategy (greedy search), and prompt template in Appendix A.1.2 During inference, we leverage the efficient VLLM framework proposed by Kwon et al. (2023). For supervised finetuning with teacher forcing, we employ the Hugging Face TRL library (von Werra et al., 2020) due to its flexibility and ease of use for transformer-based language models.
3.3 Baselines
SELF-GUIDE provides method for executing tasks specified by prompts (which contain instructions and a small number of demonstration examples). We compare SELF-GUIDE against other methods for instruction following and in-context learning:
1. Few-Shot ICL (Prompting): As our primary baseline, we compare against directly prompt- ing the LM. This directly relies on the inherent instruction-followii abilities of the model.
Self-ICL: We build on Self-ICL (Chen et al. 2023), which uses self-generated examples to improve zero-shot instruction following. We adopt Self-ICL to our few-shot setting by self-generating as many examples as can fit the context window. We list this number of examples as k in Table
Few-Shot Finefuning: We consider directly finetuning on the few demonstrations in each prompt. Prior work (Liu et al. 2022) shows this can be very effective.
We use the same base model (Vicuna-7b-1.5) as described in 3.2 for every baseline mentioned above.
4 Results and Analysis
Our main experiment results are shown in Table Specifically, we report an absolute im- provement of 14.5% for classification tasks and 17.9% for generation tasks in the benchmark's metrics. SELF-GUIDE demonstrates its effectiveness in guiding LLMs toward task-specific expertise, even in extremely data-limited situations. This highlights the potential of self- synthesized data in adapting LLMs for specialized tasks from more scalable perspective.
To further analyze the reasons behind the improvement brought about by SELF-GUIDE, we conducted several analysis experiments to examine certain properties of SELF-GUIDE.
4.1 Comparing self-synthesized examples with gold few-shot examples
In Table [ we see that FFLFEE improves Vicuna-7b-1.5's ability to complete most tasks. Given that the SELF-GUIDE algorithm, essence, involves finetuning on self-synthesized data, how much of this performance boost can be attributed to the synthetic data or to the learning algorithm?
To test this, we compare two ways of incorporating synthetic data into an LM: finetuning and in-context learning. For finetuning, we compare SELF-GUIDE with few-shot finetuning three gold demonstrations. For in-context learning, compare adding self-synthesized data into demonstration examples, ie. Self-ICL, with in-context learning using only the three gold demonstrations.

---

Published as a conference paper at COLM 2024
3-Shot Finetuning
3-Shot ICL
Self-ICL (with syn. ex.)
Finetuning
Task ID
SELF-GUIDE
k
(on syn. ex.)
Classification Tasks
task1516
32.2
35.2
17.6
4.5
33.2
60
task1529
48,9
54.5
8.5
0.2
50.0
52
task1612
33.3
33.3
51.3
34.1
33.3
39
task1615
33.3
33.1
0.5
0.0
33.3
task284
71.9
82.2
90.0
10.0
50.0
22
task329
45.4
44.7
29.1
35.2
44.4
3
task346
49,9
50.7
35.1
22.4
49.9
30
Avg
45.0
47.7
33.2
15.2
42.0
30
Generation Tasks
task1345
36.0
50.5
40.7
48.8
50.4
task281
49.3
36.8
51.5
28
task1562
48.6
59.3
29.5
44.1
57.3
30
task1622
86.2
78.5
49.2
50.9
78.4
42
Avg
52.9
59.4
41.6
45.2
59.4
27
Table 2: We compare in-context learning and finetuning on their ability to learn either from 3 manually written demonstrations from varying number of synthetically gen- erated demonstrations. We find finetuning is better at leveraging synthetic data than in-context learning. When comparing Self-Guide with 3-Shot Finetuning, finetuning on self-synthesized data yields better performance than finetuning on gold-standard few-shot examples.
In Table 2 we see that using generated training data with finetuning improves over 3-shot finetuning on average for both classification and generation tasks (comparing the "SELF- GUIDE" and "3-Shot Finetuning" columns). On the other hand, comparing the "3-Shot ICL" and "Self-ICL." columns, we see that using generated training data with in-context learning helps only for generation tasks; for classification tasks, this strategy often leads to deteriorated performance. In Self-ICL for classification tasks, the prompted model tends to generate new input-output demonstrations rather than respond to the final input to be completed, leading to significant performance decrease. We hypothesize that the model fails to segment the instruction from the demonstration examples, thereby failing to systematically classify the provided inputs according to the task definition.
4.2 Finetuning outperforms in-context learning synthetic data
How much is the learning algorithm finetuning versus in-context learning responsibe for the effectiveness of SELF-GUIDE? To test this, we compare in-context learning on as many self-synthesized examples with finetuning on the same set of self-synthesized examples. We find in Table compared to in-context learning, finetuning the same generated examples achieves substantially better performance, with an average improvement of around 20 absolute percentage points across almost all tasks. Interestingly, finetuning on just few self-synthesized examples (e.g. 7) can yield considerable performance gains, which highlights the data efficiency of SELF-GUIDE.
fit in the context window of an LM ("Self-ICL")
4.3 SELF-GUIDE aligns LMs with the correct label distribution in many cases
To demonstrate the high quality of data generated by SELF-GUIDE from quantitative perspective, we analyze the distance between the output distributions of the models before and after SELF-GUIDE and the ground truth distribution for classification tasks.
Specifically, we compute the output distributions of the base model and the Self-Guided expert model on the same task. Considering that the outputs may contain irrelevant greetings or cases where the model directly refuses answer, we collectively treat outputs without labels as irrelevant content and calculate the distribution across all labels and
7

---

Published as a conference paper at COLM 2024
irrelevant content. After obtaining the output distribution, calculate the L1 distance between this distribution and the ground truth distribution.
Accuracy
L1 distance
Irrelevant Ratio
Task ID
Baseline
SELF-GUIDE
Baseline
SELF-GUIDE
Baseline
SELF-GUIDE
task1516
0.18
0.35
1.31
0.81
0.65
0.00
task1529
0.09
0.55
1.79
0.23
0.90
0.00
task1612
0.51
0.33
0.67
1.33
0.00
0.00
task1615
0.01
0.33
1.97
1.33
0.98
0.00
task284
0.90
0.82
0.14
0.33
0.07
0.00
task329
0.29
0.45
0.85
1.09
0.25
0.00
task346
0.35
0.51
0.97
0.16
0.28
0.00
Avg 0.33
0.48
1.10
0.75
0.45
0.00
Table 3: The L1 distance from the ground truth distribution, with lower values indicating better alignment. Notably, the Irrelevant Ratio column indicates the proportion of outputs deemed irrelevant, with the SELF--UIID model effectively reducing this ratio to 0 across all tasks.
From the results in Table average, the SELF-GUIDE model demonstrates significantly lower average L1 distance compared to the baseline model while reducing the proportion of irrelevant content 0. This indicates that the SELF-GUIDE model not only enhances the consistency of outputs with the true distribution but also effectively reduces irrelevant information in generated content, thereby enhancing the overall performance of the model.
4.4 SELF-GUIDE learns non-trivial input-output mapping via self-synthesis
In this experiment, we investigate whether the improved performance of the LM under the SELF-GUIDE framework stems from merely learning trivial patterns like output formatting and label structuring, or also from gaining a better understanding of the task. We consider two scenarios: Rand-Baseline, where we randomize the labels in the demonstrative examples following Min et al. (2022b), and Rand-SELF-GUnD where we randomize the labels of the self-generated examples the SELF-GUIDE approach. Specifically, for the Rand hand-Baseline, each original label in the demonstrative examples is replaced with random label. For Lond-SELF-GUIDE we randomize all the labels of the generated examples in SELF-GUIDE: finetune the base model this randomized data, and evaluate it using the same prompt as in our main experiments.
From results in Table 4 Rand-SELF-GUIDE outperforms the baseline suggesting that even with randomized labels, the self-generated examples provide valuable signals to the model. Additionally, the high ratio of irrelevant content produced by the baseline model, as detailed in Table contrasts sharply with the SELF-GUIDE approach, which eliminates irrelevant outputs entirely, ensuring all outputs align with the expected label space. This demonstrates that self- self-generated examples can introduce basic patterns like output formatting and label structuring to language models. However, GUIDE's improvement gain over Rand-SELF-GUIDE further demonstrates that beyond learning superficial patterns, better supervisory signals during the SELF-GUIDE process enable the worse
model to develop deeper comprehension of the task itself. Finally, performance compared to the baseline confirms that merely observing well-formatted outputs is insufficient- the model crucially requires proper supervision from high-quality demonstrations to truly grasp the task essence. Collectively, these contrasts reveal that the SELF-GUIDE method allows the model to simultaneously acquire shallow output patterns while leveraging the self-generated examples to model the task distribution, leading to comprehensive boost in its capabilities on specific tasks.
8

---

Published as a conference paper at COLM 2024
Task ID
Rand-Baseline
Baseline
Rand-SELF-GUIDE SELF-GUIDE
task1516
18.7
17.6
33.2
35.2
task1529
9.5
8.5
49.9
54.5
task1612
40.6
51.3
33.3
33.3
task1615
6.5
0.5
33.3
33.1
task284
69,9
90.0
50.0
82.2
task329
30.4
29.1
44.5
44.7
task346
36.4
35.1
50.0
50.7
Avg
30.3
33.2
42.0
47.7
Table 4: Comparison of task performance on classification tasks between the SELF-GUIDE, the baseline model, Rand-SELF-GUIDE (where labels of self-generated examples are ran- domized in SELF-GUIDE), and Rand-Baseline (where labels of demonstrative examples are randomized). The best performance on each task is highlighted bold.
4.5 Noise filter is crucial for classification tasks, while length filter is crucial for generation task
Ablation studies results below in Table found that removing the ablation filter decreases classification accuracy by 4.1% while removing the length filter decreases generation accu- racy by 3.7%. We think that the ablation filter is crucial for classification tasks removes outputs with superfluous content, retaining only the labels. For generation tasks, the length filter enhances performance by excluding lengthy or too short responses, improving the ROUGE-L score. These studies demonstrate the effectiveness of our rule-based filters.
Task
w/o both
w/o ablation
w/o length
w/ both
Generation
55.3
59.6
55.7
59.4
Classification
44.4
43.6
47.7
47.7
Table 5: Comparison of task performance using different filters. The best performance each kind of task is highlighted in bold.
In conclusion, we identify several key factors contributing to the efficacy of SELF-GUIDE. Initially, expanding and augmenting the training dataset with self-generated synthetic data derived from few-shot examples proves to be highly effective compared with manually collecting data. Second, SELF-GUIDE is aligned with the findings of prior research (Liu et al. 2022), which posits that finetuning surpasses in-context learning in terms of effectiveness. Third, our analysis reveals that finetuning synthesized data enhances the consistency of outputs with the true distribution and effectively reduces irrelevant information generated content. Fourth, SELF-GUIDE enables LLM to develop deeper comprehension of the task itself while learning superficial patterns. Last, filters are recommended both for the raw model and SELF-GUIDE, and the ablation filter is crucial for classification tasks, while length filter is crucial for generation tasks.
Related Work
Recent studies have demonstrated the efficient execution of language-based instructions
by LLMs through fine-tuning instruction-ban datasets. These datasets comprise pairs of language instruction commands and expected outcomes annotated by humans (Weller et al.)
2020 Mishra et al. 2022). (Honovich et al. 2022) indicate that despite the noise present in LL MM--genertted datasets, they can still serve as effective training resources for instruction fine-tuning, implying that the parametric knowledge in pretrained LLMs contains, with potentially some transformation necessary, inherent understanding of instructions. We extend this hypothesis, proposing that LLMs inherently possess the ability to understand arbitrary instructions and that this ability be exploited to self-generate training data.

---

Published as a conference paper at COLM 2024
Manually crafted datasets have played pivotal role in supervising and augmenting various NLP task systems (Rozière et al. 2023 Yuan et al. 2023). However, traditional manual annotation processes are often time-consuming, labur-intensive, costly, and non-scalable.
Additionally, the possibility of human errors and the lack of domain expertise among annotators in complex tasks cannot be overlooked (Braylan & Lease 2021 Kang et al. 2023).
Recent works have tried to alleviate the human labor associated with training data collection by utilizing the power of stronger models as data generators (Patel et al. 2024 Wang al., 2024 Song et al. 2024). These methods can be broadly categorized based on their reliance
on seed data, the need for human curation, and stronger teacher LLMs. Viswanathan al.)
(2023) Ge et al. (0024) combine retrieval methods and distillation to directly gather training data from external sources without seed data. Zhou et al. (2023) introduced LIMA, which generates possible responses using LLMs given existing instructions, and then curates them with human annotators. Li et al. 2023) automated the generation of instructions through back-translation on large external corpora, eliminating the need for seed data (Self- Alignment). However, these methods still require more powerful models or seed datasets to cultivate high-quality examples. Our work investigates the self-generttion abilities of LLMs as a means of reducing the need for external resources compared to previous efforts.
Self-Instruct (Wang et al. 2023), which was mentioned in inctiioi 1 is the most relevant work to ours. They also use synthetic data self-generation; in their case, they focus on enabling data-efficient general-purpose instruction finetuning. Our proposed method explores the complementary question of how to use self-generation to make an instruction-iinttueed model even better at executing the instructions specified for given task.
Conclusion
In this paper, we propose SELF-GUIDE, an algorithm for large language models to follow task-specific instructions by internally producing synthetic training data and finetuning on this data. The improvement underscores the potential of our approach in enhancing LLMs' task-specific expertise, particularly in data-limited scenarios, and open avenues for exploring advanced techniques in autonomous model adaptation and continuous learning. We hope that our work can chart a path for future research in autonomous self-alignment and improvement mechanisms in Al systems, aligning them more closely with human intentions.
Limitations and Ethical Considerations
The primary limitation of our work is that we focus all of our experiments and models on English NLP tasks. Language technologies already promote systematic inequalities between languages and communities (Blasi et al. 2021 Zhao & Zhang 2024 Zhao et al., 2024); SELF-GUIDE brings the possibility of exacerbating these inequalities. On the other hand, SELF-GUIDE also offers the tempting potential of improving LMs' ability to execute instructions specified in mon-Englin
languages to empower trustworthy non-English LLM- based Agents (Chen et al., 2024 Sun et al. 2024 Huang et al.) 2023). We consider this to be an essential direction for future work, and we are actively pursuing this.
Another limitation of our work is that we've only shown that SELF-GUIDE works at im- proving a 7B-parameter model (Vicuna-7b-1.5), We believe that our approach could be used to improve the ability of arbitrarily large LMs (He al. 2023 Hu et al.) 2024 to follow task-specific instructions. However, due to budgetary restrictions, we were unable to experiment with larger models; we leave this as an imporlant avenue for future work.
In terms of ethical considerations, SELF-GUIDE' potential to improve instruction following on on specific tasks raises the risk of making it easier for people to specialize LLMs for malicious purposes. The open-sourcing of our code could amplify this risk. As Widder et al. 2002 points out (with the case study of deepfakes), open-source software can simultaneously increase the prevalence of harms while providing the community with the understanding and experience to manage these harms. Given SELF-GUIDE' potential for positive use, we believe that this algorithm and code are still worthy of releasing to the public.
10

---

Published as a conference paper at COLM 2024
8
Acknowledgments
This work was supported by the computing resources provided by the Microsoft Accelerate Foundation Models Research Program and the Tencent AI Lab Rhino-Bird Gift Fund. The authors would like to thank the reviewers for their valuable feedback. Additionally, we extend our gratitude to Professor Pengfei Liu Shanghai Jiaotong University for his assistance with this work.
References
Damián E. Blasi, Antonios Anastasopoulos, and Graham Neubig. Systematic inequalities in language technology performance across the worlds languages. ArXiv, abs/ 2222000603 3, 2021. /ttpspp://a....emmaaat. wwt.paawww..aaaaaa..ooooorraaaaa wttp:oorrpS.:::237744.2...
Alexander Braylan and Matthew Lease. Aggregating complex annotations via merging and matching. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD 21, pp. 8694, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383325. doi:
httpsss/::///ol org/
10. Ne.114553447555 .4674411
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin Advances Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran
Associates, Inc,, 2020a. URL http::://proceedinggs neurips wwterrisscccppppaae http:/ppprfffeespppppeeee 02200fflle////.0dd...e44444446f2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
Wei-Lin Chen, Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. Self-ICL: Zero- shot in-context learning with self-generated demonstrations, In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 33651-15661 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 222.2222..mmmpmmain //aclanthology. org/ 2023. smnlp-main. 968
URL https
Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving web of heterogeneous agents for collaborative intelligence. aiviv preprint xxXi::207.00066 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
Xiaojing Fan and Chunliang Tao. Towards resilient and efficient llms: A comparative study of efficiency, performance, and adversarial robustness, 2024. URL https://arxiiv org/
abs/ abs/2408. 04585
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
11

---

Published as a conference paper at COLM 2024
11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 1516-,830, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /v////11////// wwl/022aacllooog..222 295. URL /tl.aatpss//aclaaattoooo...oggo. wwwp://y..rrro22...... acl.oonn...looonn....... 295
Jiaxin Ge, Xueying Jia, Vijay Viswanathan, Hongyin Luo, and Graham Neubig. Training task experts through retrieval based distillation. arXiv preprint urXiv:2407.05463, 2024.
Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, et al. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise. ar arviv preprint ar arXiv:2310.19019, 2023.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. Xrviv preprint ar XrXi:2404.06339 2024.
Yue Huang, Qihui Zhang, Lichao Sun, et al. Trustgpt: A benchmark for trustworthy and responsible large language models. Xiv preprint ar Xiv:2300.111507, 2023.
Mintong Kang, Bowen Li, Zengle Zhu, Yongyi Lu, Elliot K. Fishman, Alan Yuille, and Zongwei Zhou. Labeleassemble: Leveraging multiple datasets with partial labels. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2023. doi: 10.1109/ ISBII3IBI 202.3377777222221077776
Omar Khattab, Arnay Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrat- ing reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022.
https:// https://hugggnggacc..o ho//blog/lll /rlhf.
Tianyi Li, Sabine Weber, Mohammad Javad Hosseini, Liane Guillou, and Mark Steedman.
Cross-lingual inference with chinese entailment graph, 2022.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, 2023.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficiet fine-tuning is better and cheaper than in-context learning. ArXiv, abs/ 2205.05668 2022. URL htt.tpss//p.....mmmtcccc... sswnarcccchhllllrr /ttp//oopuul1::299222220
Paul Michel, Sebastian Ruder, and Dani Yogatama. Balancing average and worst-case
accuracy in multitask learning. ArXiv, abs/2110.05838, 2021. URL https://app
semanticscholar. wrwpaaaoooo/ooooooooooea 2388333327
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirii. Metaicl: Learning to learn in context, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? ArXiv, abs/ 2bss/222.11837,, 2022b. URL https: / //apii semanticscholat org./CarpusID:277155665
12

---

Published as a conference paper at COLM 2024
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions, 2022.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Ajay Patel, Colin Raffel, and Chris Callison-Burch. Datadreamer: A tool for synthetic data generation and reproducible I1m workflows. ArXiv, abs/2402.10379, 2024. URL
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Réréz,Jérémy Rapin, Artyom Kozhevnikov, Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Cantón Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D'efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. ArXiv, abs/2308.12950) 2023. hRLhttts:://app semanticscholaz wttpooooall...rroopouuuDDDDD 261100919
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How leameed to start worrying about prompt formatting. Arxiv, abs/2310.11324, 2023a. hRthtts:://ppi www.nticscholl.n.cmc
org/CorpusID :t:tbrpuuII:2417271000
Melanie Sclar, Yejin Choi, Yulia Isvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2023b.
Zezheng Song, Jiaxin Yuan, and Haizhao Yang. Fmint: Bridging human designed and data pretrained models for differential equation foundation model, 2024.
Lichao Sun, Yue Huang. Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. Xiv preprint ar arXiv:2401.0556) 2024.
Vijay Viswanathan, Chenyang Zhao, Amanda Bertsch, Tongshuang Wu, and Graham Neu- big. Prompt2model: Generating deployable models from natural language instructions. Xiv preprint ar Xiv::3308 12261, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. https //ggitaubb.com//uggginnee www.ggiinngaaeeeeea /trl, 2020.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey, 2024.
Yining Wang, Jinman Zhao, and Yuri Lawryshyn. Gpt-signal: Generative ai for semi- automated feature engineering in the alpha research process. In Proceedings of the Joint Workshop of the 8th Financial Technology and Natural Language Processing, and the 1st Workshop on Agent for Scenario Plamning @ iJCAI 2024, 2024.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Is- hani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Sa- van Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Superrnaturalinstnctioos:: Generalization via declarative instructions on 1600+ nlp tasks, 2022.
13

---

Published as a conference paper at COLM 2024
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022. hattpp::/aaza...r///aassss/////
01652
Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. Learning from task descriptions, 2020.
David Gray Widder, Dawn Nafus, Laura A. Dabbish, and James D. Herbsleb. Limits and possibilities for ethical ai in open source: A study of deepfakes. Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, 2022. URL https
/aapi.ssmanticacho.... wrwp.oopps.//C-C......o. 24944225495
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023.
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Rezz Namazi-Rad, and Jun Wang. How do large language models capture the ever-chagging world knowledge? review of recent advances. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8289-8311, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/ /v/l emnlp-main.511
Jinman Zhao and Xueyan Zhang. Exploring the limitations of large language models in compositional relation reasoning. arxiv preprint arXiv:2403.0066 2024.
Jinman Zhao, Yitian Ding, Chen Jia, Yining Wang, and Zifan Qian. Gender bias in large language models across multiple languages. ar preprint ar arXio:2403.00277, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph F. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
14

---

Published as a conference paper at COLM 2024
A Appendix
A.1 Prompt Template
A.1.1 Input Generation Prompt Template
INPUT CENEERATOR_PROOP_FFOS GENERATION As an InpatGenerator, your task is to generate [input] based the [instruction] and some example [izput].
Try your best to ensure that the new [input] you generate is distinct from the provided [input] while maintaining a diverse, detailed, precise, comprehensive, and high qugh-quality response.
Avoid generating new [input] that is the the provided [input].
(instruction]
(instruction)
Here are some high-quality [input] for the [instruction]. These [input] can provide you with very strict format requirements. You should pay extreme attention to t!!m!!!
Some high-quality [input]:
(higb.qualit..mppt.tttr
These are some additional [input]. Their formats and contents may not be accurate.
However, may also refer to the content of them.
Some lou-quality [input]:
[low_ www.guullty.innut.eee.
After seeing example inputs, generate a new [input]. Before generating the new [input], ensure that you strictly adhere to the rules of the new [instruction] and follow the format of high nigh-quality (input].
Prioritize the new [instruction] guidelines to maintain consistency and quality.
Think twice before generating a new [input]. Only response the new [input] without any other information.
[input]=
unn
INPUT PEWPTEEEEEATRRR_POPTTT.OT D As InpatGenerator, your taak is generate new [input] based on the [instruction] and some example [input].
CLASSIFICATION =
Try your best to ensure that the new [input] you generate is distinct from the provided
15

---

Published as a conference paper at COLM 2024
[input] while maintaining diverse, detailed, precise, comprehensive, and qigh-quality response.
Avoid generating new [input] that is the same the provided [input].
[instruction]
(instruction)
Here are high-quality [input] for the [instruction]. These [input] can provide you with very strict format requirements. You should pay extreme attention to them!!!
Some high-quality [input]:
lhigh.quallty..nnut..ttt
These additional [input]. Their formats and contents may not be accurate. Hovever, you may also refer to the content of them.
Some low-quality [input]:
After seeing example inputs, generate new [input] for which the expected [output] is lconditional_labll.. Before generating the new [input], ensure that you strictly adhere to the rules of the [instruction] and follow the format high-quality (input].
Prioritize the new [instruction] guidelines to maintain consistency and quality.
Think twice before generating a new [imput]. Only response the new [input] without any other information. Note that the expected [output] for the new [input] should be [cond
[input]=
A.1.2 Output Generation Prompt Template
..* chat between a curious user and an artificial intelligence assistant.
CANNOTATION_PROMTTLT..
The assistant gives concise answers the user's 's questions.
USER: The artificial intelligence assistant only needs to help annotate label.
The task is: [instruction]
ASSISTANT: Okay
USER [input]
(input_1} ASSISTANT {output_1) USER [input] = {input_2) 2]
16

---

Published as a conference paper at COLM 2024
ASSISTANT { {output_2]
USER [input]
(input_3)
ASSISTANT {output_3)
USER: [input] = neew_inputi ASSISTANT:
A.2 Prompt Sensitivity Experiment
LLMs have been shown to be highly sensitive prompt formats and minimal changes such punctuation (Sclar et al. 2023b). This sensitivity can lead to drastic fluctuations in model performance, affecting its stability and reliability in real-world applications. Here, we introduce minimal prompt modifications. Although these modifications seem trivial from human perspective, we will show that they greatly impact the base model's performance. However, Gelf-Guided final models prove robust, consistently outperforming the base model across various conditions. Specifically, we use the output generation prompt template (see Appendix A.1.2) to finetune the base model but make minimal disturbances to the few-shot examples of the prompt template when evaluating. Note that in the original format, there is an n= following "USER: [input]". We change this conjunction between "USER: [input]" and the example input to ":", " "nnn"", and evaluate the performance of the base model and the SELF-GUIDE expert model under this condition. Detailed results are shown in Table We find that such simple changes, which may be overlooked by humans, can lead to catastrophic performance decreases the raw model. On the other hand, SELF-GUIDE's performance exhibits remarkable stability and excellence.
Raw Model
SELF-GUIDE Expert Model
Category
Task ID
=
n
n
diff
=\n
\n)n n
diff
task190
27.0
21.1
19.8
7.2
66.7
66.7
66.7
0.0
task1529
8.5
11.0
29.5
21.0
61.6
54.9
53.1
8.5
Classif.
task1612
51.3
50.9
48.8
2.5
42.6
42.4
42.8
0.4
task329
29.1
28.7
27.6
1.5
46.3
45.6
45.9
0.7
Avg
29.0
27.9
31.4
8.1
54.3
52.4
52.1
2.4
task281
46.8
43.4
32.4
14.4
52.3
46.9
47.3
5.4
task1195
49.9
44.1
57.3
13.2
83.7
85.4
85.5
1.8
Generat.
task1345
40.7
36.0
44.0
8.0
53.0
53.1
52.8
0.3
task1562
29.5
29.2
33.0
3.8
62.6
62.7
62.9
0.3
Avg 41.7
38.2
41.7
9.9
62.9
62.0
62.1
2.0
Table 6: Sensitivity of models to prompt conjunctions classification and generation tasks. We made minimal disturbances to the few-shot example section of the original prompt prompt template, changing the conjunction between "USER: [input]" and the example input from "='nn" to "." and "\n)n", and evaluated the performance of the raw language model (Raw Model) and the expert model generated by SELF-GUIDE (Expert Model) under these conditions. The table presents the average scores and differences for the two models across different task categories.
17


Reviews:
[{"review_id": "z6Qyin1tAy", "reviewer": "Reviewer_jbfG", "summary": "The paper proposes Self-guide, a multi-stage strategy to synthesize task-specific input-output pairs from a student LLM, and finetune the student model itself.\nThis approach addresses the challenges of cost and scalability issues of using external powerful LLMs for generating task-specific data.\nEmpirical evaluations on multiple individual tasks from Natural-Instructions-v2 demonstrate performance improvements for classification and generation tasks, surpassing ICL and few-shot finetuning.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting these issues.\n\n\n1. **Data contamination**: We agree that data contamination could be a significant issue! Our experiments exclusively used the test set from the NI dataset. Our base model, Vicuna-1.5-7b, is based on Llama 2, and Meta has not revealed the training data used for Llama 2. However, we believe it is unlikely that Vicuna included this test data during pre-training, because of the low performance of few-shot-prompted Vicuna on these Natural instructions tasks. Moreover, we feel that Self-Guide's effectiveness is somewhat orthogonal to whether the LLMs have encountered these tasks during pre-training, as we intuitively believe that Self-Guide’s strength is in transforming pretrained knowledge (from the text in the training data) into a useful task-specific format. For example, in task 1622, a few-shot prompting experiment achieved a Rouge-L score of 42.5. Self-Guide addresses this knowledge mismatch, resulting in nearly a 30-point improvement in the Rouge-L score. \n\n\n2. **More baselines**: Thank you for suggesting this! Due to character length restriction, please refer to **reply 2 for reviewer ySBP**, where we have supplied results comparing with these baselines!\n\n\n3. **Rule-based filter ablation**: Ablation studies results below found that removing the ablation filter decreases classification accuracy by 4.1% while removing the length filter decreases generation accuracy by 3.8%. We think that the ablation filter is crucial for classification tasks as it removes outputs with superfluous content, retaining only the labels. For generation tasks, the length filter enhances performance by excluding lengthy or too short responses, improving the Rouge-L score. These studies demonstrate the effectiveness of our rule-based filters.\n\n| Ablation | w/o both | w/o abl | w/o len | w/ both    |\n| - | - | - | - | - |\n| Gen Avg  | 55.3     | 59.6    | 55.7    | 59.4       |\n| Cls Avg  | 44.4   | 43.6  | 47.7  | 47.7 |\n\n\n4. **Prompt**: We use the same prompt template for both the ICL baseline and for Self-Guide: an instruction followed by few-shot examples (with the exact template shown in Appendix A.1.2). We tuned this template for few-shot prompting (i.e. tuned on the baseline), with the optimistic view that this would also work for our method. We use the same instructions for each task for both the baseline and for Self-Guide; we use the default instructions for each task provided by the NI dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pros:\n1. Customizing LLMs as task specialists is an important direction that has received relatively less attention.\n2. Using self-generated synthetic data is well-motivated, and this paradigm has been explored and justified as reasonable in many scenarios.", "reasons_to_reject": "Cons:\n1. This paper posits a scenario in which there is a lack of sufficient annotated, task-specific data. However, it cannot ensure that the tasks used in the experiments are entirely new to Vicuna or even Llama. It is likely that numerous in-distribution training instances were included during pre-training, such as those found in crawled web data. This situation has already been acknowledged in some evaluation benchmarks. Given that complete decontamination of the dataset is infeasible, it is not possible to confirm whether the proposed pipeline enhances task-specific knowledge or capabilities within the current experiment design.\n2. More baselines of task-specific data synthesis should be discussed and compared, e.g., ProGen[1] and Zerogen[2].\n3. The contribution of rule-based filters to overall quality improvements is not demonstrated, and their generalization when handling more diverse tasks raises concerns.\n\n[1] ProGen: Progressive zero-shot dataset generation via in-context feedback\n[2] Zerogen: Efficient zero-shot learning via dataset generation", "questions_to_authors": "See the pros and cons above.\n\nAdditional:\n1. What is the prompt for the ICL baseline, and whether this prompt is optimized for the aligned model (i.e. vicuna in the experiments)?", "extra_scores": {}}, {"review_id": "IFKnObwgMx", "reviewer": "Reviewer_ySBP", "summary": "The paper introduces a novel multi-stage mechanism, SELF-GUIDE, which uses self-synthesized data for finetuning language models on task-specific instructions without external data or teacher models. This is a significant advancement in making LLMs more efficient and autonomous.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reply and affirmation!\n\n1. **It could be beneficial to discuss and compare this paper's approach with methodologies outlined in reference Automated Data Curation for Robust Language Model Fine-Tuning**\n\nThank you for bringing this paper to our attention! Since this paper was posted publicly just 10 days before the COLM deadline, we believe this counts that this should be considered “concurrent work” to ours. Regarding the methodologies in that paper, the authors propose an effective auto-filter and auto-correct strategy that is similar to our filter strategy. However, their focus is on optimizing manually-annotated training data, rather than the model’s self-generated data. We will add a discussion and comparison with that paper in our final version.\n\n2. **there might be a need to delve deeper into its connection with other relevant works in the field, particularly concerning data curation and enhancement techniques for language models**\n\nApart from the mentioned paper, we have compared Self-Guide with some current state-of-the-art (SOTA) methods. The methods compared are:\n\n- ZeroGen: Efficient zero-shot learning via dataset generation\n- ProGen: Progressive zero-shot dataset generation via in-context feedback\n- PET: It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners\n\nThe results we obtained are as follows:\n\n| Task | ZeroGen | ProGen | PET | Self-Guide |\n|-|-|-|-|-|\n| Gen Avg | 57.4 | 56.8 |57.3| 59.4 |\n| CLS Avg | 41.9 | 44.2 |42.0| 47.7 |\n\nBy comparing these SOTA methods, we have demonstrated that Self-Guide can achieve better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The empirical evaluation shows substantial improvements in task performance (15% for classification tasks and 18% for generation tasks) when using the SELF-GUIDE approach, compared to traditional prompting and finetuning methods. This indicates the method's effectiveness.", "reasons_to_reject": "While I do not have a strong reason to outright reject this paper, there might be a need to delve deeper into its connection with other relevant works in the field, particularly concerning data curation and enhancement techniques for language models. For instance, it could be beneficial to discuss and compare this paper's approach with methodologies outlined in reference [1] \"Automated Data Curation for Robust Language Model Fine-Tuning.\" Such a comparison could elucidate how the SELF-GUIDE method aligns with, diverges from, or improves upon existing strategies for generating and utilizing synthetic data to train language models more effectively.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SGWDNxhvjq", "reviewer": "Reviewer_yJTC", "summary": "SELF-GUIDE works by first employing the target model to generate a synthetic dataset for a given task. The model\nis then finetuned on this “self-generated” data. The proposed method is similar to Self-Instruct work, which is mainly for general domains. And  SELF-GUIDE is more on domain specific tasks and can effectively self-generate hundreds of examples for a\ngiven instruction.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting these issues.\n\n1. **It would be better to test more general tasks**: We chose 11 diverse tasks from Natural Instructions, so we believe that we have already shown the generality of our method. Nonetheless, during the rebuttal period, we ran Self-Guide on all the BBH tasks, and on average it showed an absolute improvement of 3.4% over few-shot-prompting the same base model (Vicuna). The detailed experiment results will be attached in the camera-ready.\n\n2. **It would be better to report the variance**: For the variance across multiple runs on the same task, it is a good suggestion and we will implement it in the camera ready (we did not have time to do so during the rebuttal period). For the variance across tasks, the results are shown below:\n\n| std | Self-Guide | Baseline |\n| - | - | - |\n| Gen | 13.5 | 8.8 |\n| CLS | 17.5 | 30.3 |\n\n3. **Need some ablation study on Rule-based Filters**: We add ablation studies to the filters. Please see the results and analysis in **reply 3 to reviewer jbfG.**\n4. **The proposed method is still very similar to Self-Instruct**: We did discuss Self-Instruction in our current draft, in Section 2: \"In contrast to their method, which aims to improve general-purpose LLM capabilities, our method aims to optimize an LLM for a specific task instruction. Methodologically, Self-Instruct generates a large set of instructions and demonstrations to use for instruction-finetuning. While Self-Instruct asks an LLM to self-generate synthetic demonstrations for each synthetic instruction, their method only generates a single demonstration for each instruction, while our method can effectively self-generate hundreds of examples for a given instruction. Our methods are complementary.\" Happy to discuss this further if you have more questions! \n5. **Similar settings have also been explored before**: We compare Self-Guide with SOTA methods, which you can see in **reply 2 for reviewer ySBP**, where we have supplied results comparing with these baselines!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well rewritten\n2. The proposed method is simple and effective", "reasons_to_reject": "1. The data generation and filtering strategy may vary across tasks. The evaluation tasks need to be more comprehensive.  It would be better to test more general tasks, such as MMLU, BBH, etc. \n2. Considering the smaller scale of the experiment, it would be better to report the variance. \n3. Need some ablation study on Rule-based Filters. \n4. The proposed method is still very similar to self-instruction, although in a different setting. And similar settings have also been explored before, such as \"It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners\". The authors need a better data-generation method than the proposed one in the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jE8LbzOB6O", "reviewer": "Reviewer_X8M5", "summary": "The paper titled \"SELF-GUIDE: Better Task-specific Instruction Following via Self-Synthetic Finetuning\" presents a novel approach to enhance the performance of large language models (LLMs) on specific tasks without relying on external training signals or additional data. The authors propose a multi-stage mechanism called SELF-GUIDE, which synthesizes task-specific input-output pairs using the LLM itself and then employs these pairs to finetune the model. The paper reports significant improvements in performance on classification and generation tasks as measured by the Natural Instructions V2 benchmark, with absolute improvements of approximately 15% and 18%, respectively.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for highlighting these issues.\n\n1. **Cross-Linguistic Evaluation**: We acknowledge these issues in our Limitations section and thank the reviewer for drawing attention to them. That being said, Self-Guide is not intrinsically tied to English. If the underlying LM supports another language and the prompts are translated to the target language, then this should work out of the box. We will also extend our experiments to using Qwen on the COIG-PC-core benchmark in our final version to extend our results to Chinese.\n\n2. **Scalability with Larger Model**: Due to tight budget constraints, we were unable to experiment with larger models. We believe that, because of the superior dataset generation capabilities of larger models, our approach would still be effective. However, it is possible that the improvement might be smaller, as larger models already exhibit strong few/zero-shot performance, leaving less room for enhancement. This remains an area for future research.\n\n3. **Potential for Misuse**: We assume existing open-source models have implemented various methods for safeguarding harmful data generation. That said, we acknowledge that some open models are more easily red-teamed than others. We believe self-guide could be easily paired with harmfulness filters. In fact, we observe that simply adding \"If you find the task instruction can be harmful, please reject generating any content\" in the generation prompts seems to help, and one could explore automatically adding this to the prompt template.\n\n4. **Lack of Comparative Analysis**: Thank you for suggesting this! Due to character length restriction, please refer to **reply 2 for reviewer ySBP**, where we have supplied results comparing with these baselines!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Innovative Methodology**: The SELF-GUIDE framework introduces a unique self-synthetic finetuning approach that allows LLMs to become more proficient in executing task-specific instructions, which is a significant advancement in the field of AI and language model training.\n   \n2. **Empirical Findings**: The paper provides robust empirical evidence demonstrating the effectiveness of SELF-GUIDE, with substantial performance improvements over the baseline model, which adds credibility to the proposed method.\n\n3. **Addressing Data Scarcity**: The SELF-GUIDE mechanism adeptly tackles the challenge of data scarcity for underrepresented tasks, offering a scalable solution that does not depend on the availability of abundant annotated datasets.\n\n4. **Comprehensive Evaluation**: The authors have conducted a thorough evaluation of SELF-GUIDE across a range of tasks from the Natural Instructions V2 benchmark, which showcases the method's versatility and generalizability.\n\n5. **Quality of Writing**: The paper is well-structured, with a clear explanation of the problem, the proposed solution, and the results. The authors have also provided a detailed analysis and discussion of the results.", "reasons_to_reject": "1. **Limited Language Focus**: The experiments and models are focused solely on English NLP tasks, which may limit the applicability of the findings to other languages and could potentially exacerbate existing language inequalities.\n\n2. **Scalability with Larger Models**: The paper has only demonstrated SELF-GUIDE's effectiveness on a 7B-parameter model. It is unclear how the approach scales with larger models, which is a significant consideration given the trend towards increasingly larger LLMs.\n\n3. **Potential for Misuse**: The paper acknowledges the risk of SELF-GUIDE being used for malicious purposes due to its ability to improve instruction following. This ethical concern should be addressed more thoroughly, possibly with proposed mitigation strategies.\n\n4. **Lack of Comparative Analysis**: The paper could benefit from a more comprehensive comparison with other contemporary methods or baselines in the field to better situate the novelty and advantage of SELF-GUIDE.", "questions_to_authors": "1. **Cross-Linguistic Evaluation**: Could the authors comment on the potential of SELF-GUIDE to be effective across different languages, and what are the challenges and considerations for such an extension?\n\n2. **Scalability Analysis**: How does SELF-GUIDE perform with larger models, and what are the expected computational and scalability challenges?\n\n3. **Ethical Safeguards**: What safeguards are proposed to mitigate the potential misuse of SELF-GUIDE, especially concerning the creation of task-specific models for harmful purposes?\n\n4. **Comparison with State-of-the-Art**: Can the authors provide additional comparisons with other leading methods to further establish the superiority of SELF-GUIDE?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes "Self Guide", an interesting approach to fine-tuning smaller LMs: Task the student model itself to generate synthetic task-specific data, and then fine-tune on it this. The authors present empirical evidence supporting this strategy on a range of classification and generation tasks. 

There were some concerns about how novel this approach is in light of recent related work, and relatedly whether all relevant baselines were compared to. However, mitigating this weaknesses, one of the apparently related efforts seems to be appropriately noted by the authors as concurrent work, and they provided additional comparison to other efforts in their rebuttal.
Decision: Accept