One-Shot Learning as Instruction Data Prospector for Large Language Models (2024)

Abstract: Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance. To address this challenge, we introduce NUGGETS, a novel and efficient methodology that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. NUGGETS assesses the potential of individual instruction examples to act as effective one-shot learning instances, thereby identifying those that can significantly improve performance across diverse tasks. NUGGETS utilizes a scoring system based on the impact of candidate examples on the perplexity of a diverse anchor set, facilitating the selection of the most advantageous data for instruction tuning. Through comprehensive evaluations on two benchmarks, including MT-Bench and Alpaca-Eval, we show that instruction tuning with the top 1% of examples curated by NUGGETS substantially outperforms conventional methods employing the entire dataset.

heading: 1 Introduction
text: Large language models (LLMs) (Brown et al., 2020; OpenAI, 2023; Google, 2023; Bai et al., 2023; Li et al., 2023a) have showcased remarkable capabilities (Wei et al., 2022; Schaeffer et al., 2023; Liu et al., 2023; Cheng et al., 2024) across a wide range of language tasks by scaling the model size and training data. Despite their proficiency, it is imperative to further enhance their alignment with human instructions. This alignment process involves supervised fine-tuning (SFT) on input-output pairs, known as instruction tuning. Instruction tuning is a crucial step, serving not only to activate the
*Corresponding authors †Work done during the internship at Alibaba Group. https://github.com/pldlgb/nuggets
valuable knowledge acquired by LLMs during pretraining but also to facilitate their interaction with humans in a manner that aligns with natural conversational dynamics.
Considerable efforts in instruction tuning have been concentrated on collecting larger (Chung et al., 2022; Wang et al., 2022b), more diverse (Sanh et al., 2022; Sun et al., 2023; Wang et al., 2023b), and intricate (Xu et al., 2023a; Wei et al., 2023) datasets. This is commonly achieved through human crowd-sourcing (Aghajanyan et al., 2021; Ouyang et al., 2022; Tang et al., 2022) or extracting data from larger pre-existing models (Wang et al., 2022a; Taori et al., 2023; Chiang et al., 2023; Xu et al., 2023a). Despite the growth in the size of datasets employed for instruction tuning, certain studies (Zhou et al., 2023; Chen et al., 2023; Cao et al., 2023) suggest that smaller yet valuable datasets tend to be more effective in harnessing the capabilities of LLMs. Blindly expanding the volume of instruction data without ensuring quality may introduce noise and lead to hallucination issues (Zhang et al., 2023c; Zhao et al., 2023a). However, there is a lack of standard criteria for selecting high-quality instruction data (Li and Qiu, 2023; Har-Peled and Mazumdar, 2004; Xia et al., 2023a; Zhang et al., 2024). As depicted in Figure 1, the common practice depends on empirical methods for data selection (Xia et al., 2023b), introducing bias in determining data combinations and adjusting based on outcomes. This trial-anderror approach elevates alignment costs for models. We posit that optimal instruction combinations are present within the extensive data available, yet an efficient and cost-effective identification method remains underexplored.
In this paper, we introduce NUGGETS, a simple yet efficient method that harnesses LLMs as data explorers through one-shot (in-context) learning. This method facilitates extracting high-quality, valuable data from expansive instruction datasets.
4586
Intuitively, an instructional example holds value in training if it serves as an excellent one-shot demonstration for a specific task. If it can facilitate many tasks, it will be worth being treated as a prime data focus, i.e., "gold instruction". Another noteworthy perspective arises from the observation that in-context learning (Dai et al., 2022; Yang et al., 2023; Wang et al., 2023a) employs prompting to implicitly fine-tune the model, while instruction tuning operates through gradient descent. Leveraging the performance of in-context learning offers a promising avenue to predict the effects of instruction tuning. Concretely, we first select a set that spans multiple tasks, designated as the anchor set, and the dataset of instructions to be optimized is identified as the candidate set. One example is sequentially chosen from the candidate set to act as a one-shot example for in-context learning. Subsequently, it is scored based on its impact on the perplexity of each anchor example. This scoring mechanism enables the inference of dependencies between anchor and candidate examples, providing a reference standard for data selection.
To evaluate the effectiveness of the proposed NUGGETS, we conduct extensive evaluations on two widely recognized benchmarks, namely MTBench (Zheng et al., 2023) and Alpaca-Eval (Li
et al., 2023d). We choose a popular and powerful LLM, LLaMA (Touvron et al., 2023a), as our base model. Experimental findings demonstrate that the NUGGETS’ data filtering strategy engenders a significant improvement in comparison to vanilla fine-tuning approaches.
We summarize our main contributions as follows:
• We present NUGGETS, a methodology designed to dynamically assess the quality of instructional examples by using LLMs themselves. NUGGETS is expected to extract the most valuable data from a vast pool of instruction data for the purpose of fine-tuning.
• Fine-tuning LLMs with solely the top 1% of highest-scoring instructional examples yields superior results than using the entire instruction dataset. This observation underscores the significance of prioritizing the quality and strategic composition of the training data over sheer volume.
• The results of extensive experiments substantiate our hypotheses regarding "golden instructions", indicating that the effectiveness of an instructional example is measured by its impact on the task generalization capability of
the model following the fine-tuning process. This observation holds considerable promise, potentially providing valuable insights for future endeavors in data quality screening.

heading: 2 Related Work
text: Instruction Tuning Recent works have introduced a series of techniques that aim to refine large language models (LLMs), showcasing their ability to generalize effectively to instructions not encountered before. For instance, T5 (Raffel et al., 2020) pioneered the initial effort of training various natural language processing (NLP) tasks in a unified text-to-text format. FLAN (Wei et al., 2021) introduced the novel concept of instruction tuning, aiming to improve zero-shot task performance by transforming NLP tasks into natural language instructions during model training. Furthermore, InstructGPT (Ouyang et al., 2022) handled a wide array of human-created instructions encompassing diverse forms and a broad range of task types tailored for real-world user scenarios. In the absence of the source code release for these notable projects by OpenAI, subsequent efforts by Alpaca (Taori et al., 2023; Peng et al., 2023) and Vicuna (Chiang et al., 2023) were undertaken to explore open-domain instruction tuning, employing the open-source LLM LLaMA (Touvron et al., 2023a).
Instruction Construction The fine-tuning instruction datasets by previous methods are often created manually or tailored to specific tasks. To alleviate the issue of extensive human annotations and manual data gathering, various semi-automated techniques have emerged. Self-Instruct (Wang et al., 2022a) randomly selected a limited number of instances from the initial task pool and used them as demonstrations to guide a language model in generating new instructions, along with their corresponding input-output pairs. Evol-Instruct (Xu et al., 2023a) adopted a progressive modification strategy for the original instructions, which facilitated precise control over the difficulty and complexity levels of the generated instructions. TreeInstruct (Zhao et al., 2023b), in contrast to SelfInstruct or Evol-Instruct, guided LLMs by instructing them to append a specified number of new nodes to the semantic tree of an existing instruction rather than directly manipulating the text sequence. Conversely, certain investigations are oriented towards augmenting the performance of LLMs by leveraging a reduced yet higher-quality set of in-
struction examples. LIMA (Zhou et al., 2023) demonstrated remarkably strong performance by strategically selecting a thousand high-quality data points for learning. InstructMining (Cao et al., 2023) introduced a collection of carefully chosen natural language indicators for evaluating the quality of instruction-following data. Notably, this approach necessitates the division of data into multiple bins. Consequently, it encounters limitations in assessing the quality of individual examples at a fine-grained level. INSTAG (Lu et al., 2023) proposed an open-set instruction tagging method to identify the semantics and intentions of human instructions through tags, providing definitions and quantified analyses of instruction diversity and complexity. Moreover, ALPAGASUS (Chen et al., 2023) utilized the capabilities of an external and powerful model, ChatGPT, to directly evaluate each example. Despite the proven efficacy of this approach, a notable limitation lies in its inability to account for the inherent variations present in each model subjected to fine-tuning. It predominantly relies on the predilections of ChatGPT. Although Li et al. (2023c) proposed a self-guided method for selecting data in instruction tuning, it still requires preliminary fine-tuning of the model, introducing uncertainty into subsequent operations.

heading: 3 NUGGETS
text: Motivation As illustrated in Figure 1, the conventional paradigm for enhancing instructional data in the fine-tuning process of large language models (LLMs) has predominantly relied on empirical methods. These methods encompass the application of heuristic rules, expert analysis, and iterative adjustments to the data guided by feedback on model performance. Notably, this trial-and-error approach imposes significant costs in terms of both human effort and computational resources.
Recent scholarly consensus suggests that instruction tuning significantly enhances the task generalization capabilities of pre-trained models across various specific tasks (Longpre et al., 2023; Zhang et al., 2023a,b; Shu et al., 2023). In light of this, we posit the hypothesis of a golden instruction: the efficacy of an instructional example is gauged by its influence on the task generalization capability of the model subsequent to the fine-tuning procedure. As the extent of improvement becomes more conspicuous, the instruction gravitates towards classification as “golden instruction”.
According to this hypothesis, a straightforward method involves fine-tuning an independent model using an instruction example and then comparing the performance of the fine-tuned model with the base model on a predefined dataset containing multiple tasks. This process aims to discern whether the given example qualifies as a “golden instruction”. However, this method would lead to an impractical proliferation of fine-tuned models, equivalent to the number of distinct instructions. Furthermore, fine-tuning with only a single example may introduce unstable updates to the model’s gradients, making it challenging to ascertain the genuine acquisition of the example. Motivated by the inherent duality between In-Context Learning (ICL) and gradient descent (Dai et al., 2022; Aizerman et al., 1964; Yang et al., 2023; Irie et al., 2022), we “fine-tune” the instruction implicitly through one-shot learning, replacing the need for actually fine-tuning the model. More information can be found in Discussion 5.
Overview The framework of our NUGGETS is illustrated in Figure 2. Firstly, we evaluate the proficiency of LLMs across a diverse range of tasks using a predefined set of tasks, denoted as the zeroshot score. Subsequently, we designate each example from the instruction dataset as a distinct one-shot prompt, concatenating it in front of the predefined tasks. We then recalibrate the model’s completion level for these tasks, referred to as the one-shot score. By exploiting the disparity between one-shot and zero-shot scores, we can compute the golden score for each instruction. Once the golden scores for all instructions are computed, we can identify the highest-scoring subset, deemed the golden subset, which is subsequently provided directly to the model for the fine-tuning process.

heading: 3.1 Algorithm Details
text: Zero-Shot Score Given a predefined task set, it encompasses a variety of m tasks, where each task is structured as [Task (T),Answer (A)]. Each word in Task or Answer is denoted as wTi or w A i . Let LLM denote the pre-trained base large language model we use. For the j-th task that is represented by Tj , the probability of zero-shot inference by the model can be calculated by continuously predicting the next tokens based on the given task and the
preceding words:
sjzsl = 1
L
L∑
i=1
log p(w Aj i |C;LLM),
C = [Tj , w Aj 1 , w Aj 2 , . . . , w Aj i−1],
(1)
where L is the number of words of the ground-truth answer A. The score sjzsl is employed to signify the extent of the model’s proficiency on the j-th task. A higher sjzsl denotes superior model performance on the j-th task, whereas a lower sjzsl implies inferior performance. Therefore, we can acquire the model’s performance across m tasks as:
Szsl = [s 1 zsl, s 2 zsl, . . . , s m−1 zsl , s m zsl]. (2)
One-Shot Score With an instruction tuning dataset D, we aim to identify a set of examples Dgold that most closely align with the golden instructions. For each example zk = [InstructionQk (IQk), Instruction A k (IAk)], we initially perform implicit instruction tuning on the base model using that specific example. Here, InstructionQk denotes the question associated with the k-th example zk ∈ D, while InstructionAk signifies its corresponding answer. Subsequently, we employ the model with in-context learning to conduct another round of testing on the tasks within the predefined task set. That is,
sjiit(zk) = 1
L
L∑
i=1
log p(w Aj i | IQk, IAk︸ ︷︷ ︸
One-Shot Prompt
, C; LLM),
C = [Tj , w Aj 1 , w Aj 2 , . . . , w Aj i−1],
(3)
where IQk and IAk can be considered one-shot prompt. Similarly, we can obtain the performance of the model after implicit fine-tuning across m different tasks:
Skiit = [s 1 iit(zk), s 2 iit(zk), . . . , s m−1 iit (zk), s m iit (zk)].
(4) Afterward, we use the Golden Score (GS) to reflect the impact of this instruction tuning example on the base model. The GS of the example zk is calculated as
GS(zk) = 1
m
m∑
i=1
I [ siiit(zk) > s i zsl ] ∈ [0, 1],
(5)
where I[·] is the indicator function. At a high level, the GS measures the increment of performance improvement of the model after one-shot learning through the given instruction.
In this study, we calculate the GS score for each instructional example, facilitating the generation of a ranked list of scores encompassing the entire set of examples. Our objective is to explicitly finetune the base model by selectively employing a small subset comprising the most pivotal examples. Specifically, we prioritize examples exhibiting high golden scores, aiming to achieve superior outcomes compared to utilizing the entire dataset.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Instruction Dataset We adopt the Alpaca dataset (Taori et al., 2023) as instruction data. It is an important resource in the open-source community for instruction tuning, which is constructed by employing the self-instruct (Wang et al., 2022a) method to distill instruction data from text-davinci-
003. The success of this dataset in fine-tuning the LLaMA model has sparked a series of explorations into instruction fine-tuning (Li et al., 2023b; Ji et al., 2023; Xu et al., 2023b). Besides, we perform more types of instruction datasets to verify the transferability of NUGGETS, please refer to B.
Predefined Task Set The predefined task set plays a crucial role in computing golden scores for instructions. These data are employed to evaluate the model’s ability to generalize across diverse tasks. The adequacy of the predefined task set is contingent upon its encompassing a substantial volume of data and incorporating a broad range of tasks. As the Alpaca dataset inherently possesses these attributes, we randomly choose 1,000 examples from it to constitute the predefined task set.
Evaluation Datasets This work uses two methods to assess the model’s capabilities. The first approach involves rating the responses generated by models on a scale ranging from 1 to 10. For this purpose, we utilize the GPT-4 labeled MT-
Bench (Zheng et al., 2023) dataset, which evaluates instruction-following proficiency across eight categories: writing, roleplay, extraction, reasoning, math, coding, STEM, and humanities. Notably, since we only fine-tune on single-turn instruction data, the evaluation is restricted to Turn 1 of MTBench, similar to previous studies (Cao et al., 2023; Zheng et al., 2023; Chen et al., 2023). The second method involves comparing the model’s generated responses with those produced by the Davinci-003 model, employing the well-established AlpacaEval dataset (Li et al., 2023d). This dataset adopts the “win_rate” as the evaluation metric.
Implementation Details In our experiments, we designate the LLaMA-7B model as the foundational model. To ensure a fair comparison, we also set the maximum input length for the models finetuned with the Alpaca dataset to be consistent with LLaMA, which is 2048. In the model fine-tuning phase, we employ the Adam optimizer with a learning rate of 2 × 10−5 and utilize a batch size of 64, conducting training over three epochs. In the subsequent model evaluation phase, we maintain all parameter settings consistent with the original work (Li et al., 2023d; Zheng et al., 2023).

heading: 4.2 Experimental Results
text: The Alpaca dataset comprises a total of 52,002 instruction examples, and the distribution of their golden scores is illustrated in Appendix A. Among these examples, 42,460 instances exhibit a golden score surpassing 0.5. In addition, a subset of ex-
amples closely aligned with the golden instructions has been selected, specifically those attaining golden scores above 0.8 and 0.85. In particular, there are 7,525 examples with golden scores surpassing 0.8 and 619 examples with golden scores exceeding 0.85. Notably, the latter subset constitutes a mere 1% of the entire dataset.
We conduct instruction tuning on the LLaMA model using various subsets of examples distinguished by their golden scores: those with scores less than 0.5, greater than 0.5, greater than 0.8, greater than 0.85, and the complete dataset. The fine-tuned models are denoted as Alpaca≤0.5, Alpaca>0.5, Alpaca>0.8, Alpaca>0.85, and Alpacafull, respectively.
Main Results The experimental results are presented in Table 1 and Table 2 for the Alpaca-Eval and MT-Bench benchmarks, respectively. As expected, Alpaca>0.8 produces the most impressive outcomes. This can be attributed to its ability to maintain an optimal balance between the volume and quality of the instructions it utilizes, leading to the most desirable results. We also note that incorporating lower-quality instructions adversely affected model fine-tuning. This trend is clear when we see that Alpaca≤0.5 lagged behind Alpacafull in performance, while Alpaca>0.5 shows a slight edge over Alpacafull. Remarkably, Alpaca>0.85, using only 1% of the dataset for fine-tuning, achieved results comparable to or even surpassing those of Alpacafull. This underscores the efficacy of our data selection method. More qualitative results can
be found in Appendix C.
Ablation on Predefined Task Sets To evaluate how different predefined task sets affect the selection of instruction data for fine-tuning, we include two additional predefined task set variations. One is randomly exampled from the Alpaca dataset but with a smaller task set size, which is limited to 100 examples. The other one entails clustering the Alpaca dataset into 100 clusters using the K-Means algorithm and selecting the centroids of each cluster as examples of the task set.
We use the two predefined sets to calculate golden scores for the Alpaca dataset separately. The distribution of golden scores is depicted in Figure 3. We select instruction data with golden scores less than or equal to 0.5, greater than 0.5, greater than 0.6, greater than 0.7, greater than 0.8, greater than 0.85, and greater than 0.9 for model fine-tuning, respectively. Table 3 suggests that with random sampling, increasing the size of the task set can enhance the identification of high-quality instruction data. The logic behind this is that a larger encompasses a broader diversity of data, facilitating a more nuanced assessment of an instruction’s effect on model task generalization. However, a shift occurs when K-Means is employed to cherry-
pick more distinct examples for the task set. With as few as 100 examples, K-Means outshines the results from 1,000 examples acquired through random sampling. In this instance, Alpaca>0.8 delivered a superior performance with just 5,419 examples, compared to the 7,524 examples seen with Random1000. This outcome also indirectly confirms the validity of our hypothesis regarding the definition of golden instructions.
Ablation on Instruction Sets To delve deeper into the generalization capabilities of NUGGETS across varied instruction datasets, we undertake a series of experiments utilizing the Alpaca-GPT4 dataset (Peng et al., 2023). It generates instructional data from the powerful GPT-4 model (OpenAI, 2023), which is considered to have superior data quality. Additionally, it shares the same questions in instructions with the Alpaca dataset, which facilitates our direct comparison between the two.
Inspired by Table 3, we employ the K-Means algorithm on the Alpaca-GPT4 dataset to sample 100 examples, forming the predefined task set. Subsequently, we apply the NUGGETS method to score all instructions in the dataset with the golden score, as depicted in Figure 4. Compared to the Alpaca dataset, the Alpaca-GPT4 dataset boasts a higher
number of instructions with golden scores: 25,100 instructions exceed a score of 0.8, 16,943 surpass 0.85, and 4,250 instructions exceed 0.9. These numbers far exceed the corresponding high-scoring instructions in the Alpaca dataset. This also demonstrates that the golden score can serve as an absolute metric to assess the quality of instructional data. The results from model fine-tuning indicate that on the Alpaca-GPT4 dataset, conclusions align with those of previous experiments. The large language models fine-tuned on subsets with golden scores less than or equal to 0.5 exhibit the poorest performance, with a win rate of only 19.23% in the Alpaca-Eval benchmark. In contrast, the models fine-tuned on subsets with golden scores greater than 0.85 demonstrate superior performance, boasting a high win rate of 72.05%. This success can be attributed to the dual assurance of quantity and quality in this particular subset. It is worth emphasizing that fine-tuning on a small and high-quality dataset consistently and significantly outperforms the results of fine-tuning on the full dataset. Overall, the models fine-tuned using Alpaca-GPT4 significantly outperform those fine-tuned with Alpaca. This implicitly corroborates the superior quality of the Alpaca-GPT4 dataset compared to the Alpaca dataset. For more experiments on instruction datasets, please refer to Appendix B. Ablation on Foundation Models To verify the transferability of the NUGGETS method, we conducted experiments on different foundation models using the Alpaca instruction dataset. We selected LLaMA2 (Touvron et al., 2023b) and Mistral (Jiang et al., 2023) at the 7B size as the new base models. The distribution of the golden scores and the performance of models fine-tuned on corresponding subsets of instructions are shown in Table 4. We found that the NUGGETS method is also applicable to other models. LLaMA2 achieved the best results under fine-tuning on subsets with a golden score greater than 0.85, reaching 34.98, which is significantly higher than the 26.47 achieved under full data. Although the absolute value of the win_rate for the Mistral series of fine-tuned models is somewhat low, their performance is also significantly boosted by the NUGGETS data filtering.

heading: 5 Discussion: One-Shot Learning as
text: Implicit Instruction Tuning
Transformer has risen as the prevailing architecture for language models, where self-attention plays a crucial role as a pivotal element within Transformer. Let Xins,Xtest ∈ Rdin denote the instruction tuning sample and the test input respectively. Xins can be likened to IQk and IAk in Equation 3, while Xtest can be seen as T and wA1 , w A 2 , . . . , w A i−1. That Q = WQX⊤test be the attention query vector, K = WK [Xins∥Xtest] be the attention key vector and V = WV [Xins∥Xtest] be the attention value vector, where ∥ represents concatenation operation, WK ,WV ,WQ ∈ Rdout×din are the projection matrices for computing the attention queries, keys, and values, respectively. The result of self-attention in an arbitrary layer for a head is formulated as:
Attention(K,V,Q)
= WV [Xins∥Xtest]Softmax ( WK [Xins∥Xtest]⊤Q√
din
)
≈ WV [Xins∥Xtest] (WK [Xins∥Xtest])⊤ Q = WV Xtest(WKXtest)
⊤ ︸ ︷︷ ︸
Only test input.
Q +WV Xins(WKXins)⊤︸ ︷︷ ︸ Only instruction sample. Q
= Wzsl Q +∆Wiit Q = (Wzsl +∆Wiit) Q, (6)
where √ din serves as a scaling factor. The term WV Xtest(WKXtest) ⊤ could be denoted as Wzsl, which represents the zero-shot learning scenario where no instruction tuning is performed since it solely focuses on the test input. In addition, the term WV Xins(WKXins)⊤ can be seen as implicit instruction tuning ∆Wiit achieved via the metagradient (Dai et al., 2022; Yang et al., 2023) derived from the instruction sample. Readers can refer to previous papers (Dai et al., 2022; Aizerman et al., 1964; Irie et al., 2022) for more details on implicit instruction tuning.

heading: 6 Conclusion
text: This paper presents NUGGETS, a method leveraging LLMs to discern more pivotal data for instruction tuning. Grounded in one-shot learning, this approach facilitates the identification of examples’ value, enabling efficient data selection without dependence on additional annotation and associated costs. Benefiting from NUGGETS, we observe improved instruction following abilities even with smaller training subsets. Furthermore, we posit that
our method underscores the significance of meticulous data selection, offering valuable insights for future instruction fine-tuning endeavors.
Limitations
Although the efficacy of the proposed approach has been confirmed through empirical experiments, opportunities for refinement persist. One avenue for improvement involves a thorough investigation into the inclusion of a diverse and compact set of predefined tasks during the golden scoring phase. This exploration aims to enhance the efficiency of model evaluation on instructional data, leading to improved identification of high-quality instructions suitable for subsequent model fine-tuning. Secondly, due to resource constraints, the majority of experiments in this study are confined to the LLaMA-7B model. While this model holds significant influence within the large language model open-source community, comprehensive validation across a broader spectrum of models is imperative to ensure the generalizability of the proposed approach. Lastly, to fortify the empirical foundation of our findings, it is crucial to subject the proposed method to validation on a more extensive array of instructional datasets. This step aims to ascertain the robustness and applicability of the methodology across a diverse range of instructional contexts, contributing to its broader utility in real-world scenarios. These outlined avenues for future work are anticipated to refine and extend the scope of our proposed method.

heading: Acknowledgments
text: Min Yang was supported by National Key Research and Development Program of China (2022YFF0902100), National Natural Science Foundation of China (62376262), the Natural Science Foundation of Guangdong Province of China (2024A1515030166), Shenzhen Science and Technology Innovation Program (KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039). This work was supported by Alibaba Group through Alibaba Innovative Research Program. Xiaobo Xia was supported by the Australian Research Council project: DE190101473 and Google PhD Fellowship. Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.

heading: A The Distribution of Golden Score
text: As shown in Figure 5, in a total of 52,002 cases, there are 9,549 instructions with a gold score of less than 0.5, indicating that these data have a side effect on overall task completion. Besides, there are 7,524 instructions with a gold score greater than 0.8, suggesting that the model improves the task completion rate through one-shot learning from these data, which can be considered high-quality instruction data.

heading: B Experiment on Other Instruction Sets
text: Based on the LLaMA-7B model, we conducted experiments on several other instruction datasets, further validating the effectiveness of our NUGGETS method.
B.1 Code Alpaca The Code Alpaca instruction dataset (Chaudhary, 2023) is designed to develop large language models capable of following instructions and generating code. Leveraging self-instruct (Wang et al., 2022a) technology, it has produced 20,000 examples of instruction data. We use HumanEval (Chen et al., 2021) as a benchmark to evaluate the model’s code generation capabilities. It is used to measure functional correctness for synthesizing programs from docstrings. It consists of 164 original programming problems, assessing language comprehension, algorithms, and simple mathematics, with some being comparable to simple software interview questions. We adopt the approach outlined by Chen et al. (2021) to calculate pass rates at k values of 1, 10, and 100 for each problem. Essentially, pass@1 predicts the probability of a model producing a correct solution on the first try, while pass@10 and pass@100 predict the probability of achieving a correct solution within 10 and 100 tries, respectively. We generate 200 completions at a temperature setting of 0.2 (Luo et al., 2023) to estimate pass@1, pass@10, and pass@100 rates. The experimental results are shown in the Figure 6. Out of 20,000 instructions, 4,715 instructions have a gold score greater than 0.85, achieving the best pass@1 and pass@10 results in the HumanEval benchmark, superior to the fine-tuning results of the full dataset. Additionally, this experiment also proves that the NUGGETS method can be applied to fine-tuning for specific tasks, demonstrating good transferability.
B.2 WizardLM The WizardLM instruction dataset (Xu et al., 2023a), which employs Evol-Instruct to iteratively refine an initial set of instructions into more complex ones, contains 70,000 instruction examples. The distribution of the golden scores and the performance of models fine-tuned on corresponding subsets of instructions are shown in Table 5. We can observe that the quality distribution of the WizardLM dataset is relatively balanced, with 65,190 instruction examples having a golden score greater than 0.8, accounting for 93% of
71.39
the total number of instructions. In the evaluation of the Alpaca-Eval benchmark, models fine-tuned on subsets with golden scores greater than 0.8 achieved a win rate of 59.81, outperforming models fine-tuned on the full dataset.
B.3 FLANv2
We sampled 50,000 examples from the FLANv2 (Chung et al., 2022) dataset to constitute the instruction tuning data for this experiment. Additionally, the Predefined task set was also derived from these 50,000 examples, using the K-Means algorithm to sample 100 examples. We evaluated the performance of the fine-tuned model using MMLU (Hendrycks et al., 2020) in a 5-shot setting. MMLU is a test designed to measure a text model’s multitask accuracy. The test encompasses 57 tasks, including elementary mathematics, US history, computer science, law, and more. The experimental results are shown in the
table. It can be observed that the model fine-tuned with examples having a golden score greater than 0.8 (totaling 17,037 examples) achieved the best results, followed by those fine-tuned with examples having a golden score greater than 0.6 (totaling 46,009 examples). Additionally, we noted that the model fine-tuned
with examples having a golden score less than 0.5 (totaling 1,361 examples) outperformed the model fine-tuned with examples having a golden score greater than 0.9 (totaling 321 examples).
We speculate that the reason for this outcome is that the MMLU evaluation itself is a knowledgeintensive assessment, which thus imposes higher demands on the quantity of fine-tuning examples. In the future, some optimizations in the selection of the Predefined Task Set may potentially improve this situation.

heading: C Case Study
text: To evaluate NUGGETS qualitatively, we also select some instruction examples from the Alpaca dataset for case studies, as shown in Figure 7. We observe that noisy instructions without outputs display extremely low golden score values. Such examples consistently score very low throughout the instruction dataset. Additionally, instructions with hallucinatory outputs and some that are a bit confusing also typically exhibit lower golden scores. In contrast, instructions with higher scores often possess fluent language and meticulous logic, and are oriented towards assisting humans in problem-solving. Therefore, we believe that the high golden scores can represent high-quality instruction tuning data.

heading: Future Work
text: In the future, some optimizations in the selection of the Predefined Task Set may potentially improve this situation