SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection

Authors: liangxin  liu
xuebo  liu
derek f wong
dongfang  li
ziyi  wang
baotian  hu
min  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:04:41

Abstract: Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards human-centric interactions. Recent advancements have shown that the careful selection of a small, high-quality subset of IT data can significantly enhance the performance of LLMs.  Despite this, common approaches often rely on additional models or data, which increases costs and limits widespread adoption. In this work, we propose a novel approach, termed $\textit{SelectIT}$, that capitalizes on the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources.  Furthermore, we introduce a curated IT dataset, the $\textit{Selective Alpaca}$, created by applying SelectIT to the Alpaca-GPT4 dataset. Empirical results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks.  Our findings suggest that longer and more computationally intensive IT data may serve as superior sources of IT, offering valuable insights for future research in this area.  Data, code, and scripts are freely available at https://github.com/Blue-Raincoat/SelectIT.

Main paper: 
 related work: 
 Instruction Tuning Dataset Recent empirical research highlights the substantial benefits of finetuning LLMs on specialized datasets containing instructions and responses, significantly enhancing their generalization capabilities and responsiveness to new questions (Chung et al., 2022; Longpre et al., 2023; Honovich et al., 2022; Sun et al., 2023) . FLAN (Wei et al., 2022a) reformulates traditional natural language processing tasks as instructions formats, thereby improving model performance. Alpaca (Taori et al., 2023; Peng et al., 2023a) exemplifies the effectiveness of merging a select set of manual instruction seeds with advanced LLMs, like text-davinci-003 or GPT-4, to compile a comprehensive dataset. Similarly, Vicuna (Chiang et al., 2023) leverages 70,000 conversations from ChatGPT interactions, benefiting from the diverse data types and structures within these dialogues. WizardLM (Xu et al., 2023) introduces a novel approach by using LLMs to automatically generate open-domain instructions of varying complexities, achieving controlled instructional difficulty variation. However, LIMA (Zhou et al., 2023) demonstrates that only 1K high-quality IT data can match or exceed the performance of LLMs fine-tuned on larger IT datasets, presenting a promising direction for future research.The recognition of IT data quality's superiority over quantity in the context of IT is well-established, yet the efficient and precise identification of high-quality data continues to be a challenging frontier for research. One straightforward approach is utilizing the closed-source advanced LLMs for IT data evaluation and selection (Chen et al., 2024; Liu et al., 2023) . To circumvent the constraints associated with closed-source, existing research opt to fine-tune LLMs directly to select high-quality IT data (Li et al., 2023b; Kung et al., 2023) . Li et al. (2023c) ; Gururangan et al. (2020) ; Chen et al. (2023a) ; Cao et al. (2023) use pre-defined notions of useful data or other IT datasets to develop a data quality assessment framework. Li et al. (2023a) propose training a specialized model and utilizing two unique, condition-based losses on this for a comprehensive IT Score each assistant on a scale from 1 to 5, reflecting the accuracy of their adherence to the instructions and input, and present this score plainly. et al., 2023) or model gradients (Xia et al., 2024; Han et al., 2023) are also important features for selecting high-quality data in fine-tuned LLMs. However, the methods described above depend, to varying degrees, on supplementary datasets, the use of closed-source models, or open-source models that have been specially fine-tuned, which results in increased consumption of resources and potentially limits the broader impact. 
our selectit method: 
 Utilizing advanced LLMs for the sample evaluation is a widely adopted approach in the IT data selection (Chen et al., 2024; Li et al., 2023b; Liu et al., 2023) . Given an IT dataset D containing a sample S = (input X, response Y ), a designated rating prompt RP , and the foundation LLMs M , the goal is to leverage both RP and S to prompt M to assign an evaluation Score to the sample S on a scale from 1 to K. A higher score typically signifies superior IT data Quality.EQUATIONWhile existing methods (Chen et al., 2024; Cao et al., 2023) are adept at identifying high-quality samples, they often over-rely on external resources. To address these challenges, we introduce SelectIT, a strategy that capitalizes on the internal uncertainty of LLMs to efficiently select highquality IT data. SelectIT incorporates three grains of sample evaluation modules: token, sentence, and model-level self-reflections, which effectively improve the reliability of IT data selection. The comprehensive framework of SelectIT is depicted in Figure 2 .Numerous studies have demonstrated that foundation models exhibit robust capabilities for next-token prediction during their pre-training phase (Touvron et al., 2023b,a) . Yet, this predictive strength is frequently underutilized in evaluating IT data quality. In SelectIT, we adopt a similar idea to evaluate IT data. Specifically, we calculate the next-token probability (from 1 to K) based on the rating prompt RP and sample S. The score token with the highest probability is then considered as the sample's quality.S base = arg max k∈{1,...,K}EQUATIONwhere P k and P ′ k mean the probability and normalized probability of token k. The probability distribution among score tokens reflects the internal uncertainty of LLMs on sample evaluation. The higher P ′ S base , the more confidence of LLMs, which is not well exploited in Equation 2. To capture this subtle difference, we introduce the token-level self-reflection (Token-R), which uses the distribution between tokens that reflect the internal uncertainty of LLMs, to enhance the credibility of quality assessment. Specifically, we assess the average disparity between the predicted S base token and the other, where the greater the disparity, the more the confidence of LLMs. This disparity is then utilized to refine the original S base , resulting in a token-level score S token .S token = S base × 1 K -1 K i=1 |P ′ i -P ′ S base | U ncertainty(3)Different prompts can significantly affect outputs of LLMs (Kung et al., 2023; Peng et al., 2023b) , introducing uncertainty into IT data evaluation at the sentence level. To make better use of this uncertainty to bolster the reliability of our method, we implement sentence-level self-reflection (Sentence-R). Building upon Token-R, we devise K semantically similar rating prompts{RP 0 , RP 1 , . . . , RP K } to obtain a series of quality scores {S token 0 , S token 1 , . . . , S token K } based on a given sample S. We calculate the average of these scores to represent the overall quality of sample S, because of the importance of incorporating assessments from diverse prompts. Additionally, we use the standard deviation to quantify the LLMs' uncertainty to rating prompt; a higher standard deviation suggests greater sensitivity to prompt variation, while a lower standard deviation indicates more consistent and confident quality ratings by LLMs (Zhou et al., 2020) . By integrating a holistic sample evaluation with the quantification of model uncertainty, we derive the sentence-level score S sent , offering a more nuanced and reliable measure of IT data quality.EQUATIONwhere Avg{•} and Std{•} respectively denote the mean and standard deviation of S token i , K means the number of rating prompts RP . Moreover, we use the uncertainty factor α to control for the impact of the uncertainty of LLMs on overall scores.A sample affirmed by multiple foundation models can truly be deemed as high-quality. Different foundation models have different quality assessments of the sample, which introduce model-level uncertainty. To maximize the utilization of this uncertainty, we introduce model-level self-reflection (Model-R). This strategy leverages the capabilities of existing open-source models without the need for additional resources or the complexities associated with fine-tuning. However, the challenge lies in the diverse capabilities of various LLMs and determining how to reasonably combine their sample evaluation based on their performance. It is widely acknowledged that the capabilities of LLMs tend to increase with their parameter count (Hendrycks et al., 2021) . Thus, we suggest using the parameter count of LLMs as an initial metric for assessing their capabilities to properly weight sample quality scores. Given N foundation models with parameter counts {θ 1 , θ 2 , . . . , θ N } and their respective sentence-level scores for a sample S being {S sent 0 , S sent 1 , . . . , S sent N }, we formulate the model-level score S model to reflect a comprehensive evaluation of sample quality.Quality ∝ S model = N i=1 θ i N j=1 θ j × S sent i (5)where N means the number of the foundation models. By obtaining LLM parameters without resource expenditure, Model-R effectively allows us to employ more powerful foundation models, which is advantageous for selecting higher-quality data. Finally, we use S model as the final evaluation of sample S in SelectIT. The higher S model , the better sample quality. We sort the samples in descending order based on their S model and then select the top-ranked samples as high-quality data.We apply SelectIT to the widely-used Alpaca-GPT4 (Peng et al., 2023a) . Specifically, we use the most popular LLaMA-2 (7B, 13B, 70B) as our foundation models and set the hyper-parameters α = 0.2 and K = 5, which decides the range of LLMs rating in Token-R and the number of rating prompts in Sentence-R. We finally select the top 20%, a total of 10.4K pairs as the high-quality data and obtain a curated IT dataset called Selective Alpaca.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
setups: 
 Benchmark To gain a more comprehensive understanding of the capabilities of LLMs, we evaluate our approach in diverse downstream tasks (Wang et al., 2024; Ivison et al., 2023) . Factual knowledge: We use the Massive Multitask Language Understanding dataset (MMLU (Hendrycks et al., 2021) ) to assess the factual knowledge of LLMs and report 5-shot results. Reasoning: We evaluate the reasoning abilities of LLMs using two widely utilized datasets: the Grade School Math dataset (GSM (Cobbe et al., 2021) ) and Big-Bench-Hard (BBH (Suzgun et al., 2022) ) with the CoT setting (Wei et al., 2022b) . Multilinguality: we assess this ability by TyDiQA, a multilingual question-answering benchmark that encompasses 11 diverse languages, with the gold-passage setup. Coding: We evaluate this ability using the HumanEval dataset (Chen et al., 2021) and report pass@10 results with a temperature of 0.8. Open-ended generation: We utilize AlpacaEval (Dubois et al., 2023) , which employs GPT-4 to effectively assess model outputs. This can evaluate whether the text produced by LLMs aligns with humans.Implementation Details We use LLaMA-2 as our testbed. We fine-tune it for 3 epochs, with a batch size of 128. We use Adam with β 1 = 0.9, β 2 = 0.999, and the cosine learning rate scheduler starts from 2e-5, and decays to 0. we opted for a 4096 input length because it can show the best performance of LLMs. We employ the beam = 4 for decoding. We set the temperature parameter to 0.8 and the top-p sampling parameter to 0.9 to improve the originality of the output text while ensuring the accuracy and relevance of the content.Baselines We compare with the following baselines:• Alpaca-GPT4 (Peng et al., 2023a ) is a widely-used IT dataset that implements a self-instruct method to autonomously generate instructions by the advanced GPT4. • LIMA (Zhou et al., 2023) primarily consists of 1000 manually crafted high-quality instructional data, which can better stimulate the alignment capability of LLMs. • AlpaGasus (Chen et al., 2024) involves utilizing the robust ChatGPT to score and select data from the original Alpaca-GPT4 dataset. • Q2Q (Li et al., 2023a) operates by training a precursor model, determining the quality of the IT data based on the two different loss values within this model. • Instruction Mining (Cao et al., 2023) All the scores are averages of three independent runs with different random seeds.We focus on the discussion of LLaMA-2-13B because both 7B and 13B models exhibit similar trends in Table 1 . System (10) shows the vanilla IT on LLMs with the original Alpaca. By using the data selection strategies, the ability of LLMs has a moderate enhancement in Systems ( 12) to ( 14). Additionally, we can use S base as the input for Equations 4 and 5 to construct individual methods of Sentence-R and Model-R. Systems (15) to ( 17) illustrate that applying each submodule of SelectIT incrementally enhances LLMs' performance, rivaling contemporary advanced methods.Most remarkably, SelectIT can better boost LLaMA-2's performance compared to vanilla IT in the System (18). Compared to other IT data selection strategies, this enhancement is particularly evident in the computational and reasoning tasks on the BBH and GSM benchmarks. This may be attributed to the characteristics of selected data by SelectIT, and we will analyze this phenomenon in a later section. These gains in reasoning ability also positively impact the coding proficiency of LLMs. The improvement of LLMs on the TydiQA dataset is also obvious enough, which shows that SelectIT can effectively eliminate similar samples and retain sufficient diversity in multilingual aspects. 
analysis: 
 This part aims to answer the research questions through the following experiments: How to select high-quality data in SelectIT? ( §5.1) Is SelectIT adaptable to various models and domains? ( §5.2) How about the efficiency of SelectIT? ( §5.3) What are the advantages of Selective Alpaca?( §5.4) Effect of IT Data Quantity While SelectIT already excels at assessing and ranking samples effectively, selecting an appropriate number of samples in a redundant dataset remains a crucial aspect of our method. We divide the Alpaca dataset into multiple subsets ranging from 10% to 100% based on SelectIT's evaluation and evaluate the overall ability of LLMs on the open-instruct benchmark. As illustrated in Figure 3 , compared to using the full Alpaca dataset, we observe that LLMs achieve optimal performance using the top 20% to 40% data. Hence, considering the tradeoff of training resources, training time, and model performance, we opt for 20% for implementing the SelectIT on the Alpaca dataset. Effect of Multiple Rating Prompts K is a critical parameter for our method, impacting not only the range of scores assigned by the LLMs but also the number of rating prompts. We set K = 3, 5, 7, 9 and apply SelectIT for sample selection within the Alpaca to get different subset datasets. Table 2 indicates that variations in the value of K have a minor impact on the overall performance of the LLMs. This is attributed to our multi-granularity self-reflection mechanism, which effectively enhances the accuracy and stability of sample selection. Although the model achieves competitive performance at K values of 5 and 7, to minimize resource consumption, we set K = 5 as the default value in SelectIT. that, when facing the same amount of data, SelectIT can still demonstrate better performances, which further illustrates its effectiveness.Various Foundation Models Although Selective Alpaca achieved impressive improvements in LLaMA-2, applying it to other foundation models remains a challenging task. To address this, we apply Selective Alpaca on the Mistral-7B and LLaMA-3-8B LLMs and present our results on the open-instruct benchmark alignment with the above test configuration. As depicted in Table 6 , although Selective Alpaca is selected by the LLaMA-2 models, it is also applicable to the Mistral-7B, LLaMA-3-8B and improves their capabilities across various tasks, especially on MMLU, BBH, and GSM benchmarks. This experiment fully demonstrates the flexibility of SelectIT which does not rely on a specific foundation model for data selection and the universality of Selective Alpaca which can effectively improve the capabilities of different series or scale LLMs.Various Instruction Tuning Datasets We further validate the robustness of SelectIT by deploying it on two additional, widely-utilized datasets: WizardLM (Xu et al., 2023) and Orca-GPT4 (Subhabrata & Arindam, 2023) . WizardLM introduces an innovative method of using LLMs to auto-generate open-domain instructions of varying complexities. This allows for a controlled variation in instructional difficulty and the dataset comprises 143K samples. Orca-GPT4 on the other hand, leverages rich signals from GPT-4 that include explanation traces, step-by-step thought processes, and other multifaceted instructions, all under the guidance of teacher assistance from ChatGPT. Additionally, we maintain consistent hyperparameters, such as α and K, choosing LLaMA-2-7B as our base model. We limit the fine-tuning of these datasets to one epoch. As shown in Figure 7 , SelectIT consistently enhances the performance of the model on both the WizardLM and Orca-GPT4 datasets. Notably, this augmentative effect is especially pronounced in the computational and reasoning tasks within the BBH and GSM benchmarks. In evaluating three separate IT datasets, specifically Alpaca-GPT4, WizardLM, and the more extensive Orca-GPT4, our extensive experimental conclusions validate the broad utility and durability of SelectIT.Various Domain-specific Tasks Machine translation (MT) is a representative domain-specific task of LLMs. Previous works have already demonstrated significant improvements with LLMs, but they usually use redundant translation IT datasets. This part tests the robustness of SelectIT on the IT dataset of MT. We select the powerful MT LLM ALMA (Xu et al., 2024) examples were selected using SelectIT. We utilize both BLEU (Post, 2018; Ott et al., 2018) and COMET (Rei et al., 2022) based on the wmt22-comet-da model for evaluation. We report results for the two language pairs in four directions, using ALL to represent their average. Table 8 shows that SelectIT consistently improves ALMA's translation performance. These results indicate that SelectIT is a versatile and scalable method, effective not only for IT data selection but also for domain-specific tasks like MT. For more detailed analysis and results, please see Appendix A.1. SelectIT is a faster and more cost-effective method for IT data selection. We compared different selection methods on the Alpaca-GPT4 dataset. For ChatGPT (AlpaGasus) or GPT-4, we randomly select 500 instruction data from Alpaca-GPT4, analyze various metrics, and estimate the resource consumption for selecting the entire dataset. Using SelectIT, we employ 4 A800 80G GPUs to select high-quality IT data, calculating the total cost based on Google Cloud's rate of $1.15/h per single GPU. As shown in Table 9 , SelectIT is significantly faster and uses the least resources. This efficiency is due to computing only the probability of the next token for input sentences, bypassing the full sentence generation and decoding process, resulting in lower resource consumption. Additionally, using our own GPU at a low cost enhances transparency, allowing us to preserve all intermediate outputs and results for thorough analysis in data selection. (Zhao et al., 2024) . As shown in Table 10 , the random-based strategies show certain performance degradation and the random selection in the unselected dataset is even worse, which reflects the effectiveness of our method from the side. Selection based on sample length is a simple approach to defining high-quality data, but it does not take into account the content of IT data, resulting in the limited performance of LLMs. SelectIT can significantly improve the abilities of LLMs.Data Representation Analysis This part explores the relationship between Selective Alpaca and the original datasets from a representation perspective. Following Gao et al. (2024) , we use the outputs of the last layer corresponding to the last token in the input sequence as sample representations. We then apply T-SNE (Hinton & Roweis, 2002) for dimensionality reduction, mapping high-dimensional embeddings onto a 2D space. Figure 4 shows the intermediate representations generated by the full and Selective Alpaca datasets. Randomly selected data struggle to distinguish abnormal data far from the center, making it hard to define high-quality IT data. In contrast, Selective Alpaca data are mostly concentrated around the center, indicating that our dataset predominantly contains high-quality data near the center and effectively discards abnormal data, supporting the conclusion of Table 10 .We analyze the Selective Alpaca from the following two perspectives, to explore why our dataset is better than the original dataset and its variants. Firstly, as shown in Figure 5 , the length of instructions from the Selective Alpaca is significantly longer than those in the Alpaca dataset and AlpaGasus which is selected by ChatGPT. This implies that, with the same amount of data, our dataset contains more information, aligned with the results in Table 10 . Secondly, by using ChatGPT to examine IT data types, we find a substantial increase in the proportion of computational problems in Selective Alpaca. This indicates that Selective Alpaca tends to select high-quality mathematical data, providing a solid explanation for the observed improvement in the reasoning abilities of LLMs as demonstrated in Table 1 . Appendix A.3 shows the case study of comparing the Selective Alpaca with AlpaGasus. Insights of High-Quality Data in SelectIT Furthermore, we analyze the proportion of calculation and sample average length in Alpaca-GPT4 with different proportions after sorting by SelectIT to explore its intrinsic characteristics and the definition of high-quality data. As shown in Figure 6 , with the proportion of Alpaca-GPT4 data continuing to increase, the proportion of calculation and sample average length gradually decreases. This phenomenon clearly indicates that SelectIT can reasonably rank samples based on their characteristics. When the data size is more than 50%, the proportion of calculation IT data sharply declines, falling below 6%, causing a noticeable decrease in the model's overall capability, as depicted in Figure 3 . This analysis shows that more computationally intensive IT data may be a new perspective on the characteristics of optimal IT data, which not only effectively improves the LLMs' reasoning ability, but also further drives the improvement of other abilities. 
conclusion: 
 This paper introduces a novel data selection strategy, SelectIT, for LLM instruction tuning, which uses LLM uncertainty to efficiently identify high-quality IT data without requiring additional resources.SelectIT includes three types of self-reflection: token, sentence, and model, which can individually and jointly improve the performance of IT data selection. By applying SelectIT to the Alpaca-GPT4 dataset, we introduce a compact and strong IT dataset, called Selective Alpaca. Different models and domain tasks demonstrate the effectiveness of SelectIT. Our analysis reveals that SelectIT effectively excludes abnormal data and tends to select longer and calculational data. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In Section 5.3, we provide the type of computing workers, memory, and time of execution to help other researchers reproduce the Selective Alpaca.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: we have read the guidelines and ensured that our paper conforms to them.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In Section 5.1, Selective can use fewer computing resources to select highquality data, which has a positive impact on society. We also have a section to discuss the broader impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The model and datasets we used are all open-sourced, and we strictly follow their terms once the terms are carried out.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited the creators in the main part of the paper and the supplement material.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should citep the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: In the abstract section, we provide the GitHub link to open access to our code and data.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: SelectIT does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: SelectIT does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Large language models (LLMs) have attracted much attention due to their impressive capabilities in following instructions and solving intricate problems (Touvron et al., 2023b,a; Achiam et al., 2023; Penedo et al., 2023) . A crucial aspect of enhancing LLMs' performance is instruction tuning (IT), which involves the supervised adjustment of LLMs using pairs of instructional data, essential for refining the models' ability to accurately respond to human instructions. Recent groundbreaking research, such as the LIMA (Zhou et al., 2023) , highlights the critical importance of instructional data quality over quantity. Contrary to the approach of merely increasing the dataset size, a carefully selected, smaller dataset of higher quality can significantly improve LLMs' performance. 
 
 Appendix: This paper could be further strengthened as follows:• Instruction Data Quantity: Our findings suggest that prioritizing the top 20% of highquality data optimizes results for Alpaca. Future studies might explore adjusting this threshold based on the data quality in different datasets to enhance performance.• Models at Different Scales: Our analysis is currently limited to models smaller than 30B parameters due to computational constraints. Investigating the efficacy of Selective Alpaca on larger-scale LLMs, could provide valuable insights into the method's scalability.• Expansion to Additional Instruction Datasets: Although SelectIT has been applied to the Alpaca dataset due to its widespread adoption, extending this methodology to incorporate other IT datasets could offer substantial advantages to the broader LLM research community.Our work follows the NeurIPS Ethics Policy. Our findings are based on publicly available datasets for reproducibility purposes. LLMs can contain potential racial and gender bias. Therefore, if someone finds our work interesting and would like to use it in a specific environment, we strongly suggest the user check the potential bias before usage. In addition, it is hard to control the generation of LLMs.We should be aware of the potential problems caused by hallucinations.A.1 Applying SelectIT on Machine Translation LLMs Machine Translation (MT) is a important task for LLMs, demonstrating their domain-specific capabilities. Prior research, including TIM (Zeng et al., 2023) , SWIE (Chen et al., 2023b) , BigTranslate (Yang et al., 2023) , and Bayling (Zhang et al., 2023) , has shown significant improvements in LLMs, often relying on extensive translation training datasets. In this section, we examine the impact of training data quality on MT performance, employing the robust MT LLM, ALMA, as our foundational model (Xu et al., 2024) .For training data, we select representative language pairs: German⇔English and Chinese⇔English, sourced from WMT'17 to WMT'20 human-authored test datasets, supplemented with development and test sets from Flores-200, totaling 30K training instances. We use the corresponding language pair's test data from WMT'22 as evaluation datasets. Subsequently, 6K high-quality instances are selected for LORA fine-tuning via SelectIT.We report both the widely used BLEU score (Post, 2018; Ott et al., 2018) and the COMET score (Rei et al., 2022) based on the wmt22-comet-da model, which shows higher correlation with human judgments for evaluating the LLMs' translation abilities. Table 11 : Overall results on machine translation LLMs. " † " the improvement is significant by contrast to the ALMA model (p < 0.05).Based on the preceding analysis, Sentence-R is integral to the functionality of SelectIT. As illustrated in Equation 4, the Token-level Rating forms the foundation for the Sentence-level Rating. The Model-level Rating is derived through multiple iterations of the Sentence-level Rating across different foundational LLMs. Therefore, a detailed explanation of Sentence-R is sufficient to demonstrate the operational mechanism of SelectIT. As depicted in Figure 7 , we utilize five distinct rating prompts along with a single input to formulate the final input for Sentence-R. Initially, each rating prompt produces a score of S token . We then compute the mean and standard deviation of these S token values to obtain the final S sent , as outlined in Equation 4.As demonstrated in Figure 8 , we illustrate the selection tendencies of SelectIT in contrast to Alpa-Gasus, which leverages advanced ChatGPT for data selection. In samples 1 to 4, SelectIT shows a preference for instruction-tuning data containing intricate mathematical problems that contribute to improving the reasoning skills of the LLMs. On the contrary, AlpacaGasus frequently chooses IT data in samples 5 to 7 that primarily offer solutions to queries or lack coherent reasoning, which might limit its effectiveness. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper introduces a novel approach for instruction tuning of LLMs that leverages the model's intrinsic uncertainty to select high-quality training data without relying on external resources. The reviewers generally agree that the approach is innovative and potentially impactful. The strong empirical results across various benchmarks also support its effectiveness. The reviewers are largely positive on this submission. The authors also have addressed many concerns in their rebuttal, and provided additional experiments on diverse datasets and clarified the method's adaptability to different numbers of base models.

Reviews: review: paper_summary: paper_summary: The paper introduces a data selection approach to instruction tuning (IT) for large language models (LLMs) by leveraging the intrinsic uncertainties within the models themselves. This method, termed SelectIT, aims to enhance IT data selection without the need for additional external resources, making it more cost-effective and accessible.

SelectIT utilizes three levels of uncertainty within LLMs—token-level, sentence-level, and model-level—to effectively rate and select high-quality IT data. The paper demonstrates the effectiveness of SelectIT through empirical results, showing improvements in model abilities on the open-instruct benchmark and other domain-specific tasks. Additionally, SelectIT is highlighted as a faster and more resource-efficient method compared to traditional approaches, making it a more practical solution for widespread adoption.

---------------------------------------------------------------------------------------------------------------------------------------------------------------
Thank you for your reply and I have updated my score accordingly.
strength_weakness: strength_weakness: 1. **Reliance on Tuning K**: The method depends on tuning the parameter \( K \), which can significantly affect model performance. This dependency introduces additional complexity and may require extensive experimentation to optimize.

2. **Use of Multiple Base Models**: Although the paper claims not to use external resources, it actually relies on multiple different base models to obtain the model-level score. This reliance contradicts the claim of being purely self-reflective and can be confusing, as the title suggests the method is entirely self-contained.

3. **Sentence-Level Quality Evaluation**: The sentence-level quality evaluation requires multiple rating prompts. It is unclear whether other methods, such as AlpaGasus, would achieve similar performance if they also used multiple rating prompts. This comparison is missing and could be critical for understanding the true advantage of the proposed method.

4. **Single Dataset Evaluation**: All evaluations are conducted on a single training dataset, making it questionable whether this method can be generalized to other datasets, such as WizardLM, or multi-turn data, like the ShareGPT dataset. Testing on a more diverse set of datasets would strengthen the paper's claims of robustness and general applicability.
questions: questions: (1) How the α = 0.2 in the experiments is selected? 
(2) Will optimal K be training dataset dependent?
limitations: limitations: See more details in Weaknesses section.
paper_summary: paper_summary: This paper presents SelectIT, a new way to improve how large language models (LLMs) follow instructions using their own uncertainty to pick better training data. This method is cost-effective as it doesn't require extra tools or data. The authors developed a new dataset called Selective Alpaca using SelectIT, which significantly improved LLMs' performance. The study suggests that using longer and more detailed data could be more beneficial for training models.
strength_weakness: strength_weakness: The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources.
questions: questions: "The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources. " Clarification?

Could you consider applying the SelectIT method to larger datasets, such as the Orca dataset for ChatGPT with 5 million samples and the GPT-4 dataset with 1 million samples? This would help to evaluate whether your method scales effectively with increased data volume.
limitations: limitations: the paper lacks a separate section on limitation of the approach.
paper_summary: paper_summary: In this paper, authors leverage LLMs to score the training data in terms of the token and sentence and model. They train various models on their select models to show the effectiveness on various benchmarks. Their main contribution is to apply foundations to judge the quality of training data from different granularity without any external resources.
strength_weakness: strength_weakness: 1. From the table 3, it seems sentence-R works the best so what if we trained model on the data just from the sentence-R?  BTW, what is the ID 6, 7, 8 means in that Table?
2. My main question is that do we really need all three types selection models, can one or two of them already achieve great performance?
3. What is the overhead of this data selection model? Like how long does it take to finish the process on 10K dataset.
questions: questions: Please refer to the  above.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces SelectIT, a novel instruction tuning (IT) method that leverages the intrinsic uncertainty in large language models (LLMs) to select high-quality IT data without requiring additional resources. The authors present a curated IT dataset, Selective Alpaca, derived from the Alpaca-GPT4 dataset using SelectIT. The empirical results demonstrate significant improvements in model performance across various tasks. The paper highlights the robustness and efficiency of SelectIT in enhancing the capabilities of LLMs.
strength_weakness: strength_weakness: - Figure 2 is not clear to me. I suggest giving a brief description of each selection stage in the caption for easy understanding.
- It seems like the foundational LLM's capacity becomes the key for this data selection. In this case, the selection method may not be as effective as other external resource methods.
questions: questions: - How does SelectIT perform when applied to instruction tuning datasets other than Alpaca-GPT4? 
- Can the authors provide more detailed examples or case studies illustrating the differences between data selected by SelectIT and other methods? And how these different influence the final model performance?
- How does the performance of SelectIT change with different values of the uncertainty factor (α) ?
limitations: limitations: Yes

Comments: title: Response to follow-up questions.
Response to follow-up questions.
Request to review the rebuttal [Author-Reviewer discussion phase ending soon]
Request to review the rebuttal [Author-Reviewer discussion phase ending soon]
Request to review the rebuttal [Author-Reviewer discussion phase ending soon]
Look forward to follow-up discussion.
Look forward to follow-up discussion.
Look forward to follow-up discussion.
comment: Thank you for your response. 

We will give a clearer explanation of SelectIT's preference for longer instruction data.

As shown in the "Introduction" section and "Insights of Selective Data Curation" section, the core feature of Selective Alpaca is the increase in the proportion of computational data, which is different from other methods. Computational data is often longer than other types of data because it requires more reasoning steps, which further leads to the longer overall length of Selective Alpca. So, "Length" may still be the important and easy-to-get feature for evaluating the quality of sample data [1-4].

Secondly, SelectIT does not simply prefer longer data. As shown in Table 8 of our paper (also the table below), we compared the SelectIT with the previous work [1], which selects samples only by length. The results show that SelectIT can achieve better results with the same amount of data.

|                        | LLaMA-2-7B | LLaMA-2-13B | ALMA-7B | ALMA-13B |
|:----------------------:|:----------:|:-----------:|---------|----------|
|      Alpace-GPT4       |    34.1    |    44.2     | 29.7    | 31.5     |
|  Alpace-GPT4 + Length  |    35.5    |    47.1     | 30.1    | 31.8     | 
| Alpace-GPT4 + SelectIT |    **36.2**    |    47.1     | **30.5**    | **32.2**     |

Finally, we further compared the output lengths of models trained on Selective Alpaca and Alpace-GPT4 on the Alpaca-eval benchmark. As shown in the following table, with the output lengths being similar, Selective Alpaca achieves a 1.5% improvement, which effectively eliminates the bias in sample length and explanation of the doubts about the preference of LLMs for samples with longer length.

|                  | Length | Win Rate |
|:----------------:|:------:|:----:|
|   Alpace-GPT4    |  807   | 34.2 |
| Selective Alpaca |  815   | 35.7 | 

Thank you again for taking the time to read our response! We are more than happy to continue the discussion if you are still confused about the paper. 

[1] Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning. arXiv, 2024.  
[2] Zero-Shot Generalization during Instruction Tuning: Insights from Similarity and Granularity. arXiv, 2024.  
[3] LIMA: Less Is More for Alignment. NeurIPS, 2023.  
[4] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024.
Thank you for your response. 

For the Model-R of SelectIT, we used models to jointly determine high-quality data and reasonably weigh the sample quality according to the model parameters, which is similar to ensemble learning. As shown in "Insights of High-Quality Data in SelectIT" section and Figure 5, SelectIT which used multiple models is more conducive to screening data with biased calculation types. So, SelectIT can improve model reasoning ability and further improve the encoding ability of LLM, which ultimately leads to an overall improvement in comprehensive ability.

Secondly, MMLU consists of a set of questions about 57 subjects and its multiple-choice format makes it suitable for probing models’ knowledge [1-2]. So it is often used to evaluate the foundation model competence, as LLMs mainly learn world knowledge in the pre-training stage, not the instruction tuning stage (only containing a few instances). So there is a slight improvement on MMLU which is relatively reasonable in SelectIT, and also consistent with the results of AlpaGasus [3].

Thank you again for taking the time to read our response! We are more than happy to continue the discussion if you are still confused about the paper. 

[1] Measuring Massive Multitask Language Understanding. ICLR, 2021.  
[2] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. NeurIPS, 2023.  
[3] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024.
Thanks for the response and I am thinking maybe we still need to understand why combination of models can yield better results on some benchmarks, for example, we do not see meaningful improvement on MMLU.  I will keep my score. thanks!
Thanks for your response. Regarding the 'Length' of the selected data, I cannot be convinced that one of the primary contributions is that a longer instruction indicates better quality. LLMs have a verbosity bias, preferring longer output even when it's not necessarily clearer [1]. Therefore, I will maintain my score.

[1] ZHng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Dear reviewer 4rCk

Could you please take a look at the responses of the authors and let us know your thoughts on them? Are you satisfied with the responses and do you have some updates on your comments? Please also take a look at other reviews and share with us your thoughts on whether the paper is ready for publication. We need your response so that we can make an informed decision on this paper.

AC
Dear reviewer CKC2

Could you please take a look at the responses of the authors and let us know your thoughts on them? Are you satisfied with the responses and do you have some updates on your comments? Please also take a look at other reviews and share with us your thoughts on whether the paper is ready for publication. We need your response so that we can make an informed decision on this paper.

AC
Dear reviewer KKrK

Could you please take a look at the responses of the authors and let us know your thoughts on them? Are you satisfied with the responses and do you have some updates on your comments? Please also take a look at other reviews and share with us your thoughts on whether the paper is ready for publication. We need your response so that we can make an informed decision on this paper.

AC
Dear Reviewer 4rCk,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,  
The Authors
Dear Reviewer CKC2,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,  
The Authors
Dear Reviewer KKrK,

Thank you for your time reviewing the paper. Your constructive feedback will help improve the quality of our paper.

We have also addressed all your concerns in our rebuttal point-by-point. As we are towards the end of the author-reviewer discussion period, we request you to please go through our rebuttal, and we would be more than happy to address any more concerns you have.

We thank you again for your time!

Best,  
The Authors
Dear Reviewer 4rCk,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results will be a nice plus to improve the clarity and useability of our method.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,  
The Authors
Dear Reviewer CKC2,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results provide more evidence for our method justification.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,  
The Authors
Dear Reviewer KKrK,

We greatly appreciate your valuable feedback and suggestions! To address your concerns, we have run extensive experiments on various datasets and made additional insightful observations. The results further confirm the generality of our proposed method.

As the discussion period is coming to an end, we kindly request that you review these results. If our results and explanations help address your concerns, we would be grateful if you could acknowledge our rebuttal and consider adjusting your score accordingly.

Best wishes,  
The Authors
Thank you for addressing most of my concerns. I have increased the OA score.