SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection
Authors: Liangxin Liu
Xuebo Liu
Derek F. Wong
Dongfang Li
Ziyi Wang
Baotian Hu
Min Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Data Selection
Large Language Models
Primary area: natural_language_processing
TL;DR: A novel strategy of using internal uncertainty of LLMs to select instruction tuning data.

Abstract: Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards human-centric interactions. Recent advancements have shown that the careful selection of a small, high-quality subset of IT data can significantly enhance the performance of LLMs.  Despite this, common approaches often rely on additional models or data, which increases costs and limits widespread adoption. In this work, we propose a novel approach, termed $\textit{SelectIT}$, that capitalizes on the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources.  Furthermore, we introduce a curated IT dataset, the $\textit{Selective Alpaca}$, created by applying SelectIT to the Alpaca-GPT4 dataset. Empirical results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks.  Our findings suggest that longer and more computationally intensive IT data may serve as superior sources of IT, offering valuable insights for future research in this area.  Data, code, and scripts are freely available at https://github.com/Blue-Raincoat/SelectIT.

SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectin
Liangxin Liu' Xuebo Liu Derek F. Wong?
Dongfang Li'
Ziyi Wang' Baotian Hu² Min Zhang'
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China 2NLP2CT Lab, Department of Computer and Information Science, University of Macau 11 1iaagxinn678gmaal com, {liuxuebo, hubaotian, zhangmin20211@hhi edu.t cn derekfw@um edu mo, {crazyofapple , zlyivang676)@gmail
Abstract
Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards numan-centric interactions. Recent advancements have shown that the careful selection of small, high-quality subset of IT data can significantly enhance the performance of L.. Despite this, common approaches often rely additional models or data, which increases costs and limits widespread adoption. this work, propose a novel approach, termed Select/T, that capitalizes the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources. Furthermore, we introduce a curated GPT4 dataset. Empirical
dataset, the Selective
Alpaca, created applying SelectIT to the Alpaca- Alpaca-GPT4 results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks. Our findings suggest that longer and more computationally intensive IT data may serve superior sources of IT, offering valuable insights for future research in this area. Data, code, and scripts are freely available hitttt:////// //ti gigghu.b... com Blue Raw.iaaattaanaeeattii.
Introduction
Large language models (LLMs) have attracted much attention due to their impressive capabilities in following instructions and solving intricate prob- lems (Touvron et al. 2023b Achiam et al.] 2023 Penedo et al. 2023). A crucial aspect enhanc- ing LLMs' performance is instruction tuning (IT), which involves the supervised adjustment LLMs using pairs of instructional data, essential for fining the models' ability to accurately respond to human instructions. Recent groundbreaking search, such as the LIMA (Zhou et al. 2023), high- lights the critical importance of instructional data quality over quantity. Contrary to the approach of merely increasing the dataset size, carefully selected, smaller dataset of higher quality can sig- nificantly improve LLMs' performance.
Models
Instruction Donascts
External
Method
Vanilia zi'
Foundation
Model
EMM
Selecoed Datasess
AlpaGasus
Mining
SalceIT
Figure I: Existing advanced data selection strategies rely heavily external models or data; however, SelectIT effectively overcomes this limitation.
Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Experimental results show that SelectIT outperforms existing high- data selection methods, improving LLM's performance on the open-instruct benchmark (Wang et al.) (022)). Further analysis reveals that SelectIT can effectively discard abnormal data and tends to select longer and more Despite the development of various high-quality data selection methods, they often depend on external
resources, limiting wider implementation. External Model: Chen al. (2024): Liu al. (2023) propose the employment of glosed-sourcs LLMS to evaluate or rank IT data. To circumvent the
closed-source limitations, Li al. (2023a by: Kung et al. (2023) recommend fine-tuning open-source
LLMs, which requires more computational resources. External Data: [Co et al. (2023) split all mixed data into several bins and fully trained the models to evaluate different indicators of high-quality IT data. Despite these advancements, the challenge of precise and efficient high-quality data selection without external resources remains unresolved.
In this paper, we introduce SelectT, a novel approach designed to enhance IT data selection by fully leveraging the foundation model itself, eliminating the need for external resources. SelectIT employs different grain uncertainty of LLMs: token, sentence, and model, which can effectually improve the accuracy of IT data selection. We first use the foundation model itself to rate the IT data from to K based on the uncertainty of various tokens. Next, we use sentence-leve uncertainty to improve the rating process by exploiting the effect of different prompts on LLMss At a higher model level, we utilize the uncertainty between different LLMs, enabling a collaborative decision-makin process for IT data selection. By applying SelectIT to the original Alpaca, we curate a compact and superior IT dataset, termed Selective Alpaca.
computationally intensive IT data. The primary contributions of SelectlT are as follows:
We propose SelectIT, novel IT data selection method which exploits the uncertainty of LLMs without using additional resources.
We introduce a curated IT dataset, Selective Alpaca, by selecting the high-quality IT data from the Alpaca-GPT- dataset.
SelectIT can substantially improve the performance of LLMs across a variety of foundation models and domain-speciio tasks.
Our analysis suggests that longer and more computationally intensive IT data may be more effective, offering new perspective on the characteristics of optimal IT data.
2 Related Work
Instruction Tuning Dataset Recent empirical research highlights the substantial benefits of fine-
tuning LLMs on specialized datasets containing instructions and responses, significantly enhancing their generalization capabilities and responsiveness to new questions (Chung et al. 2022: Longpre
2023 Honovich et al., 2022 Sun et al. 2023). FLAN (Wei et al. 2022a) reformulates traditional natural language processing tasks as instructions formats, thereby improving model performance. Alpaca (Taori et al. 2023 Peng et al. 2023a) exemplifies the effectiveness of merging a select set of manual instruction seeds with advanced LLMs like text-davinci-003 or GPT-4, to compile a comprehensive dataset. Similarly, Vicuna (Chiang al. 2023) leverages 70,000 conversations from ChatGPT interactions, benefiting from the diverse data types and structures within these dialogues. WizardLM (Xu al., (023) introduces novel approach by using LLMs to automatically generate open-domain instructions of varying complexities, achieving controlled instructional difficulty variation. However, LIMA (Zhou al. (2023) demonstrates that only 1K high-quality IT data can match or exceed the performance of LLMs fine-tuned on larger IT datasets, presenting promising direction for future research.
Instruction Data Selection The recognition of IT data quality's superiority over quantity in the context of IT is well-established, yet the efficient and precise identification of high-quality data continues to be challenging frontier for research. One straightforward approach is utilizing the closed-source advanced LLMs for IT data evaluation and selection (Chen al.] 2024 Liu al.) 2023). To circumvent the constraints associated with closed-source, existing research opt to fine- tune LLMs directly to select high-quality IT data (Li et al. 2023b Kung et al. 2023). Li et al. (002cc); Gururangan et al. (2020); Chen et al. (2023a); Cao et al. (2023) use pre-defined notions of useful data or other datasets to develop a data quality assessment framework, [Li al. (2023a) propose training specialized model and utilizing two unique, condition- losses on this for comprehensive IT

---

Token-Level Self-Reflection
Rating Prompt RP
Sample.8
pritubilyy
0.50 (Arguax)
Score each assistant on scale from 5, reflecting the sccuracy of their adherence to the instructions and input, and present this score plainly.
Input: Give three tips for
Argmascll,, Pk
Response: Eat balanced...
0,10
0,10
Foundation
next
0,05
S
5 pv-
Eg.
Model
telcen
prediction
5
tokes
Sentence-level Relf-Reftcttion
Model-level Self-Reflection
Rating Prompt RP,
7B
13B
Score each assaslant on scale
Rating Prompt RP2
Evaluate the assistants by the
-2.38
Kating Prompt RP3]
Ep.
Assign score
g-etl 1,59
each assistant based how
uccurately they follow the input and output provided, ensuring the score is represented clearly.
1.59
2,13
1.89
1.56
S
1.90
Selecif
Sample S
Sample
Score
Selected
Score;
Sample
Token Reflection
9836
4.15
Score: 2.37
Sort
34616
4.15
Sample S,
Score, 4.01
Sample Sra
Instruction
Model Reflection
Score
0.51
Datnset
46171
0.49
Figure 2: Overall framework of SelectIT. In Token-level Self seff-Reflection, we employ the foundation model rate the IT data from to K. In Sentence-level Self-Reflection, we leverage the uncertainty of varied prompts on LLMs to enhance the rating process. In Model-level Self-Reflection we harness uncertainty among different L.LSs to facilitate a collaborative decision- makinn process in selecting IT data, Finally, different levels of self-reflection are reasonably combined into Selectl'T, which can effectively select high-quality IT data without relying on additional resources.
data selection. Wu et al. 2023) explore where data selection is informed by the similarity of samples within the embedding space of fine-tuned model. N-gram features (Xie al.] (2023) model gradients (Xia et al. 2024 Han et al. (023) are also important features for selecting high-quality data in fine-tuned LLMss. However, the methods described above depend, varying degrees, on supplementary datasets, the use of closed-source models, or open-source models that have been specially fine-tuned, which results in increased consumption of resources and potentially limits the broader impact.
3 Our SelectIT Method
Utilizing advanced LLMSs for the sample evaluation is widely adopted approach in the IT data selection (Chen et al., 2024, Li et al., 2023b) Liu et al.) 2023). Given an ITT dataset D containing a sample S = (input X, response Y), a designated rating prompt RP, and the foundation LLMs M, the goal is to leverage both RP and S to prompt M to assign an evaluation Score to the sample S on scale from K. higher score typically signifies superior IT data Quality.
Quality & Score E [1, K] = M (RP,S)
(I)
While existing methods (Chen al., 2024, Cao al.
2023) are adept at identifying high-quality samples, they often over-rely on external resources. To address these challenges, we introduce Select[T, a strategy that capitalizes on the internal uncertainty quality IT data. SelectIT incorporates three grains of sample evaluation modules: token, sentence, and model-level self-reflections, which effectively improve the reliability of IT data selection. The comprchensive framework of SelectI is depicted in Figure
LLMSs to efficiently select high-
3

---

3.1 Token-level Self-Reflection
Numerous studies have demonstrated that foundation models exhibit robust capabilities for next- token prediction during their pre-training phase (Tourron et al. 2023b Yet, this predictive strength is frequently underutilized in evaluating IT data quality, In SelectlT, we adopt similar idea to evaluate IT data. Specifically, we calculate the next-token probability (from to K) based on the rating prompt RP and sample S. The score token with the highest probability is then considered as the sample's quality.
Px Σ E
Sbose
arg max
PE
RE kE()...,K)
P3
where P& and Pa mean the probability and normalized probability of token A.
The probability distribution among score tokens reflects the internal uncertainty of LLMs sample evaluation. The higher Pi the more confidence of LLMs, which not well exploited in Equation To capture this subtle difference, we introduce the token-level self-reflection (Token-R), which uses the distribution between tokens that reflect the internal uncertainty of LLMs, to enhance the credibility of quality assessment. Specifically, we assess the average disparity between the predicted Slase token and the other, where the greater the disparity, the more the confidence of LLMs. This disparity is then utilized to refine the original Stase resulting in token-level score Stoken
-
Stoken
gbase
P
P'Stase
K
Uneertainty
3.2 Sentence-level Self-Reflection
Different prompts can significantly affect outputs of LLMs (Kung al. 2023 Peng et al. 2023b), introducing uncertainty into IT data evaluation at the sentence level. To make better use of this uncer- tainty to bolster the reliability of our method, implement sentence-level self-reflection (Sentence- R). Building upon Token-R, we devise K semantically similar rating prompts R.P. RPI....,RRA S } based on given sample S. We
to obtain a series quality scores {Stoken S Stoken calculate the average of these scores to represent the overall quality of sample S, because of the importance of incorporating assessments from diverse prompts. Additionally, we use the standard deviation quantify the LLMs' uncertainty to rating prompt: a higher standard deviation suggests greater sensitivity to prompt variation, while a lower standard deviation indicates more consistent and confident quality ratings by LLMs (Zhou et al., 2020). By integrating holistic sample evaluation with the quantification model uncertainty, we derive the sentence-level score gsent, offering a more nuanced and reliable measure of IT data quality.
Avg Wyg(Stoken K
Seent
1 x Std Stoken K
li
Uncertainty
where Avgf-} and Std respectively denote the mean and standard deviation of Stoken, K means the number of rating prompts RP.. Moreover, we use the uncertainty factor a to control for the impact of the uncertainty of LLMs on overall scores.
3.3 Model-level Reff-Refeetton
sample affirmed by multiple foundation models can truly be deemed as high-quality. Different foundation models have different quality assessments of the sample, which introduce model-level uncertainty. To maximize the utilization of this uncertainty, we introduce model-level self-reflection (Model-R). This strategy leverages the capabilities of existing open-source models without the need for additional resources or the complexities associated with fine-tuning. However, the challenge lies in the diverse capabilities of various LLMss and determining how to reasonably combine their sample evaluation based on their performance. It widely acknowledged that the capabilities of L.LMs tend to increase with their parameter count (Hendrycks et al. 2021) Thus, suggest using the parameter

---

count of LLMs as an initial metric for assessing their capabilities to properly weight sample quality scores. Given N foundation models with parameter counts {0,, .2.................... ................................ dN) and their respective
sentence-level scores for sample S being ( ..... : Sg we formulate the model-level score Smodel to reflect a comprehensive evaluation of sample quality.
0 E
Quality x Smodel =
x Speni
(5)
where N means the number of the foundation models. By obtaining LLM parameters without resource expenditure, Model-R effectively allows to employ more powerful foundation models, which is advantageous for selecting higher-quality data. Finally, we use 5 as the final evaluation of sample S in Select].. The higher Smmddel the better sample quality. We sort the samples in descending order based on their 5model and then select the top-ranked samples as high-quality data.
3.4 Selective Alpaca
We apply SelectIT to the widely-used Alpaca-GPT4 (Peng et al. 2023a). Specifically, we use the most popular LLaMA-2 (7B, 13B. 70B) as our foundation models and set the pyper-parameters 0.2 and K = 5, which decides the range of LLMs rating in Token-R and the number of rating prompts in Sentence-R. R. We finally select the top 20%, a total of 10.4K pairs as the high-quality data and obtain curated IT dataset called Selective Alpaca.
4 Experiments
4.1 Setups
Benchmark To gain a more comprehensive understanding of the capabilities of LLMs, we evaluate
approach in diverse downstream tasks (Wang et al.) [202 Ivison al. 2023). Factual knowledge:
We use the Massive Multitask Language Understanding dataset (MMLU (eeddrycks et al.] (021)) to assess the factual knowledge of LLMS and report 5-shot results. Reasoning: We evaluate the reasoning abilities of L.LMs using two widely utilized datasets: the Grade School Math dataset (GSM (Cobbe et al. 202T)) and Big-Bench-Hard (BBH (Suzgun et al., 2022)) with the CoT setting (Wei et al. 2022b). Multilinguality; we assess this ability by TyDiQA, multilingual question-answerin benchmark that encompasses 11 diverse languages, with the gold-passage setup. Coding: We evaluate this ability using the HvmanEval dataset (Chen et al., 202T) and report pass @ 10 results with a temperature of 0.8. Open-ended generation: We utilize AlpacaEval (Dubois et 2023). which employs GPT-4 to effectively assess model outputs. This can evaluate whether the text produced by LLMs aligns with humans.
Implementation Details We use LLaMA-2 as our testbed. We fine-tune it for epochs, with a batch size of 128. We use Adam with 31 0.9, B2 0.999, and the cosine learning rate scheduler starts from 2e-5, 5, and decays to we opted for a 4096 input length because it can show the best performance of LLMs. We employ the beam 4 for decoding. We set the temperature parameter to 0.8 and the top-p sampling parameter to 0.9 to improve the originality of the output text while ensuring the accuracy and relevance of the content.
Baselines We compare with the following baselines:
Alpaca-GPT4 (Peng et al. 2023a) is a widely- IT dataset that implements a eelf-instruct
method to autonomously generate instructions by the advanced GPT4.
LIMA (Zhou et al. 2023) primarily consists of 1000 manually crafted high-quality instruc-
tional data, which can better stimulate the alignment capability of LLMs.
2024) involves utilizing the robust ChatGPT to score and select data from the original A Paca-GGPT4 dataset.
AlpaGasus (Chen et al.
Q2Q (Li et al. 2023a) operates by training precursor model, determining the quality of the IT data based on the two different loss values within this model.
Instruction Mining (Cao et al. 2023) entails fitting data features and loss values to derive
formula for assessing data quality.
5

---

External
Overall
ID System
MMLU
BBH
GSM
TydiQA
CodeX
AE
AVG
A (t)
Model: LLaMA-2-78
Implemented Existing Method
Alpaca-GPT4
46.5
38.4
15.0
43.4
26.8
34.2
34.1
2 LIMA
45.4
37.5
14.3
45.1
24.6
33.1
33.3
-0.7
3 1 AlpoGasus
x
45.9
39.0
46.4
27.5
35.4
34,8
+0.7
I Q2Q
x
46.9
39.4
15.3
46.7
28.2
35.7
35.4
+1.3
5 Instruction Mining
47.0
39.6
16.5
47.1
28.6
34.4
35,5
+1.5
Our PeoposedMMehoad (adihhaaaa
6 1 Token-R
46.8
36.5
14.5
44.6
28.9
35.5
34.5
+0.4
7 1 Sentence-R
46,9
38.1
16.1
48.4
26,9
35.3
35,3
+12
8 Model-R
47.3
37,4
16.1
45.3
28.4
35.8
35.1
+1.0
Our Proposed Method (AIl)
Select|T 7
X
47.4
40.6
16.8
47.4
29.4
35.7
36.2
+2.2
Base Model: 1.aMA-2-1330
Implemented Fxisting Method
Alpaca-GPT4
55.7
46.6
30.5
48.1
40.8
46.5
44.7
11 LIMA
54.6
45.3
51.1
34.1
42.6
43.0
-1.7
12 10 AlpaGasus
54.1
47.3
31.5
50.6
41.3
46.3
45.2
+0.5
13 10 Q2Q
55,3
48.5
32.0
50.8
41.3
47.3
45,9
+1.2
14 10 Instruction Mining
54.1
47.3
32.5
$2.6
43.3
48.3
46.3
+16
Oür Pipposed Meihod [Individha!]]
15 10 Token-R
55.3
47.3
30.5
51.3
39.8
46.2
45.1
+0.4
16 Sentence-R
x
x
55,2
48.3
$2.2
42.5
46.3
45,9
17 + Model-R
x
47.5
31.5
52.3
40.2
46.1
45.5
40.8
Our Proposed Merhod (AN)
SeleatT (15 16 17)
55.7
48.9
33.0
54.1
42.2
47.1
42.4
Table 1: Overall results on IT. "CodeX" and "AE" mean HumanEval and AlpacaEval benchmarks. All the scores are averages of three independent runs with different random seeds.
4.2 Main Results
We focus the discussion of LLaMA-2-133 because both 7B and 13B models exhibit similar trends in [ System (10) shows the vanilla IT on LLMs with the original Alpaca. By using the data selection strategies, the ability of LLMs has moderate enhancement in Systems (12) (14). Additionally, we can use Sbase as the input for Equations and 5 construct individual methods of Sentence-R and Model-R. Systems (15) to (17) illustrate that applying each submodule of SelectIT incrementally enhances LLMs' performance, rivaling contemporary advanced methods.
Most remarkably, SelectIT can better boost LLaMA-2's performance compared to vanilla IT in the System (18). Compared to other IT data selection strategies, this enhancement is particularly evident in the computational and reasoning tasks the BBH and GSM benchmarks. This may be attributed to the characteristics of selected data by SelectIT, and we will analyze this phenomenon in a later section. These gains in reasoning ability also positively impact the coding proficiency of LLMs. The improvement of LLMs on the TydiQA dataset also obvious enough, which shows that SelectIT can effectively eliminate similar samples and retain sufficient diversity in multilingual aspects.
5 Analysis
This part aims to answer the research questions through the following experiments: How to select high-quality data in SelectIT? ($5.1) Select|T adaptable to various models and domains? ($5.2) How about the efficiency of SelectIT? ($5.3) What are the advantages of Selective ($5.4
5.1 Abalation Study of SelectIT
1LM AMAnn a
Effeet of IT Data Quantity While SelectIT already excels at assessing and ranking samples effectively, selecting an appropriate number of samples in a redun- dant dataset remains crucial aspect of our method. We divide the Alpaca dataset into multiple subsets ranging from 10% to 100% based on SelectIT's evaluation and
Size
ofThezet
Figure 3: Comparison of LLM abilities with varying Alpaca proportions.

---

evaluate the overall ability of LLMs on the open-instruct benchmark. As illustrated in Figure 3 compared to using the full Alpaca dataset, we observe that LLMs achieve optimal performance using the top 20% to 40% data. Hence, considering the tradeoff of training resources, training time, and model performance, we opt for 20% for implementing the SelectIT on the Alpaca dataset.
Effect of Multiple Rating Prompts K is critical parameter for our method, impacting not only the range of scores assigned by the LLMs but also the number of rating prompts. We set K
K
LLaMA-2-7B LLaMA-2-13B Overall
35.6
46.4
40.5
36.2
47.1
41.7
7, 9 and apply SelectIT for sample selection within the Alpaca to get different subset datasets. Table 2 indicates that vari- ations in the value of K have minor impact on the overall performance the LLMs. This is attributed to multi-gran nulti-granularity self-reflection mechanism, which effectively enhances the accuracy and stability of sample selection. Although the model achieves competitive performance at K values of 5 and to minimize resource consumption, we set K 5 as the default value in SelectIT.
35.7
47.3
41.5
36.0
46.8
41,4
Table Effect of different K.
Effect of Uncertainty is an uncer- tainty factor, integral to calibrating the equilibrium between the mean and the standard deviation of scores derived from
MMLU
Tydiqa
CodeX
AVG
47.4
40.6
16.8
47.4
29.4
36.2
0.4
47.9
39.4
15.5
46.5
29.4
35.8
35.8
47.8
39.8
16.5
45.6
29.1
35.1
35.7
Token-R. We assign a four different val- ues, i.e., 0.2, 0.4, 0.6, and 0.8, and in- corporate SelectIT for sample selection from within Alpaca to generate disparate subset datasets, with all other parameters remaining constant. As shown in Table with a rise in the value, Sentece-R tends to emphasize the uncertainty innate to LLMs. This results in the neglect of the average score, fundamental indicator of sample quality, thereby contributing to a decrease the overall performance of LLMs.. Consequently, ascertain that a value of 0.2 is optimally suited to establish an effective balance between the sample's 's quality and the model's 's uncertainty.
47.6
16.5
43.6
26.7
35.4
34.4
Table 3: Effect of different a.
Effeet of Different Reflection Strategy We analyze the relationship between individual selection strategies and Select]T, from the following two aspects. We first account for the number of high-quality data that can only be selected by unique selection strategy, referred to as unique selection. Secondly, we calculate which samples in Selective Alpaca can be selected by indi- vidual selection strategies Selective Alpaca, called overall selection. As shown in Table 4 Sentence-R plays the most important role in the final Select strat- egy. This is because rating prompts play an important role in sample evaluation and exploiting the effect of different prompts on LLM can effectively better improve the accuracy of sample evaluation than Token-R and Model-R. Additionally, this phenomenon also aligns with the model's performance reported in Table showing the rationality of our proposed uncertainty-aware self-reflection methods.
1D
Individual
Unique (%)
Overall (%)
Token-R Sentence-R
6.18
17.83
40.81
63,98
Model-R
7.37
23.08
Table 4:
The relationship between the Se- lectIT and the individual selection strategy, Sentence-R plays the most significant pact on the final rating of the IT data. IDs 7, and 8 correspond the system of the same IDs in Table [
Overall
Base Model
Datasets
Data Size
MMLU
BBH
GSM
Tydiqa
CodeX
AE
AVG
LIMA
45.4
14.3
45.1
24.6
33.3
LLaMA-2-7B
Selective Alpaca
IK
46.6
41.3
14,5
46.2
30.6
33.8
35.5
42.2
AlpaGisus
9K
45.9
39.0
14.5
46.4
27.5
35.4
34.8
Selective Alpaca
47.2
41.3
18.5
47.6
28.3
35.4
36.4
+1.6
Table Results on for different datasets with the same number of instances.
Effect of Data Imbalance To eliminate unfair comparison caused by IT data quantity imbalance, we adjust the size of the Selective Alpaca dataset to 1,000 and 9,229 respectively, aligning with the LIMA (Zhou al. 2023) and AlpaGasus (Chen et al. 2024 datasets. The results in 5
7

---

Overall
Base Model
Datasets
MMLU
BBH
GSM
Tydiqa
CodeX
AE
AVG
A (t)
Alpaca GPT4
46.5
15.0
43.4
26.8
34.2
34.1
LLaMA-2-7B
Selective Alpaca
47.4
40.6
16.8
47.4
29.4
35.7
36.2
+2.1
Alpaca-GPTT
55.7
30.5
47.1
38.8
46.3
44.2
LLaMA-2-138
Selective Alpaca
55.3
48.5
32.5
54.1
41.2
47.8
46.6
+2,4
Mistral-7B
Alpaca-GPT4
52.5
51.7
33.5
51.1
54.7
43.1
47.8
Selective Alpaca
56.9
53.7
36.0
49.3
55.3
44.3
49.3
+1.5
Alpaca-GPT4
59.6
52.3
34.5
43.1
60.2
48.2
49.7
LLMMA-3-8B
Selective Alpuca
61.2
55.0
37.5
41.1
63.4
47.7
$1.3
+1.6
Table 6: Results of IT with various foundation models.
Overall
Datasets
Data Size
MMLU
BBH
GSM
Tydiqa
CodeX
AE
AVG
A (1)
WizardLM
143K
43.8
37.8
10.0
41.2
25.2
35.3
32.2
Wizardi Select|T
11.0
43.1
34.7
33.6
Orca-GPT4
IM
40.1
35.6
13.0
46.0
23.3
38.1
32.7
Orca-GPT4 SelectIT
0.2M
43.9
38.7
16.5
42.0
27.7
37.4
34.4
+1.7
Table 7: Results of IT with various IT datasets.
that, when facing the same amount of data, SelectlT can still demonstrate better performances, which further illustrates its effectiveness.
5.2 Robustness across Models, Datasets and Domains
Various Foundation Models Although Selective Alpaca achieved impressive improvements in LLaMA-2, applying to other foundation models remains a challenging task. To address this, apply Selective Alpaca the Mistral-7B and LLaMA-3-8B LLMs and present results the open-instruct benchmark alignment with the above test configuration. As depicted in Table although Selective Alpaca is selected by the LLaMA-2 models, it is also applicable the Mistral-7B, LLaMA-3-8B and improves their capabilities across various tasks, especially on MMLU, BBH, and GSM benchmarks. This experiment fully demonstrates the flexibility of Select|T which does not rely on specific foundation model for data selection and the universality of Selective Alpaca which can effectively improve the capabilities of different series or scale LLMs.
Various Instruction Tuning Datasets We further validate the robustness of SelectIT by deploying on two additional, widely-utilized datasets: WizardLM (Xu et al.) (203)) and Orca-GPT4 (Subhabrata Arindam 2023). WizardLM introduces an innovative method of using L.LMs to auto-generee open-domain instructions of varying complexities. This allows for a controlled variation in instruc- tional difficulty and the dataset comprises 143K samples. Orea-GPT4 on the other hand, leverages rich signals from GPT-4 that include explanation traces, step-by-step thought processes, and other multifaceted instructions, all under the guidance of teacher assistance from ChatGPT. Additionally, we maintain consistent hyperparameterr, such as and K, choosing LLaMA-2-7B as our base model. limit the fine-tuning of these datasets to one epoch. As shown in Figure 7 SelectIT consistently enhances the performance of the model both the WizardLM and Orca- GPT4 datasets. Notably, this augmentative effect is especially pronounced in the computational and reasoning tasks within the BBH and GSM benchmarks. In evaluating three separate IT datasets, specifically Alpaca-GPT4, WizardL M,, and the more extensive Orca-GPT4, our extensive experimental conclusions validate the broad utility and durability of SelectIT.
Various Domain-specifie Tasks Machine translation (M') is representative domain-specifer task of L.LMs. Previous works have already demonstrated significant improvements with L.Mss, but they usually use redundant translation IT datasets. This part tests the robustness of SelectIT on the IT dataset of MT. We select the powerful MT LLM ALMA (Xu et al. 2024) as our backbone model.
We choose the representative language pairs (German, Chinese Sriinsse<><nnee Ennllss from WMT' 17 to WM hurteewritten test datasets, and development and test sets from Flores-200, totaling 30K training examples. We used WM test data for testing, and finally, 6K high-quality

---

examples were selected using SelectIT. We utilize both BLEU (Post 2018 Ott et al. 2018) and COMET (Rei et al,, 2022) based on the wm122-comet-de model for evaluation. We report results for the two language pairs in four directions, using ALL to represent their aver- age. Table shows that SelectIT consistently improves ALMA's translation performance. These results indi- cate that SelectIT is a versatile and scalable method, effective not only for IT data selection but also for domain-specific tasks like MT. For more detailed anal- ysis and results, please see Appendix A.I
Method
CUMET
BLEU
So7A Models
NLLB Eonoaaat a. proc
34B
78.8
26.3
GPT-33
85.6
34.8
GPT-4
85.8
35.1
Fristing Merloud
IL-MA-2
TIM (emeen
26.4
27.6
78.8
21,9
Bayling (owe aill caalony
133
82.0
27.8
UUr Impleveented Meihid
ALMA
78
83.2
29.7
Sclectf
78
83.7
30.5
ALMA
13B
$1.5"
SelectIT
32.2
5.3 Efficiency of SelectIT
Table 8: The overall results on MT LLMs.
SelectIT is a faster and more cost-effective method for IT data selection. We compared different selection methods on the Alpaca-GPa dataset. For ChatGPT (AlpaGasus) GPT-4, we randomly select 500 instruc- tion data from Al Alpaca-GPT4, analyze various metrics, and estimate the resource consumption for selecting the entire dataset. Using SelectlT, we employ 4 A800 80G GPUs to select high-quality IT data, calculating the total cost based on Google Cloud's rate of $1.15/h per single GPU. As shown in Table SelectIT is significantly faster and uses the least resources. This efficiency due to computing only the probability of the next token for input sentences, bypassing the full sentence generation and decoding process, resulting in lower resource consumption. Additionally, using our own GPU at low cost enhances transparency, allowing us to preserve all intermediate outputs and results for thorough analysis in data selection.
Method
Speed
Time
Cost
ChaiGIT API
0.76 it/s
$52.02 $2871.56 38.98h
19.07h
GPT4 API
0.37 it's
SelecdI
9.34 it/s
5,80h
$26.68
Table 9: Comparison of selection efficiency.
5.4 Insights of Selective Data Curation
Different Selection Strategies This part compares three different selection strategies, namely, randomly selecting 20% in the full Alpaca and unselected dataset of Selective Alpaca, and selecting 20% data based on sample length (Zhao et al. 2024p. As shown in Table the random-based strategies show certain performance degradation and the random selection in the unselected dataset is even worse, which reflects the effectiveness of our method from the side. Selection based on sample length is a simple approach to defining high-quality data, but it does not take into account the content of IT data, resulting in the limited performance of LLMs. SelectIT can significantly improve the abilities of LLMs.
Method
LLaMA-2
ALMA
AT))
13B
7B
13B
Full Damset
29.7
Rundom (Fulli
0.0
Random (Unselected)
29.1
31.2
-0.4
w/ Length w/ Selscif
35.5
47.1
30,1
31.8
+5.0
36.2
47.1
30.5
32.2
46.5
Table 10: Comparasion with variants.
Data Representation Analysis the original datasets from representation perspective. Following Gao et al. (2024), we use the outputs of the last layer corresponding the last token in the input sequence as sample representations. We then apply T-SNE (Hinton & Roweis 2002) for dimensionality reduction, mapping high-dimensional embeddings onto a 2D space. Figure shows the intermediate representations generated by the full and Selective Alpaca datasets. Randomly selected data struggle to distinguish abnormal data far from the center, making hard define high-quality IT data. In contrast, Selective Alpaca data are mostly concentrated around the center, indicating that our dataset predominantly contains high-qualis data near the center and effectively discards abnormal data, supporting the conclusion of Table I0
This part explores the relationship between Selective Alpaca and
Data Characteristic Analysis We analyze the Selective Alpaca from the following two perspectives, to explore why our dataset better than the original dataset and its variants. Firstly, as shown in Figure [5] the length of instructions from the Selective Alpaca is significantly longer than those in the Alpaca dataset and AlpaGasus which is selected by ChatGPT. This implies that, with the same amount of data, our dataset contains more information, aligned with the results in Table 10 Secondly, by using ChatGPT to examine IT data types, we find substantial increase the proportion of computational problems in Selective Alpaca. This indicates that Selective Alpaca tends to select

---

100
300
Full Alpacs
Selective Alpaca
7.5
250
Percent (%g
-30
-200
Fut Alpacs
100
100
That Alpucs
Scleatte AQUAO
100
200
2,
1,53
Figure 4: Instruction embeddings representations of different selection strategies. The red and blue points are representations of full Alpaca datasets and selected data respectively.
0.0
Calorlaaas
Figure 5: Left: The average length of samples. Right: The proportion of calculation type.
high-quality mathematical data, providing a solid explanation for the observed improvement in the reasoning abilities of LLMs as demonstrated in Table comparing the Selective Alpaca with AlpaGasus.
Appendix A.3 shows the case study of
Insights of High-Quality Data in SelectIT Further- more, we analyze the proportion of calculation and sample average length in Alpaca-GPT4 with different proportions after sorting by SelectIT to explore its in- trinsic characteristies and the definition of high-quality data. As shown Figure [6] with the proportion of Alpaca-GPT4 data continuing to increase, the propor- tion of calculation and sample average length gradu- ally decreases. This phenomenon clearly indicates that SelectlT' can reasonably rank samples based on their characteristics. When the data size more than 50%, the proportion of calculation IT data sharply declines, falling below 6%, causing a noticeable decrease in the model's 's overall capability, as de- picted in Figure This analysis shows that more computationally intensive IT data may new perspective on the characteristics of optimal IT data, which not only effectively improves the LLMs' reasoning ability, but also further drives the improvement of other abilities.
Pooees Culooaation
Calculation Length
AATTA Amgh
15 5%
10% 30. 37% 45% Wh: 87% 3P, 97% 1005
Proportion Alpac--PP14
Figure 6: Changing trends of the calculation and sample length with different data sizes.
Conclusion
This paper introduces a novel data selection strategy, SelectIT, for LLM instruction tuning, which uses LLM uncertainty to efficiently identify high-quality IT data without requiring additional resources. SelectIT includes three types of self-reflectton: token, sentence, and model, which can individually and jointly improve the performance of IT data selection. By applying SelectIT to the Alpaca-GPT4 dataset, we introduce a compact and strong dataset, called Selective Alpaca. Different models and domain tasks demonstrate the effectiveness of SelectI.. Our analysis reveals that SelectIT effectively excludes abnormal data and tends to select longer and calculational data.
Limitation
This paper could be further strengthened as follows:
Instruction Data Quantity: Our findings suggest that prioritizing the top 20% of high- quality data optimizes results for Alpaca. Future studies might explore adjusting this threshold based on the data quality in different datasets to enhance performance.
Models Different Scales: Our analysis is currently limited to models smaller than 30B parameters due to computational constraints. Investigating the efficacy of Selective Alpaca on scale LLMs, could provide valuable insights into the method's 's scalability.
Expansion to Additional Instruction Datasets: Although SelectIT has been applied to the Alpaca dataset due its widespread adoption, extending this methodology incorporate other IT datasets could offer substantial advantages to the broader LLM research community.
10

---

Broader Impacts
Our work follows the NeurIPS Ethics Policy, Our findings are based on publicly available datasets for reproducibility purposes. LLMs can contain potential racial and gender bias. Therefore, if someone finds our work interesting and would like to use it in a specific environment, we strongly suggest the user check the potential bias before usage. In addition, it is hard to control the generation of LLMs. We should be aware of the potential problems caused by hallucinations.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China
(Grant No. 62206076), Guangdong Basic and Applied Basic Research Foundation
(Grant No. 2024A1515011499 Shenzhen Science and Technology Program (Grant
SCBS20021000809321003 and Shenzhen College Stability Support Plan (Grant Nos. GXWD202208 1173340003, Derek F. Wong was supported in part by the Science and Tech- nology Development Fund, Macau SAR (Grant Nos. FDCT/060/202//FF, FDCT/0000/202//MM) National Natural Science Foundation of China (Grant No. 62261 166488) the Multi-year Research Grant from the University of Macau (Grant No. G5
Nos. ZDSYS202306260J 120000,,,00100008, 1203008, KJZDZ003102004400000,
WWWD20220817123100000
and the Tencent AI Lab Rhino-Bird Gift Fund (Grant No. FF2002-00151-FSTT) We would like to thank the anonymous reviewers and meta-reviewer for their insightful suggestions.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ArXiv preprint, abs/2303.0877 08774, 2023. UtLtttss:://axtt org/ abtp:/bbb/23330033 08774
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models. ArXiv preprint, abs/2307,06290, 2023. /ttptttps://axxi org/
abs/2307.nm .62900
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data. In The Tweifth International Conference on Learning Representations, 2024. URL htttps://openreview wttp:/rrwwieeeeeeeeooouuumeeeeee met./Eorumm.d-FdVXgSS...
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et Evaluating large language models trained on code. ArXiv preprint, abs/2107.033744 2021. URL https://arxiv
org/abs/2107.c 03374
Mayee F Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Re. Skill-it! a data-driven skills framework for understanding and training language models. In Thirty-seventh Conference Neural Information Processing Systems, 2023a. URL https httpaarwwweee..eeeffuouiuiiii-/
Yijie Chen, Yijin Liu, Fandong Meng. Yufeng Chen, Jinan Xu, and Jie Zhou. Improving translation faithfulness of large language models via augmenting instructions. ArXiv preprint, abs/2308. org/ abs////s///////////////////aaa 12674
2023b. URL attpss//arrie
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https://imsys
org/blog/2023- 0300003-30 30-300vv-0 vicuna/
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. ArXiv preprint, abs/2210.11416 2022. URL httphhttt///a/a https//org//b/b//////eeee 11416

---

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. ArXiv preprint, abs/21 10.14668, 2021. URL /ttxps://arii.... org/ abs/
2110. 14168
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. ArXiv preprint, fas/2207.044622 2022. URL https://
arxiv.org/abee 020../2207.00 2207.04672
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. J httpt::t stttps://oppnreeveee http://www..eeeeeooaararaatttraa
Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang. Towards boosting many-to-mant multilin- gual machine translation with large language models. ArXiv preprint, abs/2401.05861, 2024. URL https://arxiv. org/ abs/2401.... 05861
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, 2020. Association for Computational Linguistics. doi: (0.18653/11/2020. acl-main.740, URL
org 22000 acl- m.llmmmin..74.....
Xiaochuang Han, Daniel Simig. Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12660-12673, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 10.18863311/2222aclll 708. URL. https://aclanthon
2023 acl-long. 708
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representatiins, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL http:s:://ppeeeeeaaiiii
Geoffrey E. Hinton and Sam T. Roweis. Stochastic neighbor embedding. In Suzanna Becker, Sebastian Thrun, and Klaus Obermayer (eds.), Advances in Neural Information Processing Systems 15 INeural Information Processing Systems, NIPS 2002, December 9-14, 2002, Vancouver, British neurips Columbia, Canada). pp. 843.840. MIT Press, 2002. URL Abstracttbbbtatt.aa.aa.acccaac.e html
cc/ httpppp//002/haaa///// http://000001///ccooooo666666660
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv preprint, abs/2212.09689, 2022. URL
https://arxiv.. org/ abs/ abs/2212. 2212. 09689
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023. URL WtRthttpsstt wwwxiv.oor//ae abs/
2311 10702
Po-Nien Kung, Fan Yin, Di Wu, Kai- Wei Chang, and Nanyun Peng. Active instruction tuning: Improv- ing cross-task generalization by training on prompt sensitive tasks. ArXiv preprint, hbs/2311.00288. 2023. URL/https: //arxiv org/ abs/ 00288
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. ArXiv preprint, (ts//3388.10322. 2023a. URL https
//arxiv. wwwp:ogg//abbaaaeeee 12032
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. ArXiv preprint, abs/2308.06259, 2023b. URL https: httpp/:///xxv..o// org/ http://.b/230...000
12

---

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. One shot learning as instruction data prospector for large language models. ArXiv preprint, abs/2312.103022 2023c. URI pettps//arriie org/abs/ 2312. 10302
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. ArXiv preprint, abs/2312.156855, 2023. URL https: //trx/aaxvv.///////////eaeeaaaa. wwwziii.rgg//oo abs/ aba/2312 2312. 15685
Shayne Longpre, Le Hou, Tu Vu, Albert Wehson, Hyung Won Chung, Yi 'ay,, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. ArXiv preprint, abs/2301. 13688, 2023. URL httph:tt////rate http//rxxvvv..aaaaaaa htrg/abb/23111 13688
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, Brussels, Belgium, 2018. Association for Computational Linguisties. doi: 10.1 11111166//1111--/11 URL ptttps://aclathhan
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. ArXiv preprint, abs/2306.01 16, 2023. URL https://axxi org/ abs abs/a3/6.....////// 2306 01116
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, ArXiv preprint, abs/2304.032777, 2023a. URL htt.thttss/:aaii..attt abs/ 0tbbs/330400332
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards making the most of ChatGPT for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Singapore, 2023b. Association for Computational
Linguistics: EMNLP 2023, pp. Linguistics. doi: 10. 101118655/110/ 122033///222322// madings-emmlp.373.. (tt.sttss://cccaatoooo 2023. eandingg-eenl..... 373
org/
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1//533/v1WIB htt.//111-660 6319. htLittts//cclaatthoo
org/
W18-6319
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), 2022. Association for Computational Linguistics. URL Intpp:: http://acaaathooogoooo org/ 2022. wmt- 1.52
Mukherjee Subhabrata and Mitra Arindam. Orca: Progressive learning from complex explanation traces of gpt-4. https://arivvorr//////20000..000 2023. URL htttp::/aaxxvvorrg/ppee.
02707
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. ArXiv preprint, abs/2305.0304f 003047, 2023. URL httptt/////axii org/ abs/2305.08 03047
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether of can solve them. ArXiv preprint, abs/2210.09261, 2022. URL
https: /ottps://arxiv... wwwaxii.ogg/aaeee abs/ abtgg//bb/2210./ 09261
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instruction-folloving llama model.
https: httpps////ttttbbb... wwt.:uu...oottttttoeeaaoaaa ltmm///aatsulaaa/ttaooooord alpaca 2023,
13

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023a. URL https
//arxiv.org/aal htrg/abb/23000 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307 000//337.00288, 2023b. URL pattps://arxila org/
abs/2307.0922
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36, 2024. URL https://arxxi:i wwwpxii.or//aae abs/2006...... 04751.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representatiins. ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net, 2022a.
net/ wttp://weeeeefotnoneeeeeee http//oooomndddzddeeeeeee
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. ArXiv preprint, abs/2201.11903, 2022b. BtRpsstts:///rrttttt www.aiiwwwwiaaaaaaaeaaaaooeeeeee
org/ a2t/ 11903
Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. ArXiv preprint, abs/2311 1.01111.122,, 2023, URL htttps//arxiv..rgg abs/ 2tw1.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. ArXiv preprint, abs/2402.04333, 2024.
URL https://arxivv wwg.i.ogg./..i..a............otr abs 2202. 04333
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances Neural Information Processing Systems, 36: 34201-34227, 2023. URL bttps::////aax org/abs. awwpaa//.a.aaaaaeaa/. 2302. 03169
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304.12244 2023. Jtttp::s:://arzii org/
12244
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in ma- chine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htt.htttt///pperee/eeee net/
Forum?id=ffrTTXXim
Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. Bigtranslate: Augmenting large language models with multilingual translation capability over 100 languages. ArXiv preprint, abs/2305.18094 18098, 2023. JRLptttps:/// /tttpss:/arrxiie wwtp://v.rrr///////////aaase
18098
Jiali Zeng. Fandong Meng, Yongjing Yin, and Jie Zhou. Tim: Teaching large language models to translate with comparison. ArXiv preprint, 0w//2307.04005 2023. https://///arxi org/
abs/2307.044p
Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. ArXiv preprint, abs/2306. 10968, 2023. URtpssss:///axi org/ 230/ 10968.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning, 2024. URL
htttss://arxe org/oog//aa abs//402. 04833
14

---

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpss:/ http:///www...eeeeeeeeee//////aa wett:/oormmm//-4aMMMa.t...
Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. Uncertainty-awar curriculum learning for neural machine translation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6934-694 Online, 2020. Association for Computational Linguistics. doi: 10.18653/v 1/263//1/2020.ac// wcl--mcccmmmm main 620. URL https: accltthoolll
orr/20200. acl main.62n 620
15

---

A Appendix
A.1 Applying SelectIT on Machine Translation LLMs
Machine Translation (MT) a important task for LLMss, demonstrating their domain-speciniii capabili- ties. Prior research, including TIM (Zeng et al. 2023). SWIE (Chen
al. al. 2023bb) BigTranslate (Yang et al. 2023), and Bayling (Zhang et al., 2023 has shown sigmificant improvements in LLMs, often relying on extensive translation training datasets. In this section, we examine the impact of training data quality on MT performance, employing the robust MT LLM, ALMA, as our foundational model (Xu al. 2024).
For training data, we select representative language pairs: "rrmann==Engiisi and Chinese+ sourced from WMT" 17 WMT' human-authored test datasets, supplemented with development and test sets from Flores-200, totaling 30K training instances. We use the corresponding language pair's test data from 2M as evaluation datasets. Subsequently, 6K high-quality instances selected for LORA fine-tuning via SelectIT.
We report both the widely used BLEU score (Post 2018 Ott et al. 2018) and the COMET score (Rei ett al. 2022) based on the wws22-comet-eeee model, which shows higher correlation with human judgments for evaluating the LLMs' translation abilities. Table consistently demonstrates that SelectIT enhances ALMA's translation efficacy. Notably, SelectIT primarily focuses on improving translations from English to other languages, likely due to ALMA's inherent proficiency in English, which presents challenges for further enhancements. These findings highlight SelectlT's adaptability and scalability, validating its effectiveness not only in IT data selection but also in domain-specific tasks such as MT.
Method
Size
En=De
De=
Zh->En
En
ALL
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
SoTA Models
NLLB
54B
86.5
34.5
78.9
26.9
70.7
16.6
78.9
27.4
78.8
26.3
GPT-3.5
87.0
34.4
85.5
33.1
82.9
26.6
87.0
44.9
85.6
34.8
GPT-4
87.4
35.4
85.6
33,9
82.8
27.2
87.5
44.0
85.8
35.1
Existing Method
LLaMA-2
7B
76.4
19.0
82.7
30.4
75.0
18.2
71.8
17.0
76.5
21.1
TIM
7B
74,2
20.6
77.7
24.3
79.5
23.4
84.9
37.2
26.4
SWIE
7B
82.4
27.2
83.0
30,5
76.5
21.3
80.6
31.2
80.6
27.6
BigTrunslate
13B
78.8
80.7
23.4
81.3
28.6
78.8
21.9
Bayling
13B
82,7
25.6
83,0
27.3
77.7
20.1
84.6
37.9
82.0
27.8
Our Implemented Methoi"
ALMA
7B
85.0
29.9
83.9
30.0
79.2
22.7
84.8
36.3
83.2
29.7
w/ SclectT
7B
85.2
30.2
84.1
30.4"
80.0
24.2
85.31
37.31
30.5
ALMA
138
85.2
31.6
84.2
30.9
80.6
25.0
85.5
39.2
83.7
31.5
w/ SelectfT
138
85.8
31.7!
84.6
31.4
80.3
25.4
86.1
40.4
84.2
32.2
Table 11: Overall results machine translation LLMs. to the ALMA model (p < 0.05).
4f" the improvement is significant by contrast.
A.2 Details of Sentence-level Rating
Based the preceding analysis, Sentence-R is integral to the functionality of SelectlT. As illustrated Equation 4 the Token-level Rating forms the foundation for the Sentence-level Rating. The Model-level Rating is derived through multiple iterations of the Sentence-level Rating across different foundational LLMs. Therefore, a detailed explanation of Sentence-R R is sufficient to demonstrate the operational mechanism of SelectIT. As depicted in Figure 7 utilize five distinct rating prompts along with a single input to formulate the final input for Sentence-R R. Initially, each rating prompt produces score of gtoken We then compute the mean and standard deviation of these Stoken values to obtain the final 5eent, as outlined in Equation 4
A.3 Case Study
we illustrate the selection tendencies of SelectIT in contrast to Alpa- Gasus, which leverages advanced ChatGPT for data selection. In samples to 4, Select|T shows preference for instruction-tumn tutionntttnnnnnnnnnnnnn data containing intricate mathematical problems that contribute
As demonstrated in Figure 8
16

---

to improving the reasoning skills of the LLMs. On the contrary, AlpacaGasus frequently chooses IT data in samples 5 to that primarily offer solutions queries or lack coherent reasoning, which might limit its effectiveness.
17

---

<| Rating Prompt |
Assign score
each input based on accurately they follow and respomse provided, ensuring
represented clearly its
<| Input !>
Input: For the given imput, need to predict the result of the operation 5-9
Oulgul:
The result of the opuration 5 - is
Next Token Prediction |>
Score 1.1
Proh
0.05
0.5
0.05
<| Rating Prompt 2 |>
input scale from reflecting accuracy their adherence instructions inpat, present plainly without ueed
details.
Imput |>
Input: For given inpur, you need predict the result operation
Output: The result the operation 9
< Next Token Prediction
Token
5
Eq..
Score
< Rating Prompt 3
Rate each input accuracy to the given task and input on scale of to 5, with being the most precise; the score should be sell-explamatory and presented as single line.
<| Input
Input: For the given inpur, need predict the result of the operation 9
Ourput:
The result of the opemtion 5 9 is
4 Next Token Prediction
Token
2
5
Eq.
Prob
0.18
0.02
0.1
0.1
0.6
<|
Rating Prompt /
each ispul scale based their adberence instructions and the accuracy their resporaes, cleatly displayed.
<| Input |>
loput: the given input, you need to predict the result the operation 5
Output: result the operation
<| Next Token Prediction |>
Ey.
Token
2
5
Score
Prob
0.05
0.1
0.2
0.15
0.5
<|
Rating Prompt |>
Assign to every imput score ranging from 5, evaluating their compliance with instructions and the precision of their feedback, with the score heing conspicuously presented.
< Input
Input: For the given input, you need predict the result the operation -
Output: The result of the operation 5 9 is
< Next Token Prediction
Token
2
5
Eq.
Score 4.4
0.03
0.01
0.04
0.9
Evaluation
Rating
2
5
Score
1.1
1.5
1,9
2.5
4.4
SID 1.2 sient Fg.
1.9
Figure Example on Sentence-R calculation of SelectIT.
18

---

Sample
Selectl AlpaGasus
Input:
Compute the value following equation: 4+4/4.
The value of the cquation 4 4- -444 is 5. According to the order of operations, division must be done before addition, the equation be rewritten as 4 (4/4), which simplifies to 4+ =5
Response:
Included
Discarded
Sample
Input:
Compute derivative of 2x 5x.
Response: The formula for the derivative of function is: f(x) d/dx(f(x))
Using this formula, let's differentiate the functiin I(x) 2x*2 5x with respect
x.d/dx(2xX2 5x) 2ydx d(5x)/dx. Recall that the derivative of x'n
Included
Discarded
is (n-(1-)), Applying this to the first term: d(2x*2)/dx
d(xx2)dx 2 2x
And the derivative of x*x, where is constant, is k. Applying this the second term:d(5x)ddx - S.Therefor,, the derivative of 2x^2 + 5x is:
f(x) 4x 5
Sample
Input:
Find the solution to the quadratic equation:3x + 7x
Response:
The given quedratic equation is 3x' + 7x 4 = 0.
can solve this equation using quadratic formula: x (-b sqri(b² 4ac))
2a.Here, a 3, b = 7, and с 4.Plugging these values into the quadratic
Included
Discarded
formula, we have:x [-7 sqri(7² - -4)] (2 = sqri( 773)] 6
So, solutions the quadratic equation are: (-7 sqrt(73)) / and x
(-7 sqrl(73)) /
Sample
Input: Variable x is defined as "4x 2y 10". Find the value x.
Response: Given the equation 4x 2y = 10, there infinitely many solutions
for the value of x, depending on the value of other words, the value is dependent on the value of Howeerr, it is still possible to express x in terms of To do so, can rearrange the equation solve for
Included
Discarded
2y
4x = 10-2y
2y)/4
So, according the equation, value x (10- 2y)/4. Without knowing the specific value of it is not possible to determine the exact value of x.
Sample
Input: Calculate volume of cylinder with a radius of em height of 5 cm.
Discarded
Included
Response: The volume of the cylinder is 785 cm3.
Sample
Summarize the paragraph into sentence: When buying furniture, it's important take into account the overall look and feel of the space. That means that you should ask yourself the pieces you are selecting are good match for the existing furnishings as well as what you imagine the overall style to be.
Input:
Discarded
Included
Response: It's essential to consider the overall aesthetic the space and ensure the furniture matches the existing furnishings and desired style when purchasing furmiture.
Sample
Input: Render 3D model of house.
Response: I'd love to create 3ID model you! However, I'm
language model and I do not have the necessary tools to create visual models. You may try using modeling software such Blender, Spetchl/p or AutoCAD create your model. If you don't know how to get started, you may find numerous online tutorials to guide you through the process.
Discarded
Included
Figure 8: Examples of IT data selected by SelectIT or AlpaGasus.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
See the abstract and introduction sections. Our proposed SelectIT can capitalize on the foundational capabilities the LLM itself more effectively select high-quality IT data, without the need for extra resources. We run comprehensive experiments to support our assumption. Our contributions are stated clearly in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the limitation part, we have discussed the points where SelectIT could be further optimized, including the data quantity, model scales, other foundation models, and datasets.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
the abstract section, provide the GitHub link open source all the code, scripts, and datasets (Selective Alpaca) for other researchers to replicate the results. We also provide the implementation details better reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: In the abstract section, we provide the GitHub link to open access to data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wttpcc/pcccccccccccooooooop httec/ggguudeeeeeeeuuuubiiiiiiii wwweiisssonPoPoiiiioioo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4.1, we have discussed datasets, baselines, and experimental setup used in our experiments. More training details are included in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the section about applying the SelectIT on MT LLMs, we do the statistical significance tests in Table 9, which is reported in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 5.3, we provide the type of computing workers, memory, and time of execution to help other researchers reproduce the Selective Alpaca.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: we have read the guidelines and ensured that our paper conforms to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 5.1, Selective fewer computing resources to select high- quality data, which has positive impact on society. We also have a section to discuss the broader impacts.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model and datasets we used are open-sourced, and we strictly follow their terms once the terms are carried out.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: material.
We have cited the creators in the main part of the paper and the supplement
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should citep the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: In the abstract section, we provide the GitHub link to open access to our code and data.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: SelectIT does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Selectl'T does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "DbS4P6Avtu", "reviewer": "Reviewer_KKrK", "summary": "The paper introduces a data selection approach to instruction tuning (IT) for large language models (LLMs) by leveraging the intrinsic uncertainties within the models themselves. This method, termed SelectIT, aims to enhance IT data selection without the need for additional external resources, making it more cost-effective and accessible.\n\nSelectIT utilizes three levels of uncertainty within LLMs—token-level, sentence-level, and model-level—to effectively rate and select high-quality IT data. The paper demonstrates the effectiveness of SelectIT through empirical results, showing improvements in model abilities on the open-instruct benchmark and other domain-specific tasks. Additionally, SelectIT is highlighted as a faster and more resource-efficient method compared to traditional approaches, making it a more practical solution for widespread adoption.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------\nThank you for your reply and I have updated my score accordingly.", "questions": "(1) How the α = 0.2 in the experiments is selected? \n(2) Will optimal K be training dataset dependent?", "limitations": "See more details in Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Originality**: The paper presents a data selection approach to instruction tuning by leveraging the intrinsic uncertainties within LLMs, without relying on additional fine-tuned models or API calls of ChatGPT or Claude.\n\n**Quality**: The methodology has a clear explanation of the multi-granularity self-reflection technique. The empirical results are robust, demonstrating the effectiveness of SelectIT across various benchmarks and tasks.\n\n**Clarity**: The paper is clearly written, with well-organized sections and comprehensive explanations of the proposed method and its implementation.\n\n**Significance**: SelectIT addresses a critical limitation in the field of instruction tuning by removing the dependence on additional models or data, making the approach more cost-effective and scalable. The curated Selective Alpaca dataset offers performance improvements for LLMs, highlighting the practical impact of the research.", "weaknesses": "1. **Reliance on Tuning K**: The method depends on tuning the parameter \\( K \\), which can significantly affect model performance. This dependency introduces additional complexity and may require extensive experimentation to optimize.\n\n2. **Use of Multiple Base Models**: Although the paper claims not to use external resources, it actually relies on multiple different base models to obtain the model-level score. This reliance contradicts the claim of being purely self-reflective and can be confusing, as the title suggests the method is entirely self-contained.\n\n3. **Sentence-Level Quality Evaluation**: The sentence-level quality evaluation requires multiple rating prompts. It is unclear whether other methods, such as AlpaGasus, would achieve similar performance if they also used multiple rating prompts. This comparison is missing and could be critical for understanding the true advantage of the proposed method.\n\n4. **Single Dataset Evaluation**: All evaluations are conducted on a single training dataset, making it questionable whether this method can be generalized to other datasets, such as WizardLM, or multi-turn data, like the ShareGPT dataset. Testing on a more diverse set of datasets would strengthen the paper's claims of robustness and general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review.\n\n>Q1: Reliance on Tuning K: The method depends on tuning the parameter ( K ), which can significantly affect model performance. Will optimal K be training dataset dependent?\n\nThe performance of SelectIT is not critically dependent on the parameter $K$, based on the conclusions of Section 5.1 \"Effect of $K$ \". As illustrated in Figure 2, when $K$ is sufficiently large, such as 5, 7, or 9, the impact of parameter $K$ on LLMs performance is negligible, with performance fluctuations around just 0.3. This is due to the multi-level self-reflection mechanism, which effectively enhances the accuracy and stability of sample selection.\n\nFurthermore, based on Section 5.2, **we also apply the SelectIT to the representative domain-specific task, machine translation (MT),  which reuses the same hyperparameters $K$ and $\\alpha$**. As shown in Table 6, we present outcomes for two distinct language pairs in four distinct directions and, SelectIT can consistently enhance the translation performance of ALMA [1],  which is widely recognized as a leading machine translation LLM.\n\nFinally, referring to the response to Q1 of General Response, we also apply the same hyper-parameters of $K$ on the other datasets such as the WizardLM and larger Orca. The results demonstrate that the same $K$ hyper-parameters of SelectIT still yield consistently strong performance across different datasets. Therefore, extensive experimentation is not required to adjust K hyper-parameters, and it is independent of the training dataset.\n\n[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.\n\n\n>Q2: Use of Multiple Base Models: Although the paper claims not to use external resources, it actually relies on multiple different base models to obtain the model-level score. This reliance contradicts the claim of being purely self-reflective and can be confusing, as the title suggests the method is entirely self-contained.\n\nPlease refer to the Q2 of General Response.\n\n>Q3:  Sentence-Level Quality Evaluation: The sentence-level quality evaluation requires multiple rating prompts. It is unclear whether other methods, such as AlpaGasus, would achieve similar performance if they also used multiple rating prompts. This comparison is missing and could be critical for understanding the true advantage of the proposed method.\n\nThank you for your advice. We will provide a clearer explanation of Sentence-Level Quality Evaluation. To the best of our knowledge, **we are the first one to use the effect of different prompts on LLMs to select high-quality data.** Following Figure 2 and Equation 4, we do more than use K times prompting and averaging the rating score. More importantly, we calculated the standard deviation of the rating score to quantify the model's uncertainty, which is not only the first attempt in our work but also an important contribution to our work.\n\nFor the previous techniques, we compare Selective Alpaca with the official open-source dataset [1] in Table 1, which we think is fair enough. **Additionally, based on the resource consumption we reported in Table 7 of the efficiency of SelectIT, using ChatGPT to K times-ly selects IT data is significant consumption and challenging to implement, which is also proved by recent research** [2].\n\n[1] https://github.com/gpt4life/alpagasus\n\n[2] Liu et al., What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. ICLR 2024.\n\n>Q4: Single Dataset Evaluation: All evaluations are conducted on a single training dataset, making it questionable whether this method can be generalized to other datasets, such as WizardLM, or multi-turn data, like the ShareGPT dataset. Testing on a more diverse set of datasets would strengthen the paper's claims of robustness and general applicability.\n\nSorry for the misunderstanding. **In Section 5.1 of \"Various Domain-specific Tasks\", SelectIT has applied to different datasets such as MT datasets.** As illustrated from Lines 267-272 in the paper, following the configuration of ALMA [1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.\n\n**To further alleviate concerns, we apply SelectIT to other instruction datasets such as WizardLM and larger Orca-GPT4.** For more details, please refer to the Q1 of General Response.\n\n[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.\n\n>Q5: How the $\\alpha$ = 0.2 in the experiments is selected? \n\nThe value of $\\alpha$ is determined through our analytical research on Sentence-level Self-Reflection. To further justify the appropriateness of $\\alpha$ = 0.2, we will conduct supplementary experiments on varying α values of  SelectIT on the Alpaca datasets. For more details, please refer to the Q3 of General Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93kCCexllz", "reviewer": "Reviewer_6ofY", "summary": "This paper presents SelectIT, a new way to improve how large language models (LLMs) follow instructions using their own uncertainty to pick better training data. This method is cost-effective as it doesn't require extra tools or data. The authors developed a new dataset called Selective Alpaca using SelectIT, which significantly improved LLMs' performance. The study suggests that using longer and more detailed data could be more beneficial for training models.", "questions": "\"The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources. \" Clarification?\n\nCould you consider applying the SelectIT method to larger datasets, such as the Orca dataset for ChatGPT with 5 million samples and the GPT-4 dataset with 1 million samples? This would help to evaluate whether your method scales effectively with increased data volume.", "limitations": "the paper lacks a separate section on limitation of the approach.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "SelectIT is a bew and unique method that improves how large language models (LLMs) learn by using the model's own uncertainty to choose better training data. This approach is special because it doesn't need extra tools or data, making it less expensive and easier to use.\n\nThe paper includes detailed studies that show the method works well and supports its findings with strong evidence. SelectIT significantly improves how models perform compared to older methods, showing that it's a big improvement in teaching models.\n\nThe researchers made their methods and data public, allowing others to use and test them. This openness helps the scientific community to work together and trus the results more.\n\nThe paper is also written in a way that's easy to understand, which helps more people learn about SelectIT. It discusses how using longer and more detailed data can make models learn better. Additionally, the paper compares the costs of using SelectIT with using other common tools like the GPT-4 and ChatGPT APIs, showing that SelectIT can save money. This shows that SelectIT is not only effective but also cost-efficient for training models.", "weaknesses": "The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review.\n\n>Q1:  The paper states initially that it does not use extra resources for its new method. However, it mentions later that they used different sizes of LLaMA 2 models to help choose the best instruction data. This contradicts their first claim about not needing extra resources.\n\nThe 'Main Results' and 'Effect of Different Reflection Strategy' sections highlight the effectiveness and importance of Sentence-R in SelectIT. Serving as a highly adaptable method, SelectIT can handle varying quantities of base models. In instances where there is a single base model, SelectIT leverages Sentence-R for data selection without requiring additional models or data. However, when multiple base models are present, SelectIT appropriately allocates weights to the result of each model based on the number of model parameters for data selection, thereby boosting the algorithm's robustness. \n\nIn summary, SelectIT possesses the ability to use the base model directly for data selection without needing extraneous resources and it could achieve comparable performance with only using one base model.\n\n>Q2:  Could you consider applying the SelectIT method to larger datasets, such as the Orca dataset for ChatGPT with 5 million samples and the GPT-4 dataset with 1 million samples? This would help to evaluate whether your method scales effectively with increased data volume.\n\n**In Section 5.1 of \"Various Domain-specific Tasks\", SelectIT has applied to different datasets such as machine translation datasets.** As illustrated from Lines 267-272 in the paper, following the configuration of ALMA [1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.\n\n**To further alleviate concerns, we apply SelectIT to other instruction datasets such as WizardLM and larger Orca-GPT4, while keeping the hyperparameters consistent.** The results are shown in the following table.\n\n\n|          | Data Size | MMLU | BBH  | GSM  | Tydiqa | CodeX | AE   | AVG  |\n|:---------------------:|------|:----:|:----:|:----:|:------:|-------|------|:----:|\n|       WizardLM   | 143,000     | 43.8 | 37.8 | 10.0 |  41.2  | 25.2  | 35.3 | 32.2 |\n| WizardLM w/ SelectIT |  28,600 | 45.1 | 40.1 | 11.0 |  43.1  | 27.5  | 34.7 | **33.6** | \n|       Orca-GPT4    |  994,895    | 40.1 | 35.6 | 13.0 |  46.0  | 23.3  | 38.1 | 32.7 |\n| Orca-GPT4 w/  SelectIT |198,979 | 43.9 | 38.7 | 16.5 |  42.0  | 27.7 | 37.4 | **34.4** |\n\nBased on various task types, including instruction tuning (IT),  machine translation (MT), and diverse instruction datasets such as Alpaca, WizardLM, and the larger Orca-GPT4, extensive experimental results have robustly validated the general applicability and robustness of SelectIT.\n\n[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.\n\n\n>Q3: The paper lacks a separate Section on the limitations of the approach.\n\nSorry for the misunderstanding. We have already provided the \"Limitations\" Section in Appendix A.1 of our original submission. Later, we aim to integrate this section into the main text to provide a clearer understanding of SelectIT to audiences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MZxlAOFNXW", "reviewer": "Reviewer_CKC2", "summary": "In this paper, authors leverage LLMs to score the training data in terms of the token and sentence and model. They train various models on their select models to show the effectiveness on various benchmarks. Their main contribution is to apply foundations to judge the quality of training data from different granularity without any external resources.", "questions": "Please refer to the  above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Their idea is easy to understand and the writing is pretty clear to me.\n2. They conduct a lot of experiments on several benchmarks to show the effectiveness.", "weaknesses": "1. From the table 3, it seems sentence-R works the best so what if we trained model on the data just from the sentence-R?  BTW, what is the ID 6, 7, 8 means in that Table?\n2. My main question is that do we really need all three types selection models, can one or two of them already achieve great performance?\n3. What is the overhead of this data selection model? Like how long does it take to finish the process on 10K dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review.\n\n>Q1: From the table 3, it seems Sentece-R works the best so what if we trained model on the data just from the Sentece-R? BTW, what is the ID 6, 7, 8 means in that Table?\n\nThank you for your suggestion. Actually, in Table 1, we have provided the experiment that trained the model on the data from Sentece-R, referring to System 7 and System 16. The results in Table 1 indicate that Sentece-R achieved the best outcomes among three different level methods (Token-R, Sentece-R, Model-R) of SelectIT,  which aligns with the conclusions in Table 3 and further dissects the key aspects of SelectIT performance.\n\nAdditionally, IDs 6, 7, and 8 in Table 3 correspond to the system of the same IDs in Table 1, which means using the same methods to select data. We will provide clearer explanations for Table 3 in the next version.\n\n>Q2: My main question is that do we really need all three types selection models (method), can one or two of them already achieve great performance?\n\n\nThere might be an ambiguity that the three types are either the three different levels of self-reflection strategies or the three different bases models, we will address both aspects as follows.\n\n- **Different Levels of Self-Reflection Strategies**: \nBased on Table 1 and Table 3, Sentence-R performs better than other select methods, such as Token-R and Model-R,  which is attributed to the use of different prompts to quantify the uncertainty of the LLMs. So the single method, Sentence-R is good enough to select data. Furthermore, we integrate various levels of self-reflection: token, sentence, and model, to form the final SelectIT, achieving the best performance.\n\n- **Different Types of Bases Models**: \nAs you expect, SelectIT still performs well when there are only one or two base models. Based on Table 1 and Table 3, Sentence-R plays the most important role in SelectIT, which only uses the single base model. Model-R, as an extension of SelectIT, allocates weights to the result of each model based on the number of model parameters for data selection, mainly enhancing the robustness of the algorithm.\n\nAdditionally, we will supplement the experiments which using varying numbers of models to select the  instruction data, with the results presented below: \n\n|                    Used Models          | MMLU | BBH  |   GSM   | Tydiqa | CodeX | AE   |     AVG     |\n|:--------------------------------------:|:----:|:----:|:-------:|:------:|------|------|:-----------:|\n|                   None (Baseline)                    | 46.5 | 38.4 |  15.0   |  43.4  | 26.8 | 34.2 |    34.1     |\n|               LLaMA-2-7B               | 47.6 | 39.1 |  15.0   |  46.1  | 28.2 | 35.2 |    35.2     | \n|        LLaMA-2-7B + LLaMA-2-13B        | 47.7 | 38.1 |  15.5   |  47.8  | 28.4 | 35.1 |    35.4     |\n| LLaMA-2-7B + LLaMA-2-13B + LLaMA-2-70B | 47.4 | 40.6 |  16.8   |  47.4  | 29.4 | 35.7 |  **36.2**   |\n\nBased on the above analysis and experimental results, it is evident that SelectIT still works well when there is a single base model, referred to as the Sentece-R. More base models are also beneficial for selecting high-quality data, which can effectively enhance the robustness of SelectIT.\n\n\n>Q3:  What is the overhead of this data selection model? Like how long does it take to finish the process on 10K dataset.\n\nIn Section 5.3, we have elaborated on and compared the efficiency of SelectIT with other methods. Based on the Alpaca dataset, we deployed 4 A800 80G GPUs to select high-quality IT data, which ultimately required **5.8 hours**. In contrast, utilizing the ChatGPT API required significantly more time, **19.07 hours**. This issue is even more pronounced with the more powerful GPT-4, demonstrating the practical utility of SelectIT. For the 10K dataset, keeping the same GPU devices as above, it would necessitate 1.2 hours.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d9aDUp2pa8", "reviewer": "Reviewer_4rCk", "summary": "The paper introduces SelectIT, a novel instruction tuning (IT) method that leverages the intrinsic uncertainty in large language models (LLMs) to select high-quality IT data without requiring additional resources. The authors present a curated IT dataset, Selective Alpaca, derived from the Alpaca-GPT4 dataset using SelectIT. The empirical results demonstrate significant improvements in model performance across various tasks. The paper highlights the robustness and efficiency of SelectIT in enhancing the capabilities of LLMs.", "questions": "- How does SelectIT perform when applied to instruction tuning datasets other than Alpaca-GPT4? \n- Can the authors provide more detailed examples or case studies illustrating the differences between data selected by SelectIT and other methods? And how these different influence the final model performance?\n- How does the performance of SelectIT change with different values of the uncertainty factor (α) ?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-  The approach of using the intrinsic uncertainty of LLMs for IT data selection is innovative and eliminates the need for external resources, making it cost-effective and widely applicable. \n-  The submission is technically sound, with well-supported claims through theoretical analysis and comprehensive experimental results.", "weaknesses": "- Figure 2 is not clear to me. I suggest giving a brief description of each selection stage in the caption for easy understanding.\n- It seems like the foundational LLM's capacity becomes the key for this data selection. In this case, the selection method may not be as effective as other external resource methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review.\n\n>Q1: Figure 2 is not clear to me. I suggest giving a brief description of each selection stage in the caption for easy understanding.\n\nThank you for your advice. Following your suggestion, we will revise the caption of Figure 2 to: `Overall Framework of Our Proposed SelectIT Method. In the Token-level Self-Reflection, we employ the foundation model to rate the IT data from 1 to $K$. For Sentence-level Self-Reflection, we leverage the uncertainty of varied prompts on LLMs to enhance the rating process. In the Model-level Self-Reflection, we harness uncertainty among different LLMs to facilitate a collaborative decision-making process in selecting IT data. Finally, different levels of self-reflection are reasonably combined into SelectIT, which can effectively select high-quality instruction tuning data without relying on additional resources.`\n\n>Q2:  It seems like the foundational LLM's capacity becomes the key for this data selection. In this case, the selection method may not be as effective as other external resource methods. \n\n- **Comparison with other external resource methods:** \nIn Table 1, we compare SelectIT with various other external resource methods such as AlpaGasus [1], Q2Q [2], and Instruction Mining [3]. SelectIT demonstrates superiority in the abundant experiments including six different tasks and varying model parameter sizes. Furthermore, as shown in Table 7, SelectIT requires less time and fewer resources compared to AlpaGasus or its variant. Therefore, we have grounds to believe that leveraging the capacity of LLMs without external resources is sufficient for selecting high-quality samples.\n\n- **Emergence of more powerful LLMs:**  Based on the main result of Table 1, the LLaMA-2 series base models we used have exhibited sufficient capability in selecting higher-quality samples. With the continuous development of our LLM community, more powerful open-source LLMs are emerging, such as LLaMA-3.1, Mistral-7B-v0.2, and Qwen2 etc. Given that the performance of SelectIT is intrinsically connected to the capacities of foundation LLMs, its efficiency is not only unrestricted but also anticipated to enhance concurrently with the advancement of LLMs in the future.\n\n[1] AlpaGasus: Training A Better Alpaca with Fewer Data. ICLR, 2024\n\n[2] From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning. NAACL, 2024\n\n[3] Instruction Mining: Instruction Data Selection for Tuning Large Language Models. arXiv, 2023.\n\n>Q3:  How does SelectIT perform when applied to instruction tuning datasets other than Alpaca-GPT4?\n\n**In Section 5.1 of \"Various Domain-specific Tasks\", SelectIT is also applied to different datasets such as machine translation datasets.** As illustrated from Lines  267-272 in the paper, following the configuration of ALMA[1], we opt for a total of 30K training examples for representative language pairs and utilize SelectIT to select a subset of 6K data for comparative training. As Table 6 demonstrates, SelectIT can invariably boost the translation performance of ALMA [1] using fewer yet higher-quality training data.\n\n**To further alleviate concerns, we apply SelectIT to other instruction tuning datasets such as WizardLM and larger Orca-GPT4,** while keeping the hyper-parameters consistent. For more details, please refer to the Q1 of General Response.\n\n[1] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models. ICLR, 2024.\n\n\n>Q4 :  Can the authors provide more detailed examples or case studies illustrating the differences between data selected by SelectIT and other methods? And how these different influence the final model performance?\n\nIn Figure 5, we have comprehensively discussed the differences in the samples selected by SelectIT and other methods such as AlpaGasus.\n\n- **Length**: Relative to AlpaGasus and the full set of Alpaca, the average sample length of SelectIT is the largest, reaching 241.85. Length is a crucial dimension in assessing sample quality [1]. Longer sentences typically contain more information or procedural steps, which are beneficial in enhancing model capabilities.\n\n- **Calculation**: As shown in Lines 61-62 and Figures 5-6,  one of the primary contributions of SelectIT is that it prefers additional calculation-type IT data which enhances the LLMs' reasoning capabilities, offering a new perspective on the characteristics of optimal IT data. As demonstrated in Table 1, SelectIT achieved optimal results on benchmarks assessing computational and mathematical capabilities. The enhancement in model capabilities further improved the LLMs' coding abilities, ultimately leading to an overall enhancement in comprehensive capabilities.\n\nFurthermore, in Appendix Section A.4, we have presented relevant case studies contrasting SelecIT and AlpaGasus. In samples 1 to 4 of Figure 8, SelectIT demonstrates a preference for instruction-tuning data featuring complex mathematical problems, aligning with the findings of Section 5.4. In contrast, AlpacaGasus usually selects IT data predominantly providing solutions to queries or lacking coherent reasoning in samples 5 to 7, potentially hindering its effectiveness.\n\n[1] Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning. arXiv, 2024.\n\n>Q5:  How does the performance of SelectIT change with different values of the uncertainty factor (α) ?\n\nPlease refer to the Q3 of General Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the insightful comments and constructive suggestions, which will serve to improve the paper considerably. We will attend to all comments to the best extent.

> Q1: Applying the SelectIT on other representative instruction tuning datasets.

To further demonstrate the robustness of SelectIT, we apply it to two additional commonly used datasets: WizardLM and Orca-GPT4. **At the same time, we maintain consistent hyperparameters such as $\alpha$ and $K$ and use the LLaMA-2-7B as our base model.**  The results are presented in the following table.

|          | Data Size | MMLU | BBH  | GSM  | Tydiqa | CodeX | AE   | AVG  |
|:---------------------:|------|:----:|:----:|:----:|:------:|-------|------|:----:|
|       WizardLM   | 143,000     | 43.8 | 37.8 | 10.0 |  41.2  | 25.2  | 35.3 | 32.2 |
| WizardLM w/ SelectIT |  28,600 | 45.1 | 40.1 | 11.0 |  43.1  | 27.5  | 34.7 | **33.6** | 
|       Orca-GPT4    |  994,895    | 40.1 | 35.6 | 13.0 |  46.0  | 23.3  | 38.1 | 32.7 |
| Orca-GPT4 w/  SelectIT |198,979 | 43.9 | 38.7 | 16.5 |  42.0  | 27.7 | 37.4 | **34.4** |

Based on three diverse instruction datasets such as Alpaca-GPT4, WizardLM, and the larger Orca-GPT4, extensive experimental results robustly validate the general applicability and robustness of SelectIT.

> Q2: The use of extra models.

The 'Main Results' and 'Effect of Different Reflection Strategy' sections highlight the effectiveness and importance of Sentence-R in SelectIT. Serving as a highly adaptable method, SelectIT can handle varying quantities of base models. In instances where there is a single base model, SelectIT leverages Sentence-R for data selection without requiring additional models or data. However, when multiple base models are present, SelectIT appropriately allocates weights to the result of each model based on the number of model parameters for data selection, thereby boosting the algorithm's robustness. 

In summary, SelectIT possesses the ability to use the base model directly for data selection without needing extraneous resources and it could achieve comparable performance with only using one base model.

> Q3:  How does the performance of SelectIT change with different values of the uncertainty factor (α)?

We will supplement the experiments about different $\alpha$ of SelectIT on the Alpaca datasets, which use the LLaMA-2-7B as the foundation model and keep other parameters unchanged. 

|                 | MMLU | BBH  | GSM  | Tydiqa | CodeX | AE   |   AVG    |
|:---------------:|:----:|:----:|:----:|:------:|-------|------|:--------:|
| $\alpha$  = 0.2 | 47.4 | 40.6 | 16.8 |  47.4  | 29.4  | 35.7 | **36.2** |
| $\alpha$  = 0.4 | 47.9 | 39.4 | 15.5 |  46.5  | 29.4  | 35.8 |   35.8   | 
| $\alpha$  = 0.6 | 47.8 | 39.8 | 16.5 |  45.6  | 29.1  | 35.1 |   35.7   |
| $\alpha$  = 0.8 | 47.6 | 36.4 | 16.5 |  43.6  | 26.7  | 35.4 |   34.4   |


α is the uncertainty factor used to balance the mean and standard deviation of the score from token-R. As α increases, Sentece-R tends to focus more on the uncertainty of LLMs, neglecting the average score which fundamentally reflects the sample quality. Therefore, α = 0.2 is an appropriate value to effectively balance sample quality and model uncertainty. We will add the results and discussion in the next version.

Meta-review: This paper introduces a novel approach for instruction tuning of LLMs that leverages the model's intrinsic uncertainty to select high-quality training data without relying on external resources. The reviewers generally agree that the approach is innovative and potentially impactful. The strong empirical results across various benchmarks also support its effectiveness. The reviewers are largely positive on this submission. The authors also have addressed many concerns in their rebuttal, and provided additional experiments on diverse datasets and clarified the method's adaptability to different numbers of base models.
Decision: Accept (poster)