AlpaGasus: Training a Better Alpaca with Fewer Data
Authors: Lichang Chen
Shiyang Li
Jun Yan
Hai Wang
Kalpa Gunaratna
Vikas Yadav
Zheng Tang
Vijay Srinivasan
Tianyi Zhou
Heng Huang
Hongxia Jin
Venue: ICLR 2024 poster (2024)
Keywords: data selection; large language model; instruction tuning;
Primary area: generative models

Abstract: Large language models~(LLMs) strengthen instruction-following capability through instruction-finetuning (IFT) on supervised instruction/response data. However, widely used IFT datasets (e.g., Alpaca's 52k data) surprisingly contain many low-quality instances with incorrect or irrelevant responses, which are misleading and detrimental to IFT.  In this paper, we propose a simple and effective data selection strategy that automatically identifies and removes low-quality data using a strong LLM (e.g., ChatGPT). To this end, we introduce Alpagasus, which is finetuned on only 9k high-quality data filtered from the 52k Alpaca data. Alpagasus significantly outperforms the original Alpaca as evaluated by GPT-4 on multiple test sets and the controlled human study. Its 13B variant matches $>90\%$ performance of its teacher LLM (i.e., Text-Davinci-003) on test tasks. It also provides 5.7x faster training, reducing the training time for a 7B variant from 80 minutes (for Alpaca) to 14 minutes \footnote{We apply IFT for the same number of epochs as Alpaca(7B) but on fewer data, using 4$\times$NVIDIA A100 (80GB) GPUs and following the original Alpaca setting and hyperparameters.}.  In the experiment, we also demonstrate that our method can work not only for machine-generated datasets but also for human-written datasets. Overall, Alpagasus demonstrates a novel data-centric IFT paradigm that can be generally applied to instruction-tuning data, leading to faster training and better instruction-following models.

Published as a conference paper at ICLR 2024
ALPAGASUS: TRAINING A BETTER ALPACA WITH
FEWER DATA
Lichang Chen*i,
C Shiyang Li " Jun Yan', Hai Wang 1, Kalpa Gunaratna', Vikas Yadav Zheng Tang', Vijay Srinivasan', Tianyi Zhou', Heng Huang', Hongxia Jin f University of Maryland, College Park Samsung Research America University of Southern California (bobchen, tianyi, heng) @umd.
(shiyang. h. h.wang2, k. k.gunaratna, vikas.y, theng.tang, v. vrsiiniasan, hongxia, jintesannnneeeeeeeeeee @sarsung. com yanjunêuse.edu
ABSTRACT
Large language models (LLMs) strengthen instruction- soweuttiionoooowiinng capability through (IFT) on supervised instructiio/responee data. However, widely used IFT datasets (e.g.. ALPACA'S 52k data) surprisingly contain many low- quality instances with incorrect or irrelevant responses, which are misleading and detrimental to IFT. In this paper, propose a simple and effective data selection strategy that automatically identifies and filters out low-quality data using a strong LLM (c.g.. ChatGPT). To this end, we introduce ALPAGASUS, which is finetuned on only 9k high-quality data filtered from the 52k ALPACA data. ALPAGASUS significantly outperforms the original ALPACA evaluated by GPT-4 multiple test sets and the controlled human evaluation. Its 13B variant matches > 90% performance of its teacher LLM (i.e., Text-Davinci-003 generating the 52k data) on test tasks. It also provides 5.7x faster training, reducing the training time for 7B variant from 80 minutes (for ALPACA) 14 minutes Moreover, the experiments prove the efficacy of our method across diverse datasets, base models, and I.I. filters. Overall, ALPAGGASU demonstrates novel data-centric IFT paradigm that can be generally applied to instruction-tuning data, leading to faster training and better instruction-followinn models. Our project page is available at: git.ub..h.......g.. wwchhuu...oo/AA.. io/Al
INTRODUCTION
Instruction fine-tuning (IFT) Longpre et al. 2023) has been recently applied as an essential con- tinual training stage for pre-trained large language models (LLMs) achieve instruction-oolowwii capability (Ouyang et al. 2022b Chen et al.) 2023b), which is often attributed to aligning the models' behavior with diverse 202T). The recent series of open-sourced instruction-tundr models (Taori et al.) 2023 Xu et al. (023) reveal that the alignment of better IFT data could result in better wwwrrructooofoollwwwai
of human instructions and responses (Taori al. 2023 Askell al.)
skills. For
example, GPT-4-LLM (Peng et al. 2023) (with GPT-4 (OpenAl 2023b) as its teacher) exhibits better reasoning and math ability than ALPACA (Taori et al. 2023) (with Text-davinci-000 as its teacher),
though they share the same base model LLaMA ('ouvrron et al. 2023). demonstrating the importance
of data quality.
Although stronger teachers can usually bring further improvement by providing better IFT data, their responses inevitably include incorrect or irrelevant answers to the corresponding instructions (see examples in Fig. which can be misleading or detrimental to IFT. Moreover, these data also increase unnecessary training costs. Alpaca-cleaned is the pioncer of filtering bad data in ALPACA dataset though it requires humans fully involved in examining and filtering the data. Nonetheless, how to automatically filter out poor-quality data from IFT datasets has not been investigated yet. A primary
Eqqual Contribution, This work was done when Lichang Chen and Jun Yan interned at Samsung Research America.
'We apply IFT for the same number of epochs as ALPPACA(7B) hut on fewer data, using 4x NXNIIIIA A100 (80GB) GPUs and following the original ALPACA setting and hyperparameters.
https: // thub. com www../oo//gg...eeeeee se/ /ttpprrrss////ppcaaaaaaaaaa pacaaaaaaaaaaaaaaaaaaaaaa eaned/

---

Published as a conference paper at ICLR 2024
bottleneck is that rating the data quality usually requires expensive human labor but still may not be accurate for IFT because stronger teachers are more powerful in generating eloquent but incorrect more subtle to detect by humans. When considering datasets crafted by humans,
responses that such as the Dolly dataset (Dolly) 2023). assessing quality becomes even more intricate, given that responses stem from seasoned writers.
This paper aims to bridge the gap by proposing a novel data-filterngg strategy for IFT that is efficient, automatic, and accurate. Specifically, we design prompt applied to powerful LLM (e.g.. ChatGPT) for evaluating the quality of each (instruction, input, response) tuple and then filter out the ones with scores lower than threshold. By applying this filter the 52k data used to train ALPACA, we find that majority of the data suffer from low low-quality issues. Using the LLM filter, IFT on a much smaller but carefully filtered subset of 9k data produces a much better model, i.e., ALPAGASSS, than the original ALPACA, shown in Fig. 1 following exactly the same training configuration of ALPACA. This also reduces the training time from 80 minutes to merely (80GB) GPUs. Moreover, validate the versatility of our method, demonstrating its effectiveness on a range of datasets(e.g)) g.,. Dolly, Alpaca, GPT4LLM), base models(e.g.. LLaMA-I and LLaMA-2), and LLM filters(e.g.. ChatGPT and Claude-2). This discovery is inspiring, it shows that the data quality in IFT can outweigh the quantity. In addition, this shift towards prioritizing data quality presents a new and more efficient paradigm that generally improve the fine-tuning of LLMs.
minutes on 4x NVIDIA A100
Our experiments include comprehensive eval- uations for ALPAGASUS, incorporating free-form instruction evaluation, various bench- marks, and human studies. We select four dif- ferent
Four Bestsets-e-Vicnt
-i Koala
WizardLM
Self-nstructt
AlpaGaous Wining Store
1.15
test sets for evaluat- ing instruction-follett following capability, including the ones used by WizardLM (Xu et al.] (023).
Vicuna (Chiang et al.) 2023), Koala (Geng et al.] 2023). and Self- Ielf-unstruct (Wang et al. 2022). Given the notable advantages that GPT- 4 judge could match with both the controlled and crowdsourced human preferences (> 80% agreement) (Zheng al.) 2023), we employ GPT-4 as our judge for the major evaluations. In the 7B and 13B model comparisons, ALPAGA- SUS performs significantly better than ALPACA on all four test sets. To address potential con- cerns regarding biases in model-based evalua- tions, we conduct human studies and benchmark evaluations, both of which corroborate the su- periority of our model compared to baseline counterparts. Furthermore, we present a fine-grained evaluation of ALPAGASUS on individual tasks including Generic, Roleplay, Knowledge, and Com- monsense from the Vicuna test set. The results indicate ALPAGASUS exhibits advantages on a majority of the tasks.
Best Size
0.7
Training Data
Figure 1:
Performance of ALPAGASUS on four test sets when increasing its finetuning data, where the winning score is #Win-#Loes #Testset #Win + #Tic + #Lose to be the test set size and #Win/#Tie/HL
with
Pllestet
be the number of sam- ples on which ALPAGASUS wins/ties/loses com- pared ALPACA 52K.
To sum up. our Kata-filtering approach exhibits significant benefits in terms of scalability and automation. We also demonstrate that prudent management of training data quality can lead to substantial performance improvement and computation savings of IFT. In addition, our data selection and evaluation strategies generalize to other instruction finetuning datasets and LLMs, thereby paving the way for a promising new research trajectory aimed at pragmatic LLM deployment.
2
METHODOLOGY
2.1 OVERVIEW
Unlike the recent work (Zhou et al. 2023), which relies on human labor to curate 1K high-quality instruction data that leads to better finetuned model, we aim to avoid the expensive and time- consuming human annotations. Hence, we exploit the potential of strong LLMs to be auto-graders of the training data and then filter out the data with lower scores.
In particular, we prompt a strong API LLM, i.c., ChatGPT, to produce a score for each triplet of (instruction, input, response). The prompt is given in Fig. 3 where "dimension" denotes
2

---

Published as conference paper at ICLR 2024
Training
Included
Included
Included
Instruction: Classity the tiem as either
Rewiite the following omiiting the pranouns. Inpur: She told us she was busy.
Instruction:
Instruction: Translate the pirase "Bonne chance" into English
animal vegetable.
Iaput: Banana
Response: Good Luck.
Respurse: Animal: No, it's vegetable.
Response: Told busy.
ChatGPT-basee
Respunse Quality Evaluator
Score: resporse partially the instraction umnitting the pronouns, but does not provide complete and The response lacks
Score 5.0: The Al assistant provided a accurite response the given instraction and input. ("onnn chance!" a French phrase that translates 'Good lock'" in English, and the assistant provided the correct translation.
The assistunt's response incomeet. A banana a fruit, which is type of plant based food, making it vogetable
colireett
subject and verh, making it difficult to understand intended meaning. correct response woald be "Told busy"
Discarded X
Discarded
Included
/ Less Data
Training
/ Faster Training
/ Stronger Performance
The fine-tuning pipeline ALPAGASUS. We prompt ChatGPT as our auto-grader to score each training triplet a scale of 0 to 5. We then use the exact same instruction fine-tuning script of ALPACA to train ALPAGASUS on the filtered data with scores higher than a threshold.
Figure 2:
System Prompt:
We would like request your feedback the performance of Al assistant in response instruction and the given input displayed following.
Instruction: [Instruction]
Input: [Input]
Response: [Response]
User Prompt:
Please rate according to the [dimension] of the response to the instruction and the input. Each assistant receives scale of where higher score indicates higher level of the [dimension]. Please output single line containing value indicating the scores. the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias.
Figure 3: Prompt Pa to ChatGPT for rating and filtering training data in Eq.
user-prefereed property such as helpfulness and accuracy. We then only select the triplets with scores higher than certain threshold to fine-tune LLaMA-serte model following existing IFT pipeline. Fig. illlustrates the data selection and training pipeline.
2.2 DATA RATING AND FILTERING
Given IFT dataset V of triplets =iinttruction, input, response) with E V and an open-sourced LLM 8 (e.g., LLaMA), let dy denote the finetuned on V. our overarching goal is to select a subset S V such that IFT on S results in better model 0g than 0v.
In order to select S' from V, we prompt an API LLM G(.) (e.g., ChatGP an auto-grader rating each sample x V by score G(x,pG) wherein pe is the rating prompt in Fig. then select J;; whose score is above a certain threshold r, i.e.
S A (ar EV G(z,pe) 2 r).
(1)
We achieve 0s by finetuning on using an existing IFT framework.
We also claude-2 response quality evaluator, which be found in Appendix A.2
3

---

Published as a conference paper at ICLR 2024
2.3 ALPAGASUS: 9K TRAINING DATA FILTERED FROM ALPACA
For "dimension" in the rating prompt PG shown in Fig. B given that "accuracy" closely aligns with hu- man expectations of LLMs' responses, we designate "accuracy" as the dimension for rating purposes Correspondingly, we establish T in Eq. (I) as an accuracy threshold for the subsequent experiments. The distribution of scores in relation to the 52k Alpaca dataset presented Fig.
In particular, we choose the threshold т = 4.5 according to the score histogram. For the ALPACA dataset V with 52,002 samples this filtering criterion leads to a subset 5 of 9,229 samples
Score Distribution
32000
30240
16000
3 EXPERIMENTAL SETUP
10811
9218
8000
3.1 FREE-FORM INSTRUCTION EVALUATION
1550
Most instruction-tune sunion models are evaluated on one test set that might not cover sufficient diverse instructions and thus leads to a risk of biased evaluation (Chia et al.] 2023). To conduct a holistic evaluation of ALPAGASUS, we curate our test sets from Self-instruct (Wang et al., 2022), Vicuna (Chiang et al.)
3.5
4.5
Count
Figure 4: Histogram of Scores (Al- paca Dataset).
2023), WizardLM (Xu et al.] 2023). and Koala (Geng et al.) (023). which together can cover more types of instructions and reduce the evaluation bias. Details of these four test sets are provided in T
3.2 BASELINE MODELS
We compare our ALPAGASUS with the following four recent LLMs.
Test Samples
Category
Koala
180
ALPACA TTori al. 2023) is an open-sourced model developed by Stanford University through IFT of LLaMA on a training dataset of 52,002 (instruction, input, re- sponse) samples with the responses generated by Text- Davinci-003 (teacher).
Vieuna
80
WizardLM
218
Self-Instruct
252
Table 1: Four sets used in this paper.
TEXT-DAVINCI-003 is an OpenAl LLM trained with an increased emphasis contextual understanding and response accuracy. Its proficiency in capturing complex linguistic patterns makes it powerful teacher LLM for generating high-quality training data for finetuning LLMs such as ALPACA.
CHATGPT (OpenAl 2023a an Al chatbot finetuned via reinforcement learning with human feedback (RLHF), It cxhibits exceptional capability across wide range of tasks and might be the most popular chatbot recently. Hence, would be interesting to study to what extent ALPAGASUS can match its performance.
CLAUDE (Bai et al. 2022) is an Al chatbot developed by Anthropic. It was finetuned by RLHF to align with humans' preference on three dimensions, i.c., helpful, honest, and harmless. We use Claude- v1.1 for comparison, which is comparable to ChatGPT on the AlpacaEval (Li et al. 2023).
3.3 EVALUATION METRICS
The evaluation of the dostruction-folloiing capability of LLMs is usually challenging due to the existence of multiple eligible responses to one instruction and the difficulty of reproducing human evaluations. In light of the recent advancements automated evaluation (Dubois al., [023) Zheng al. 2023; Chiang et al.) (202)), which offer superior scalability and explainability than human studies, we also apply an API LLM J() (e.g., GPT-4) as the judge to evaluate Os and compare it with 8y. In particular, we apply J(.) compare the responses of fs and 8y each instruction drawn from a test set D. Let F(2; By) and F(2; 0s) denote the two models' responses to instruction E D,
Wee defer experiment of other dimensions, e.g., helpfulness, to the Appendix A.5
$52k denotes 52002 samples from the original Alpaca training set and 9k represents 9229 data samples. (either randomly sampled or filtered in our experiments)
4

---

Published as a conference paper at ICLR 2024
the judge outputs score for each response and we aim achieve higher score 8s, i.e.,
(F 8g)) 2 ((F(e:
for most 2 E D. our experiments, we include both models' responses in the input to the judge (e.g., GPT-4), followed by instruction the judge, which aims to rate the responses with a score between and 10. Details of the input and prompt to the judge can be found in Appendix
Since there cxists position bias within LLM judges, which refers to a phenomenon where LLM judges have tendencies prefer specific positions over others (Wang et al. 2018 Ko al. 2020 Wang et al. 2023), to mitigate it, we try both orders (i.e., placing ALPAGASUS ASUS'S response betore/atter the baseline model's s response) and define the final judge of "Win- "Win-Tie-Lose
(1: Win: ALPAGASUS wins twice, or wins once and draws once. (2) Tie: ALPAGASUS draws twice, or wins once and loses once. (3) Lose: ALPAGASUS loses twice, or loses once and draws once. To avoid cut-off responses, we allow models to generate up to 1024 tokens. For ChatGPT, Claude, and Text-Davinci-003, we set the temperature to 0.0, respectively, to reduce randomness and ensure a fair comparison.
EXPERIMENTAL RESULTS
4.1 QUALITY MATTERS MORE THAN QUANTITY
7B: AlpaGasus-9K vs. Alpaca-52k
13B: AlpaGasus-9H vs. Alpaca-52k
Vieara
(looCoorr((9))) wine
70
66
A2poca/52k) wins
Koula
67
49
Woanf
WizariLM
Intruct
Sef:
86
76
Instruct
Self-
8BI
X1
Figure 5: Main results: comparing ALPAGASUS and ALPACA on their 7B and 13B models. ALPAGASUS-9% achieves much better performance than ALPACA-52K on all four test sets: Vicuna, and WizardLM.
Koala,
AlpaGasus-9k vs. Alpaca-52k We compare ALPAGASUS and ALPACA on two sizes of models Fig. [ They only differ in the training data: ALPACA uses all the 52k data while ALPAGASUS only uses 9k data selected from the 52k. Their hyperparameters and training scripts are the same. As shown in the evaluation results, ALPAGASUS significantly outperforms the original ALPACC across all four test sets. Moreover, when using LLaMA-2 as the base model, we observe consistent outcomes (See Appendix A..3). This consistency underscores the universality of our data filtering method, irrespective of the model choices. These findings also confirm that our training data selection approach leads to superior performance even when the selected training data are only 17.75% the original dataset.
7B: AlpaGasus-9k vs. Alpaca-9k- Ra-nado
13B: AlpuGasus-9k vs. Alpaca-ll--aaeee 9K-Randmm
AlpuCasuu
Vicuna
Vicuna
Tic
Appca
Koola
Kools
WinarLMM
WizerLM
Self-
Self-
Instrret
Figure 6: Comparing ALPAGASUS with LLaMA finetuned on randomly selected data.
TTo address potential concerns regarding bias in the evaluation prompts, also present results of using alternative evaluation prompts in Appendix A.

---

Published as a conference paper at ICLR 2024
Quality-Guided Filtering Random Filtering To investigate the efficacy of our data selection strategy, we compare ALPAGASUS with LLaMA models fine-tuned on a randomly sampled subset of the ALPACA 52k data, denoted by initial model (i.e., I.aMA) and are then finetuned on the same number samples (i.e., 9k). They only differ in terms of the data selection criteria. In Fig. 6 compare the types of models under two model sizes, i.c., 7B and 13B. ALPAGASUS-95 significantly outperforms ALPACA-9K-rande showing the high quality of selected data and their importance to the performance of IFT.
in Fig- 6 Both models start from the same
4.2 How MUCH DATA SHOULD BE FILTERED?
Threshold data filtering. Eq. (T. we select data with score 7 and we set 4.5 in our main experiments, which results in 9k out of the 52k data to finetune ALPAGASUS, To study the impact of the threshold T on IFT, we com- pare ALPAGASUS with LLaMA finetuned 39k data selected by applying a lower threshold of 4.0. We report the comparison results in Fig-Z When tested on the Koala and Wizardl .M test sets, ALPACA-39K model outperforms the original ALPACA-52K model. However, when using the Vicuna and Self-Instruct as test sets, ALPACA-39K does not exhibit advantages over the original ALPACA-52 model. Hence, loose criterion (a lower threshold) includes more data in the selected data and model with comparable performance the original ALPACA. However, it still performs poorer than ALPAGASUS trained on much fewer but higher-quality data, indicating the negative impact of luwlltlytty data to IFT.
Alpaca-39k- 39keae-----FFteeeddd vs. Alpaca-52k
Vicana
31
19
Alpaca wins
Tic
Koala
51
70
Alpacs-52k
Wiandd.M M
77
Self-
Figure 7: Comparing ALPACA-7B (39k data) with ALPACA-7B (52k data).
AlpaGasus trained on 3k/6k/9k selected data. On the other hand, high-quality data show positive impact on IFT. To verify this, we randomly draw 3k and 6k data from the 9k data selected for training ALPAGASUS and finetune two variants of ALPAGASUS from LLaMA using the same training script. Fig. [ reports the evaluation results of these variants: ALPAGASUS trained on 9k data performs the best all four test sets, indicating that more high-quality data leads to better IFT models.
AlpaGasus-7BBPe
Alpaca- (B6k)
AlpaGasus-7B(9% vs. Alpaca-7B(3k)
Vicuna
22
AlpaGasus-
Vicuna
Alpagaous 78194) wins
Kozla
59
Alpaco- 7elocc--81100 wins
Koala
78
57
45
Alpace-78) wins.
WizarLM
Instract
Sclf-
96
82
74
Self- Instract
117
69
66
Figure 8: Comparing models finetuned on 3k/6k/9k high-quality data (3k and 6k data are randomly drawn from the 9k data selected for ALPAGASUS).
Minimum training data for AlpaGasus to match the performance of Alpaca. According to Fig. D ~6k high-quality data suffices to finetune LLaMA achieving similar performance as the original ALPACA.
4.3 HUMAN STUDY
We further undertake human studies by enlisting three participants tasked with labeling the ques- tion/answer pairs. To be specific, we select 40 prompts from each test set, resulting in a total of 160 prompts. These are then presented to the participants alongside the corresponding responses generated by both ALPAGASUS- and Alpaca-13B. The final answers are determined by majority voting. There 63/160 wins for ALPAGASUS- -3BB 64/160 ties and 33/160 loses, which indicates the superiority of our ALPAGASUS. Comprchensive results on each test set and user guidelines could be found in Appendix

---

Published as a conference paper at ICLR 2024
AlpaGasus-133 138 Davinci-003
AlpaGasus-139 vs. Claude
AlpaGasus-139 vs. ChatGPT
AppaGanu-138
-Txi-Daaiini000
- AlgoCinnss-133
-Clande
ChanGPT
Georlo
Gener
Krowladge
Writy
Eawwlaage
Kanskdaz
Bokpley
Bolobay
Roleyly
Cudia
Cammur
Counatfatal
Texi
Teni
Figure 9: ALPAGASUS-I32 13B vs. Davinci-003, Claude, and ChatGPT, ALPAGASUS achieves average 90.1% capacity of Davinci003, 81.2% of Claude and 78.4% of ChatGPT.
4.4 COMPARISON WITH CHANGPT//LLAUDE//AVVVNGCIO
In Fig. 9 we compare ALPAGASUS with text-Davinci-O0 Davinci-003, ChatGPT, and Claude. The results show that ALPAGASUS-I3B achieve 2 90% capacity of its teacher model, text-Davinci-003, which is used to generate the ALPACA-52K instruction data.
4.5 BENCHMARK PERFORMANCE
Following InstructEval (Chia et al. 2023 we also evaluate our models on benchmark datasets,
ie.. MMLU (Hendrycks et al. 2020). DROP (Dua et al. 2019) Humaneval (Chen et al. 2021]
BBH (Suzgun et al.) 2022). to evaluate the models' performance. The details of the benchmark setting can be found in Appendix B Benchmark results of our ALPAGASUS are shown in Table 2 where higher values indicate better performance. ALPAGASUS-77 7B, 13B show superiority on the 3/4 datasets, which demonstrates the effectiveness of our filtering algorithm. Another interesting finding is that the models trained with our filtered data can be better on all the benchmarks than training with randomly selected data
Datasets
7B(Kkranddom)
7B(9k)
7B(52k)
13B(Ok-random)
13B(9k)
13B(52k)
BBH 31.89
33.76
33.01
38.60
38.92
38.67
Drop
25.88
26.03
25.87
33.40
34.4
33.84
Humaneval
11.59
12.20
11.69
15,24
15.86
15.74
MMLU
36.93
38.78
40.86
44.98
46.12
47.89
Table 2: The benchmark results of filtering the Alpaca dataset.
5 HUMAN-WRITTED INSTRUCTION SET FILTERING
In addition to filtering machine-generatedd datasets, our approach is capable of filtering human- written datasets. Specifically, we investigate the Databricks-dolly. 15k dataset (Dolly 2023), a seminal collec- tion of 15,000 high-quality numan-generatee prompt/response pairs. Notably, this unparalleled dataset a product of the collective efforts of more than 5,000 Databricks contributors and the included prompts and responses human cognition, covering activities from inventive brainstorming to succinct summarization.
than just simple text; they embody comprehensive spectrum of
We also applied threshold of 4.5 for data filtration, resulting in a filtered dataset of 2,996 samples. (Score distribution can be found in Appendix B A comparison between the 7B/13B I.LaMA trained on our filtered 3k dataset and the one trained on the entire Dolly 15k dataset illustrated in Fig. I0 and Fig. 21 Our evaluation suggests that the model trained on our filtered data exhibits superior performance, thus underscoring the efficacy of our filtering method on human compose datasets. Comprehensive details regarding training hyperparameters are provided in the Appendix
"We observe similar performance gains of the 7B model on Dolly, and our 13B (3k) model consistently outperforms haselines, i.c., 1338(andom-3k) and 13B())k), all four benchmark datasets, which deferred the Appendix B
*The result in Appendix A.4 (GPT4LLM dataset) shows the potential of applying our ChatGPT-based response quality evaluator filter GPT-4's responses, which considered the most powerful model.
7

---

Published as a conference paper at ICLR 2024
Dolly-7B(3k) wins
Dolly-78[34)
Vicuna 40
13
29
The
Tie
Dolly 7B(15K) wins
Dolly-78j 7aa77a///aaaaeee (k-random) wins
Koola
73
42
65
83
WizarL. M
WizurdL
98
46
Instrct
107
57
B8
115
Figure 10: Comparing models finetuned on filtered 3k data and original Dolly 15k data.
CASE STUDY & ANALYSIS
AlpaGasus
AlpaCiasus vs. Alpaca-52k
Instruction
Find the area of circle with radius cm.
Instruction
Implement a regulor expression in Python to validate an address.
ppacs-9k-
The area of circle with radius cm 50. 20.6655544543666.
random
Alpaca-52k
The regular expressioo in Python validate an email address is:'n'al (as
AlpaGasus
of circle with radius 78.54 square centimeters.
AlpaGasas
"[arAA-Z0-9, (6t]) (10- 2222333
Figure 11: Case study on 13B models of ALPAGASUS and ALPACA. Left:
Math capability comparison based on WizardI test set. Right: Coding skill comparison based on Vicuna test set.
Fig. II shows two case studies of 13B models trained on 52k data (ALPACA), 9k selected data (ALPAGASUS), and 9k randomly selected data (ALPACA-9K-random) The left case study focuses on the math capability, where ALPAGASUS can produce correct answer while ALPACA-9K-rande cannot. As the judge, GPT-4 rates the answer of ALPAGASUS by score of 10.0 while ALPACA- 9k-random receives score of 2.0. The right case study focuses on coding skills, ALPACC-522 cannot follow the instructions but produces regular expression to validate the website address while ALPAGASUS directly generates the correct code.
We also conduct fine- fine-grained evaluation of ALPAGASUS on each skill/category in the WizardLM and Vicuna test sets, whose samples are split into a list of skill sets/categories and thus facilitate detailed analyses of the capabilities achieved by IFT (Appendix HH. We compare two 7B models on the WizardLM test set and report the results in Fig. 25 Our ALPAGASUS achieves better or equally good performance than ALPACA on 22/29 skills but does not show advantages on the remaining 7 skills such as coding (e.g., code generation). To investigate the reasons, notice that the coding categories include "python". "Java", "C++", and "C#", which indicate that we can allocate training samples regarding coding skills based these related keywords (Appendix (E). We find that our data selection/filtering, without specifying the proportions skill categories, leads much higher filtering ratio of related data 718-85 68.16% than the average filtering 718
ratio 52002
52002-9229 82. 22.25%. Hence, the resulting coding skill is weaker than other skills. This indicates the importance of keeping the training data diverse and balanced across different categories IFT.
COST SAVING
We compare the training cost of ALPAGASUS and ALPACA in terms of the estimated expenses for the required computation on AWS. Notably, the training time is reduced from 80m to 14m for the 7B model and 5.5h to lh for the 13B model. Such training time reduction not only substantially enhances model iteration speed, but also reduces the cost from $27.31 1 to $4.78 78 for the 7B model and $225.28 $40.96 for the 13B model. It's noteworthy that instruction-tuning 65B L.LMMA models require greater number of GPUs and an extended training duration. Consequently, as the model size scales up, our data selection method yields progressively pronounced cost savings.
"The hyperparameters for IFT and the projected calculation method deferred in Table
8

---

Published as a conference paper at ICLR 2024
8 RELATED WORK
Open-sourced Instruction-following models. Instruction-tuning datasets can be gathered in two ways. A number of studies (Köpi et al. |023) Dolly 2023] Zhou et al. 2023) utilize crowdsourcing
to produce human-generated pairs of instructions and responses. This approach, while effective, can be laborious and costly. Alternatively, ALPACA (Taori et al. 2023) opens the door to create machine-generatee IFT sets from the distillation of the "teacher" LLM,, i.e., Text-Davinci-003, [Peng] ct al. 2023) keep the instructions from ALPACA intact but using GPT-4 as the "teacher" LLM, which enhances model on 3H (Helpfulness, Honesty and Harmlessness) (Askell et al. 2021 alignment criteria. Vicuna (Chiang et al. 2023) is the first to adopt ShareGPT [ (023) data, which
is the realistic dialogue data chatting with ChatGPT shared by users. Xu et al. (2023) and Luo
eet al. (2023) evolve the original Alpaca instruction set and obtain more complex instructions which help better elicit the instruction-following ability of LLMs. There also exists concurrent work like Koala (Geng et al. 2023) and UltraChat (Ding et al.) 2023), using dialogue & preference data as well as the adversarial prompts to conduct safe alignment.
Data-centric AL. Over the last decade, the realm of data-centric Al (Chu et al. 2016 Motamedi al. 202T] has witnessed substantial progress. Central this concept the belief that the quality of data (Haji et al. [202]] /Zha et al. 2023 Chen et al. 2202a d warrants the same level of importance as algorthmms within the AI/ML lifecycle. As noted by Chu al. (2016), for an effective engagement with diverse types of data across various domains, data cleaning processes should exhibit higher degree of automation and adaptability. With the advent of the Transformer architecture (Vaswani et al., 2017b). shift in the paradigm of language models has occurred. Models such RoBERTa (Liu ct al. 2019), BERT (Vaswani et al. 2017a) and Bard 10 all have incorporated this effective structure, stacking varying quantities of transformer blocks to create more potent models. This marked turning point in NLP research, signifying a heightened emphasis on data as opposed to model structure. Presently, SOTA LLMs like ChatGPT also underscore this shift toward data. They employ user data to conduct Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022a Gao et al. 2022). which further aligns with the Data-centric Al philosophy.
Evaluation of LLMs. Evaluating the open-ended instruction-followe following ability of LLMs is often
neglected by previous works (Chung et al., [2022] Anil al. (023), though they conduct a series of benchmark evaluations centered around factuality (Hendrycks et al. (020)) and reasoning (Bisk et al.
2020 for their pre- pre-training models. Similarly, the frameworks proposed by Liang et al. (2022) and
Gao et al. (2021) focus more on the evaluation of the base models but not on the evaluation of the IFT models, where open-ended fnstruction-folloiing is general ability but the scope of benchmarks is limited, the recent works such as Koala (Geng et al. 2023), Vicuna (Chiang et al. 2023), Self-Instruct (Wang et al. [2022), and Wiz- ardLM (Xu et al. 2023) all provide the instruction sets they collected and some of them also include the categories of the instructions for the evaluation of instruction-tunee LLMs. There are also some leaderboards like Alpaca-Eval Eval (Li et al., 2023) measuring the model's foslloiton-oollowinng ability. Leveraging these recent advancements, we evaluate our models on human instruction sets.
cntrructionfoollowiiii
capability are supposed
be prioritized. Since
CONCLUSION
In conclusion, our study reveals significant insights about the influence of data quality over quantity in IFT. Through our proposed data-filtering method, we have demonstrated that relying small subset of high-quality IFT data can lead to LLMs that exhibit enhanced instruction foolowwig capabilities, while also offering substantial computational advantages. Notably, our method proves versatile across different rating dimensions ge.g., Accuracy and helpfulness). LLM filters (e.g., ChatGPT and Claude-2), base model families (e.g.. LLaMA-1 and LLaMA-2), model sizes (e.g.. 7B and 13B), By emphasizing the importance of data
dataset types(c.g.. machine- -eeiineee and quality, we advocate for transition in the existing paradigm where data accumulation has been a primary focus. This perspective transition can lead to more meaningful advancements in the field of LLMs, making models more aligned with human intentions and less prone to errors induced by poor-quality data.
"tttps://bard. googl comm/
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGE
Lichang Chen and Heng Huang were partially supported by U.S. NSF IIS 2347592, 2347604, 2348159, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10400 2023.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2 12.00611 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08003, 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, 2020.
Jiuhai Chen, Lichang Chen, and Tianyi Zhou. It takes one to tango but more make trouble? in-context training with different number of demonstrations. arXiv preprint arXiv:2303.08119, 2023a.
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03085 2023b.
Lichang Chen, Minhao Cheng, and Heng Huang. Backdoor learning on sequence sequence models. arXiv preprint arXiv:2305.02424 2023c.
Lichang Chen, Heng Huang, and Minhao Cheng. Ptp: Boosting stability and performance of prompt regularizer. arXiv preprint arXiv:2305.02423, 2023d.
tuning with perturbation-
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1rXiv:2107.03374 2021.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprini TrXiv:2306.0475: 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URLhttps
//1 mllmss..... orgyy..org/oe http://ll///// /ttpoo/gg//22////////// 2023-03 30 -iina
Xu Chu, Ihab F Ilyas, Sanjay Krishnan, and Jiannan Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data, pp. 2201-2206, 2016.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction fineiton-- language models. arXiv preprint arXiv:2210. 1416, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 23055 14233, 2023.
Dolly. Free dolly: Introducing the world's first truly open instruction-tuned 11m.
Blog Post, 2023.
URL http:://www..ataaab.....ccooo/oo
dol 1olly-firee open www.-/-commmmmaaaaccaoaccc.acaae http://e..........toeo.eeeeeatee
10

---

Published as a conference paper at ICLR 2024
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387, 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster. Laurence Gold- ing, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httthhtts://eet /ttp::::/ddii.. http://o...rrr//////// wtt.:oor./........... www///221////... 5371
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210. 10760, 2022.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL
Mustafa Hajij. Ghada Zamzmi, Karthikeyan Natesan Ramamurthy, and Aldo Guzman Saenz. Data- centric requires rethinking data notion. arXiv preprint arXiv:21 10,0249, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. Look at the first sentence: Position bias in question answering. arXiv preprint arXiv:2004. /4602, 2020.
Andreas Köpf, Yannic Kilcher, Dimitri Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant convessatons-demoorriinnnggg large language model alignment. arXiv preprint arXiv:2304.07327, 2023.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instructionnfollloiing models. httpps://itthb. ht//://thha..mmmmmaaaalalaaaaaaa
laalaaaaaaaaaaaa_aaaaaaaaaa paca_evall 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211 00ti::2211009110,, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1r0iv 11692, 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin. and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
Mohammad Motamedi, Nikolay Sakharnykh, and Tim Kaldewey. data-centric approach for training deep neural networks with less data. arXiv preprint 1rX0v:2110.03611, 2021.
OpenAl. Chatgpt. http://openaai.om/hhlggcaha 2023a.
OpenAl. Gpt-4 technical report. arXiv, 2023b.

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022b.
Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprini arXiv:2304.03277, 2023.
ShareGPT. Sharegpt. 2023. URL shwregpt....o. com
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-hench tasks and whether chain-of-thought can solve them. arXiv preprint 10X0v:2210009261, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloinng llama model. https://githun. www.tatstooeaaeooooooooco www.laaabssaafrddoooaa alpaca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017a. URL
Guyon, U. Von
wwt..ps...pppppppppp.eaeeeeeeee. wwtp:ccrpppppppeeeeeeeeee taapper_filee/eeeeeeppeeeeeeeeee esspppper22017 FLle:fffee223337/6e21111032222.2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017b.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.1792 2023.
Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 610-618, 2018
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions, 2023.
Daochen Zha, Zaid Pervaiz Bhat, Kwei-Hemg Lai, Fan Yang. Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv: 10158, 2023.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arxiv preprint arXiv:2306. 05685, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. /206, 2023.
12

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Frequently Asked Questions)
14
Is there any bias contained in evaluation prompts?
A.2
Have you tried other LLM filter?
14
A.3
What about the results on other base models, e.g. LLaMA-2!
15
A.4
Can your LLM lilter evaluate the stronger model's responses, e.g., liltering the
responses given by GPT-42|
A.5
Results on other rating dimensions, e.g., helptulness?
B
Additional Results on Dolly Dataset
17
B.
Score Distribution
17
B.2
Benchmark results
17
B.3 Dolly-13B Results
Details of GPT-4 Evaluation Prompt
18
Training Hyperparameter Details
19
D.1 Alpaca Dataset
19
D.2 Dolly Dataset
19
E Keywords set for detailed analysis
19
F Rated examples in Alpaca Dataset
20
Rated examples in Dolly Datasel
23
Analysis
26
H.l Analysis on WizardLM M Test Sct
26
HI2 Analysis on Vicuna Test Set
27
Detailed Analysis the WizardLM testset
27
Human Study
31
Limitations
31
13

---

Published as a conference paper at ICLR 2024
A FREQUENTLY ASKED QUESTIONS
A. Is THERE ANY BIAS CONTAINED IN THE EVALUATION PROMPTS?
We also explore alternate evaluation prompts such as the prompts provided by Zheng et al. (2023). which are shown in Table 3
We apply the same rules to calculate the Win-Tie-Loss and show the results in Fig. 12 Notably, ALPAGASUS consistently outperforms across all test sets.
Vicuna
40
11
21
AlpaGasus
Koala
68
59
Alpaco wind
WizardLM
Self-
Instract
96
31
91
Figure 12: The experimental results when using the evaluation prompt from Zheng et al. (023) to judge the responses. ALPAGASUS could still maintain its advantage.
System Prompt
Please act as an impartial judge and evaluate the quality of the responses provided by two Al assistants the question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective possible. After providing your explanation, output your final verdict by strictly following this format: "[[A]" if assistant A is better, "[[]]" if assistant B is better, and "[[C]]" for a tie.
Prompt Template
[User Question] {question}
[The Start of Assistant A's Answer] {Answera)
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer] [Answer)]
[The End of Assistant B's Answer]
Table 3: The GPT-4 evaluation prompt from Zheng et al. (023)
A.2 HAVE YOU TRIED OTHER LLM FILTER?
Yes, we also try use Claude-21D III
our response quality evaluator (LLM filter). Fig. 13 and
Fig. [4] demonstrae the score distribution and evaluation results on the four testsets, respectively. Remarkably, the 7B model instruction-tune -uned with 8k selected data could be better than the model instruction- with 52k Alpaca data on 3/4 testsets and achieves significantly better over the model instruction-tuned with 8k random selected data.
14

---

Published as a conference paper at ICLR 2024
Score Distribution(lludd-2 as LLM filter)
40000
32000
24000
16000
8000
<3
Count
Figure 13: The score distribution of using Claude2 as the LLM filter.
Vieuna
38
19
Alpaca- 7Bclaudee
selected 8k) wins
Vicuna
41
14
51
Alpaca selected wins
Tie
Koala
Koals
(opora-70[0--randoo wins.
Alpaca-78//53)
69
78
WizudLM
58
70
Self: Instret
102
57
83
Instruct
Self
107 76 68
Figure 14: The experimental results by using the Claude2 response quality evaluator.
As Fig. [3 shows, the interval between two scores is which is different from the ChatGPT-based filter, where the interval is 0.5. Thus, if we would like to have fine- fine-grained scores, larger rating scale should be applied to the prompt as the present 5-point scale does not suffice. We leave the exploration of the rating scales to future work.
A.3
WHAT ABOUT THE RESULTS ON OTHER BASE MODELS, E.G., LLAMA-22
We also have the results of LLaMA2 in Fig. 15 which shows the superiority of our method.
A Alpagusus2-7Bou vs. Alpaca2-7B(5222
AlpaGasus2-7Bp09 Alpaea2-7B49k-mmmaae wins
AlpGGasus2 781SN] wins
Vicuna
Ti
Vieuma
Tim Alpaca2 St-random wins
Alprcot-70/saa wins
Konla
70
Kosla
Wiand1AM
91
58
63
Self- Instract
Self- Instract
102
72
Figure 15: The experimental results on LLaMA2. Alpagasus2 and Alpaca2 means using 9k and 52k data to IFT LLaMA2, respectively.
A.4 CAN YOUR LLM FILTER EVALUATE THE STRONGER MODEL'S RESPONSES, E.G., FILTERING THE RESPONSES GIVEN BY GPT-4?
To answer the question, apply LLM filter to GPT4LLM (Peng et al. 2023) data. According to the score distribution, we use 4.5 as the threshold and select 13721 data samples from the GPT4LLM dataset for IFT I.L.MA-BB.
15

---

Published as a conference paper at ICLR 2024
Score Distribution(Alpacaggg4))
30000
27124
20000
15000
13710
10000
9372
5000
1685
100
11
<3
3.5
5
Count
Figure 16: The score distribution of Alpaca-GPT4 dataset.
(Ipact-GPT4(33k) vs. (PT4ca)GPT4(28k)
Alpaca-GPT<(133)
Alpcca-GPTT4(33--aadoo
Alpace-gpHi1 wins
Vicuna
30
Appcc--gnetA33 wins
Vicuna
35
23
22
Alpate-gpMi11
64
Alp Alacc-ggrAAbaa wins
Koala
WizzeLLM
Self-Instruct
Figure 17: The evaluation results Alpaca-GPTT dataset.
The results presented in Fig. 17 demonstrate the superiority of our method on the Vicuna and WizardLM test sets. Even though the responses from GPT4LLM are generated by GPT-4, recognized the most advanced LLM globally, of the original data. Notably, the performance our method markedly surpasses that of randomly selected counterparts. In summary, our LLM filter exhibits promise discerning superior responses from teacher models.
approach attains comparable outcomes using merely 25%
A.5 RESULTS ON OTHER RATING DIMENSIONS, E.G., HELPFULNESS?
We also use "helpfulness" as rating dimension and find that only need 2k data train the base model that can surpass the base model trained with 52k Alpaca data. The score distributions are shown in Fig. 18
16

---

Published as a conference paper at ICLR 2024
Score Distribution(hellfuunes)
42063
40000
32000
24000
5280
2516
2
<3
3.5
5
Count
Figure 18: The score distribution of helpfulness.
Evaluation Results From Figure 19 it is evident that the models trained using our filtered Alpaca dataset outperform those trained on randomly selected datasets across all instruction test sets. Fur- thermore, our model outperforms the model trained on the complete Alpaca set in out of 4 test sets. This underscores the significant potential of filtering approach, especially considering that a model trained with mere 2k data points can surpass one trained with the original 52k Alpaca dataset.
Vicuna
47
14
Alpaca-78Ptt
helpfunnes) wins
Vicuna
14
31
Alpaca- helpfrsinens
Koala
Alpxcx
Koala
45
Alpaca-78pp% random) Wins
WizardI
74
74
WizardLM
Self
Self- Instruct
Instract
77
102
73
108
93
Figure 19: Evaluation results regarding on the "helpfulness" dimension.
B ADDITIONAL RESULTS ON DOLLY DATASET
B.1 SCORE DISTRIBUTION
We show the score distribution of Dolly dataset(rated by ChatGPT) in Fig. 20
B.2 BENCHMARK RESULTS
We the code provided by Chia al. (2023) conduct benchmark evaluation. For MMLU, BBH, Drop, and humaneval, we also use 5-shot, 3-shot, 3-shot, and 0-shot settings, respectively. We show the benchmark results in Table of Dolly and the filtered set.
Datasets
7B(3k-random)
7B(3k)
7B(15k)
13B(3k-random)
13B(3k)
13B(15k)
BBH 31.33
31.76
30.73
36.15
36.37
35.8
Drop
20.73
22.45
22.33
31.61
34.24
26.94
Humaneval
9.76
9.78
7.93
10.98
14.92
14.63
MMLU
35.01
35.83
36.25
44.39
46.92
46.13
Table 4: The benchmark results of filtering the Dolly dataset.
Here are the hyperparameters we select for the training the LLaMA-7B and LLaMA-13B are the same the Alpaca except for the training epochs. To avoid the under-train issue, train 10 epochs,
17

---

Published as a conference paper at ICLR 2024
Score Distribution(Dolll)
10000
8948
8000
7000
6000
5000
2981
3000
2335
2000
2
15
0
<3
3.5
4
4.5
Count
Figure 20: The score distribution of the Dolly.
instead of in Alpaca, for all the 7B models and 15 epochs, instead of Alpaca, for all the 13B models.
B.3 DOLLY-13B 13B RESULTS
We show the doBly-13B results. As Fig. shows, our filtered Dolly dataset is better than the original Dolly dataset since achieve stronger instruction instructionfollowinng capacity of the instrutiio-tunnd LLaMA-7B models via ours. (See the results on the four tests)
Dolly-13B(3k) vs. 1Bl15-13B(15k)
Dolly-13B(3k) vs. Dolly- 13B(333r33doommmm
Vieusa
138136)
Vicuna
Delly-138
Koala
72
54
54
Dolly 138 wins Koola
71
42
67
Delly-188 3k-random wins
WizadEMM
WicardLM
Self-
Instruct
103
54
Instruct
105
62
54
Figure 21: Dolly 13B results. We show the dolly-13B results here. With the model size going our method can still perform pretty well.
C DETAILS OF GPT-4 EVALUATION PROMPT
We provide the detailed form of the prompt to GPT-4 used for evaluation in Fig. 22 It is the prompt for evaluation used in the original Vicuna blog
18

---

Published as a conference paper at ICLR 2024
System Prompt:
You are helpful and precise assistant for checking the quality of the answer.
User Prompt:
[Question]
[The Start of Assistunt Answer]
(answer 1}
[The End Assistant I's Answer]
[The Start of Assistant Answer]
(answer 2]
[The End Assistant 2's Answer]
We would bike request your feedbuck the performance of two assistunts in response the question displayed ahove.'nPleas. rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives overall score scale of I to 10, where higher score indicates better overall performance. 'nPlesse first output single line containing only two values indicating the scores for Assistant 1 and 2, respoctively. The two scores are sepurated by space. In the subsequent line, please provide comprehensive explanation of your ovaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment."
Figure 22: The prompt for evaluation using GPT-4 as the judge.
D TRAINING HYPERPARAMETER DETAILS
D.I ALPACA DATASET
We show the training hyperparameters and costs Table
Model Size
Size
GPUs
Epoch
Batch Size
Time
Cost
7B
9k
4
3
2e-5
128
14m
$ $..78"
7B
52k
4
2e-5
128
80m
$ 27.31"
13B
9k
8
5
le-5
128
1h
$ 40.96
13B
52k
8
le-5
128
5.5h
$ 225.28
Table 5: All the cost is estimated based on the price provided by AWS, We assume the training scripts for all models are the same (e.g., training epochs, batch size on each GPU, accumulation steps, etc.)
D.2 DOLLY DATASET
We show the training hyperparameters in Table
Model Size
Data Size
Epoch
LR
Batch Size
7B
3k
3
2e-5
128
7B
15k
3
2e-5
128
13B
3k
5
le-5
128
13B
15k
5
le-5
128
Table 6: IFT hyperparameter details. (Dolly Dataset)
KEYWORDS SET FOR DETAILED ANALYSIS
We use the keyword set of JJaa,, java, C++, C++, C#,, ctf, Python, python] and count the number of (instruction, input, output) tuples which contain the keyword in this set.
Phttpss//aws. amazon com / /cc// settanccetypes. /p4// 244ee22xxaargeee lareeee node has 8 80GB A100 and it costs $40.96/h. *w assume training time of using GPUs is half of using 4 GPUs
19

---

Published as a conference paper at ICLR 2024
F RATED EXAMPLES IN ALPACA DATASET
We include more examples rated by the response quality evaluator, i.e., ChatGPT, in this section. The examples of Score 5.0, Score 4.5, Score 4.0, Score 3.5, Score 3.0, Score 2.5, Score 2.0 are shown in Table 7 Table
g Table 9 and Table TO respectively.
[Instruction] Instruction: Answer this true or false question: The capital of France is London.
[Inptt] None
[Response] False. The capital of France is Paris
[Score& & Explanation]: 5.0. The Al assistant provided a correct and accurate response to the instruction and input given. It correctly identified that the statement The capital of France Londonïs false and provided the correct answer that the capital of France Paris. The response was clear and concise, and there were no errors or misunderstandings in the Al assistant's 's interpretation the question. Therefore, the Al assistant deserves a score of 5 for its performance in this task.
[Instruction] Translate the phrase "Bonne chance" into English [Inptt] None
[Response] Good luck.
[Score & Explanation]: 5.0. The Al assistant provided a correct and accurate response to the given instruction and input. Bonne chanceis French phrase that translates Good luckin English, and the Al assistant provided the correct translation. Therefore, the assistant deserves score of for its accuracy
[Instruction] Complete the following sentence given the context: [Input] My grandmother always said," " An apple day
[Response] Keeps doctor away."
[Score & Explanation]: 5.0. The AI assistant provided perfect response to the given input and instruction. The response accurately completed the sentence with the correct phrase that follows An apple a day: Therefore, the assistant deserves a score of for its accuracy.
Table 7: The score=5.0 examples in Alpaca dataset.
20

---

Published as a conference paper at ICLR 2024
[nstruction] Instruction:
Suggest ways to reduce plastic waste in everyday life.
[[пut]] None
[Response] One way to reduce plastic waste in everyday life is to make the switch to reusable items. This includes bringing a reusable water bottle instead of buying bottled water, investing in reusable coffee cup, bringing your own bags when shopping, and using reusable containers for storing and transporting food. You can also reduce the amount of plastic waste you create by avoiding or limiting the use of take-out or single-use containers, utensils, and straws. Addition- ally, try to find sustainable altemaaties plastic items when shopping and avoid purchasing items wrapped in plastic whenever possible. Finally, make sure you recycle any plastic items you do have.
[Score & Explanation]: 4.5.The response provided by the Al assistant is highly accurate and relevant to the given instruction. It suggests various ways to reduce plastic waste everyday life, including using reusable items, avoiding single-use containers and utensils, finding sustainable alternatives, and recycling. The only potential improvement could be providing more specific examples of sustainable alternatives to plastic items. Overall, the response is informative and helpful in addressing the given instruction
[Instruction] Generate list of non-alcoholic drinks.
[Input] None
[Response]
Fruit juices
Tea
Smoothies
Iced coffee
Milkshakes
Fizzy drinks
Hot chocolate
Lemonades
Kombucha
Energy drinks
Milkshake
Coconut water
Ginger beer
Ice tea
Sparkling water
Aloe vera drink
Vegetable juice
[Score & Explanation]: 4.5. The Al assistant provided a comprehensive list of non-alcoholic drinks that commonly consumed. The list includes variety of options such as fruit juices, tea, smoothies, and fizzy drinks. However, there are some duplicates in the list such as milkshakes and iced coffee. Additionally, some of the options may not be as commonly consumed as others, such as kombucha and aloe vera drink. Overall, the response is accurate and provides a good range of options for non-alcoholic drinks.
Table 8: The score=4.5 examples in Alpaca dataset.
21

---

Published as a conference paper at ICLR 2024
[Inttrcctio]] Instruction: Compare and contrast commercial banks and investment banks.
[[пptt] None
[Response] Commercial banks are financial institutions that provide services to their customers, such loans, deposits, and investment management. Investment banks, on the other hand, are type of financial institution that specializes in the trading of securities, such as stocks and bonds. The main difference between the two is that commercial banks provide services customers, while investment banks provide services to govermments. corporations, and wealthy individuals. Commercial banks are also subject to more regulations than investment banks, as they offer deposit accounts to customers. Investment banks, however, do not accept deposits from the general public and are subject to fewer regulations. Both types of banks are important for the global economy, but their roles and services are different.
[Score & Explanation]: 4.0. The response provided by the Al assistant is mostly accurate and relevant to the given instruction and input. It clearly explains the differences between commercial banks and investment banks, highlighting their respective roles and services. However, it could have provided more specific examples of the services offered by each type of bank, and could have elaborated further on the regulations that apply to them. Overall, the response is informative and helpful, but could benefit from more detail.
[Instruction] Determine the application of statistics in the field of economies [Input] None
[Response] Statistics is widely used in economics to analyze and predict markets, examine the performance of companies, and evaluate economic policies. Statistics is used to identify trends and correlations and to measure variables such as prices and wages. Statistical models are also used to predict economic growth and changes in interest rates and to test the effects of regulation and taxation. Statistical tools are also used to estimate the impacts of policy changes, assess the effectiveness of government programs and identify the root causes of problems.
[Score & Explanation]: 4.0. The AI assistant provided a fairly accurate response to the in- struction. It correctly identified the various applications of statisties in the field of economies, including market analysis, company performance evaluation, policy evaluation, and prediction of economic growth and interest rates. However, the response could have been more detailed and provided specific examples of statistical methods used in economics. Overall, the response was informative and provided a good overview of the topic
Table 9: The triplets (instruction, input, response) obtain Score 4.0.
[Instruction] Design a company logo.
[Input] None
[Response] <nooutput>
[Score& & Explanation]: 2.0. The Al assistant did not provide any response to the given instruction and input. Therefore, it cannot be evaluated for accuracy.
[Instruction] Classify the item as either animal or vegetable. [Inptt] Banana
[Response] Animal: No, it's vegetable.
[Score& & Explanation]: 2.0, The AI assistant's response is incorrect. A banana is a fruit, which is type of blant-based food, making it a vegetable. However, the Al assistant classified it as an animal, which is completely inaccurate. Therefore, the score low.
[Instruction] Design a poster to advertise a product. [Inutt] None
[esspons]] No output required.
[Score& & Explanation]: 2.5.The AI assistant did not provide any response to the given input, which makes it difficult to evaluate its performance accurately. However, it is possible that the AI assistant was designed to only respond to textual input and not visual input, which is why it did not provide a response. Therefore, I have given it score of 2.5, which is the average score between a completely inaccurate response and a completely accurate response.
Table 10: The score=2.0 and score=2.5 examples in Alpaca dataset.
22

---

Published as a conference paper at ICLR 2024
G RATED EXAMPLES IN DOLLY DATASET
[Instruction] What is Jenkins?
[Context] None
[Response] Jenkins is open-source automation server. helps automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery.
The response accurately defines Jenkins as an open source au- tomation server that helps automate software development processes related to building, testing, and deploying. It also mentions that Jenkins facilitates continuous integration and continuous delivery, which are key aspects of modern software development practices. Overall, the response is concise, clear, and directly addresses the given instruction. Therefore. I have given it score of
[Score & Explanation]: 5.0.
[Inttrcctio]] Who was the first president of the United States of America? [Context] None
[Response] George Washington
[Score & Explanation]: 5.0. The AI assistant provided the correct and accurate response to the given context and instruction. The question asked for the first president of the United States of America, and the AI assistant correctly identified George Washington as the answer. Therefore, the Al assistant deserves score of 5 for its accuracy.
[Instruction] What country is to the north of the United States?
[Context] None
[Response] Canada America's northern neighbor.
[Score& & Explanation]: 5.0. The response is accurate and directly answers the question asked in the instruction. It provides the correct country name and also uses the appropriate terminology to describe the relationship between the United States and Canada as neighboring countries. Therefore, the AI assistant has performed well this context.
Table 11: The score=5.0 examples in Dolly dataset.
23

---

Published as a conference paper at ICLR 2024
[Instruction] Help me with spending a day in New Delhi. I like architecture, culture, and am foodie.
[Context] None
[Response] North India has lot to offer for aesthete, epicure, and xenophile. Day 1 arrive in Delhi as many flights arrive early in the morning, settle into a hotel of your choice. Today travel around Delhi especially old Delhi and start your trip by visiting Red Fort which is historical Mogul Fort commissioned by Mogul Emperor Shah Jahan in 1638. The fort is steeped in history and has a beautiful combination of Persian architecture with Indian influence. After that have lunch in Chandi Chowk where you will find incredible street food and especially Karim's 's which specializes in Mughlai food. Chandi Chowk is full of verve and energy and you will experience slice of Indian culture. After the delicious lunch and walking through the narrow streets of Chandi Chowk, it time to visit Qutb Minar a Unesco World Heritage Site which is considered the tallest minaret in the world at 240 ft constructed out of sandstone in 1192. Close by you can also visit Jamal Kalami mosque and relax at the nearby 200 acre park. In the evening you can travel to visit the Lotus Temple in Delhi which is Bah00e102be000 102bc00ed House of Worship and architected by Fariborz Sahba in 1976. For dinner, it is time to indulge yourself at Oberoi Delhi at Dhilli under the mentorship of Michelin-starred Chef Vineet Bhatia MBE and finish your night at the rooftop bar Cirrus 9 with smooth cocktail.
[Score& & Explanation]: 4.5, The response provides comprehensive itinerary for spending day in New Delhi, catering to the interests of the user who likes architecture, culture, and food. The assistant suggests visiting historical sites such as Red Fort and Qutb Minar, experiencing the culture and food of Chandi Chowk, and visiting the Lotus Temple. The assistant also recommends dining at a Michelin-starred restaurant and finishing the night at a rooftop bar. The response is accurate and provides a well-rounded itinerary for day in New Delhi. However, it could have been improved by providing more specific details such as the opening and closing times of the suggested.
[Instruction] Why is the sky blue? [Context] None
[Response] All the light in our daytime sky comes from the sun, and our sun produces light in many different colors. To understand this, just think of a prism. White light enters, and rainbow of colors exits. Our sun is like that. Its light may seem uniform but is actually emitting many different colors. Our sky though is not mix of blue and red and green because the earth's 's atmosphere is sandwiched between the sun and us, getting in the way. Atmospheric gases and particles scatter the incoming colors. bouncing them in all different directions. The amount of scattering increases as the wavelength of light decreases. Red light which has a long wavelength is scattered much less than blue light which has short wavelength. As blue is scattered more, it fills our daytime sky. As the sun traverses the sky and moves low on the horizon, the light that reaches our eyes must actually travel further through the atmosphere. This has the effect of completely bouncing away all the blue light. We are left with the reds and yellows and oranges in wonderful mixture, making sunrises and sunsets so beautiful.
[Score& & Explanation]: : 4.5. The response accurately explains why the sky appears blue during the day and why it appears red during sunrise and sunset. also provides clear explanation of how atmospheric gases and particles scatter light, causing blue light to be scattered more than other colors. The only potential improvement could be to provide a more concise explanation, as the response is somewhat lengthy. Overall, the response is highly accurate and informative.
Table 12: The 5c examples in Dolly dataset.
24

---

Published as a conference paper at ICLR 2024
[Istrucciio]] What is a bicameral legislature?
[Context] None
[Response] A legislative body comprised of two separate chambers. [Score & Explanation]: 4.0. The response provided by the AI assistant is accurate and directly answers the question asked in the instruction. The definition provided is concise and clear, and demonstrates good understanding of the concept of bicameral legislature. However, the response could have been improved by providing additional information or examples to further clarify the concept.
[nstructioo]] When is tax day? [Context] None
[Response] Tax day usually refers to the day in the United States when tax forms must submitted and paid for individual and businesses. It is typically April 15 or later.
[Score & Explanation]: 4.0. The response accurately defines what tax day is and provides the typical date for in the United States. However, it could have been more specific by mentioning that the deadline for tax day can vary depending certain circumstances, such as weekends or holidays.
[Intrrcctio]] Write a bullet point list at least 7 crops that are grown in California. [Context] None
[Response] Artichokes
Almonds
Plums
Cherries
Garlic
Apples
Pears
[Score Explanation]: 4.0. The AT assistant provided a list of crops that are grown in California, which is accurate and relevant to the given instruction. However, there are many other crops that are also grown in California, and the assistant could have provided a more comprehensive list. Therefore, the response is not perfect but still satisfactory.
Table 13: The score=4.0 examples in Dolly dataset.
25

---

Published as a conference paper at ICLR 2024
[Inttructio]] How to make a cup of spiced chai?
[Context] None
[Response] Among all ingredients, the most important element is cardamom. Of course, you cannot skimp loose black tea leaves too!
[Score & Explanation]: 2.5. The response mentions two important ingredients for making spiced chai cardamom and loose black tea leaves. However, it does not provide any instructions on how to actually make the chai. The response is partially accurate as it addresses the context of making spiced chai but does not fully answer the instruction of providing the steps to make it.
[Instruction] Is Seattle a nice place to live?
[Context] Seattle came in at No. 39 in the 2023 World's Best Cities Report, a comprehensive annual study by Resonance Consultancy. Seattle is a seaport city on the West Coast of the United States. The culture of Seattle is heavily defined by its significant musical history. Between 1918 and 1951, nearly 24 jazz nightclubs existed along Jackson Street, from the current Chinatown//nternationna District the Central District. The jazz scene nurtured the early careers of Ernestine Anderson, Ray Charles, Quincy Jones, and others. The city is also the origin of rock acts such as Foo Fighters, Heart, and Jimi Hendrix, well as the grunge subgenre of rock and its pioneering bands Alice in Chains, Pearl Jam, Nirvana, and Soundgarden. The city itself is hilly, though not uniformly so. Like Rome, the city said lie on seven hills; the lists vary but typically include Capitol Hill, First Hill, West Seattle, Beacon Hill, Queen Anne, Magnolia, and the former Denny Hill. Due to its location in the Pacific Ring of Fire, Seattle is in a major earthquake zone. Temperature extremes are moderated by the adjacent Puget Sound, greater Pacific Ocean, and Lake Washington. Thus extreme heat waves are rare in the Seattle area, as are very cold temperatures (below about 15 F). The Seattle area is the cloudiest region of the United States, due in part to frequent storms and lows moving in from the adjacent Pacific Ocean. With many more "rain days" than other major American cities, Seattle has well-earned reputation for frequent rain. Seattle's 's economy is driven by mix of older industrial companies and "new economy" internet and technology companies, as well as service, design, and clean technology companies.
[Response] Yes.
[Score & Explanation]: 2.5. The Al assistant's response is not accurate enough. While the context provides information about Seattle's culture, geography, climate, and economy, the instruction asks whether Seattle is nice place to live. The Al assistant's 's response is too brief and does not provide any supporting evidence or explanation to justify its answer. Therefore, the response is not helpful or informative enough for the user.
[Instruction] Best workout in the world. [Context] None
[Response] Walk. Just walk as much as you can.
[Score & Explanation]: 2.0. The response provided by the AI assistant is not very accurate. While walking is good exercise, it is not necessarily the "best workout in the world" as the instruction suggests. The response also lacks specificity and does not provide any additional information or guidance how make the most out of the workout. Therefore, I would rate the accuracy of the response as 2 out of 5.
Table 14: The score=2.0 and score=2.5 examples in Dolly dataset.
H ANALYSIS
H.1 ANALYSIS ON WIZARDLM TEST SET
We conduct a fine-gres ane-grained evaluation of ALPAGASUS on each skill/category in the WizardLM and Vicuna test sets, whose samples are split into list of skill sets/categories and thus facilitate detailed analyses of the capabilities achieved by IFT.
ALPAGASUS-7B(9h) vs. ALPACA-7B(52B6) test set and report the results in Fig. 25 Our ALPAGASUS achieves better equally good performance than ALPACA on 22/29 skills but does not show advantages on the remaining skills such as coding (e.g., code generation). To investigate the reasons, notice that the coding categories include
We compare these two 7B models on the WizardLM
26

---

Published as a conference paper at ICLR 2024
"python", "Java", "C++", and "C#", which indicate that allocate training samples regarding coding skills based on these related keywords (Appendix E We find that our data selection/filtering, without specifying the proportions of skill categories, leads much higher filtering ratio of coding- related data 718 85 88. 16% than the average filtering ratio 52002
10002-222 82.25%. Hence, the
718
resulting coding skill is weaker than other skills. This indicates the importance of keeping the training data diverse and balanced across different categories in IFT.
H.2 ANALYSIS ON VICUNA TEST SET
Wiiling
Wrtting
Ap
Math
Coding
Coding
Apaco Conterfactual
Fermi
Fenmi
Releplay
Roleplay
Knowledge
Knowledge
Genzric
Generic
Figure 23: Fine- theeggrineed evaluation of ALPAGASUS-133-B ALPACA-11B-52K categories of the Vicuna test set.
demonstrates the detailed analysis on Vicuna testset. ALPAGASUS-7B is better than the ALPACA-7B in the majority of the categories, including Counterfactual, Roleplay, Knowledge, and Generic, etc. Another strong point is that when the base model scales up, the conclusion still holds. (See right part of the Fig. 23
Fig. 23
I DETAILED ANALYSIS ON THE WIZARDLM TESTSET
In Fig. 26 Fig. 27 and Fig. 28 we compare ALPAGASUS with text-Davinci-03,, ChatGPT, and Claude, respectively. The results show that ALPAGASUS-13B can achieve > 91% capacity of its "teacher" model. text- text-Davinci-00)) Dati--iiinn-0 (all the responses in the ALPACA-522 dataset are generated by call it "teacher" LLM). The results also show that model could achieve pretty good performance on tasks like Writing, RolePlay, Toxicity, Art, etc., while it still needs improvement coding and math capacity when compared with stronger L.Mss.
27

---

Published as a conference paper at ICLR 2024
Math
Code Generation
Writing
Computer Seience
Reasoning
Complex Fornal
Commonsennse
Counterfactual
RokePlay
Biology
Technology
Bilics
Sport
History
Academic Writing
Chemistry
Code Debug
Muttilingual
TruthfulQA
Medicine
Literature
Entertainment
Music
Toxicity
AlpaGiasa-90 wins
Physics
Tic
Philosophy
Alpaca 52k wins
Economy
Figure 24: Fine-grained evaluation ALPAGASUS-9K113B) vs. ALPACA-52K(13B) on categories of the WizardLM test set.
28

---

Published as a conference paper at ICLR 2024
Math
Writing
10
2
Computer Science
8
Multilnguual
Roleplay
Technology
Ethics
Sport
Law
2
Economy
2
History
2
Academic
Writing
Chemistry
Reasening
Code Debug
2
Commonsemse
Philosophy
Complex
Counterfactual
TruthfalQA
Entertainment
Music
AlpaGasus-96
Texicity
Tic
Alpaca-52k wins
Physics
Figure 25: Fine-grained evaluation of ALPAGASUS-9K/TH ALPACA-52K(7B) categories of the WizardLM test set.
29

---

Published as a conference paper at ICLR 2024
LM Test Set (Skills Details)
Alpaca-13B-9h vs. ChatGPT
100.00%
Avg. 78.28%
80.00%
Alpaca
40.00%
20.00%
Academic Writing
Econeryy
Chemistry
Entertainnent
Etfics
Biology
Medicine
Art
History
Writing
Law
Tadicty
Music
Literature
Technology
Roleplay
Computer Science
TruthfuiQA
Muhtlingual
Sport
Physics
Philosphly
Reasoning
Comples Format
Common Sonse
Code Debug Math
Counter
Cosle Generatio
Figure 26: Compare with ChatGPT, Achieve average 78.26% capacity of ChatGPT on all 29 skills.
WizardLM Test Set (Skills Details) Alpaca-13B-9k Claude
120.00%
100.00%
Aug.
80.00%
60.00%
20.00%
0.00%
Roleplay
Academic Writing
Hitory
Writing
Eaaonyy
Medicine
Ethics
Entertainment
Sport
Technology
Law
Literature
Art
Music
Computer Science
Chemistry
Toxicity
TrathfulQA
Counterfactual
Multilngual
Common Common-Sermn Seree
Biology
Philosophy
Physics
Code Debug
Comples Formut
Math
Code Gereration Reasoning
Figure 27: Compare with Claude-vl. Achieve average 78.41 % capacity of ChatGPT all 29 skills.
WizardLM Test Set (Skills Details) Alpaca-13B-88 Davinci-003
140.00%
100.00%
=91 s91.11%
80.00%
60.00%
Alpaca-1333
40.00%
20.00%
0.00%
Toxtoity Academic Writing
Physics mntertinment
Art
Ethles
Chemintry
History
Law
Writing
Ecanmyy
Complex Farmat
Medicine
Roteplay
Technology
Computer Science
Counteractual
Sport
Biology
Literature
Music
Common-Sarss
Mathi
Muttlingual
Phitsophy
Reasoning
Code Geveration Code Debug
Truthffaaa
Figure 28: Compare with Davinci-003, Achieve an average 91.11% capacity of ChatGPT on all 29 skills.
30

---

Published as a conference paper at ICLR 2024
J HUMAN STUDY
We conduct the human study among three different users. The evaluation interface is shown as Table 15
You'l 'll be presented with a series of questions. For each question, two answers will be provided. Your task is to read both answers carefully and decide which one you believe is better. When judging, consider:
Relevance:
Does the answer directly address the question?
Completeness: the answer comprehensive?
Coherence: Is the answer logically structured and easy to understand?
Accuracy: the information provided the answer correct?
Question:
<QUESTION>
Answer A:
Answer B:
<ANSWER A>
<ANSWER B>
Comparing these two answers, which answer is better?
1. Answer A is significantly better.
Answer significantly better.
3. Neither is significantly better.
Table 15: Human annotation interface.
We show more detailed results of human evaluations in Fig. 29
Human (tudyyAlpagaasu--1BBB11 vs. Allpaca-13B(52))
Vicuna
17
17
Koala
7
Alpagasus-9k wins
Tie
WizardLM
22
10
Alpaca-52k wins
-enstrutt
25
10
Figure 29: The detailed results of human study.
K LIMITATIONS
Model Size. In our experiments, we evaluated IFT strategy by training models of different sizes, 7B and 13B, since they are the most common sizes for recent open-source LLMs. We plan to extend this study to larger model sizes such as 33B, 65B, or even 175B, and verify whether the same conclusion still holds, i.c., a small subset of high-quality data selected by our method can improve the instruction- finettuioo-iieueedd model. We leave analysis on the IFT of larger models as future work.
31


Reviews:
[{"review_id": "58mPXwhF3x", "reviewer": "Reviewer_qmSJ", "summary": "This paper proposes filtering low-quality instruction data by an LLM judger (GPT-4 specifically). The authors show that fine-tuning an LLM on the filtered data offers performance better than on the entire raw dataset.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of filtering low-quality data is reasonable. The experiments are extensive: The authors compare the the model fine-tuned on the filtered datasets on several types of evaluation suites.", "weaknesses": "- The filtering model ablation study: The performance of Alpagasus seems to highly depend on the quality of the filtering model. How often does GPT-4 as the filtering model makes a mistake? What is the effect of using different filtering model (GPT-3.5, Claude-2, Llama, etc)? As of now, the paper does not offer any study or motivation of using GPT-4 as the filtering model, and simply assumes that is the best choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhDvThsKgV", "reviewer": "Reviewer_BizM", "summary": "This paper proposes a data selection method to remove noisy or low-quality data from the 52k Alpaca data for instruction-finetuning (IFT). Specifically, the authors use ChatGPT to evaluate the quality of the Alpaca 52K instruction tuning dataset and filter out the data samples of low quality. The experimental results show that LLaMA finetuned on the selected dataset outperforms the LLaMA finetuned on the whole Alpaca 52K dataset, as evaluated by GPT-4.", "questions": "- What are the standard deviations of the results obtained from the random data splits? And how many runs?\n- Are the results in Figure 14 also evaluated by GPT-4?\n- What is the agreement between the scores obtained using Claude2 as the LLM filter and those using ChatGPT?\n- How reliable is the LLM filter?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is generally clear to read and easy to follow.\n- The demonstrated method for data selection is simple and easy to understand. \n- The experimental results show some improvement over the baseline finetuned on the whole dataset. \n- The overall experiments are quite extensive.", "weaknesses": "- The authors claim that it costs less to finetune with the filtered dataset by comparing the computation costs in the finetuning stage. However, the authors didn't consider the costs in the data filtering stage where they used the ChatGPT to get the scores for the data samples, which also cost money, computation, and time.\n- Alpagasus is partially weaker than Alpaca (as in Table 2), especially on the large MMLU dataset. The datasets where Alpagasus outperforms Alpaca seem to be small datasets, mostly consisting of a few hundred samples. Moreover, the improvement of Alpagasus over Alpaca is marginal. These issues make the method of Alpagasus not quite convincing. \n- Since the data quality scores are evaluated by ChatGPT (using the OpenAI GPT-4 API?) and the final model output evaluation on test sets is performed by GPT-4, it may lead to a bias towards the GPT-4's preference in the finetuning and testing stages. In other words, GPT-4 selects the data samples it prefers, which can naturally be used to finetuned a model with the same output characteristics that is preferred by GPT-4.\n- It is unclear how many runs are performed for the random splits. The standard deviations of the results are also needed in the paper. \n- In some test sets, the results of the model finetuned on a random split are comparable to, sometimes even better than, those finetuned on the whole dataset. This observation may suggest that the instruction tuning actually suffers from overfitting to the whole training set so the results obtained by training on random splits are even better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7TSaO3McS", "reviewer": "Reviewer_egVm", "summary": "This paper presents a data filtering method based on ChatGPT to filter out the low-quality instruction-finetuning data. With the proposed mechanism, the authors demonstrate using only 9k high-quality data from ALPACA can achieve good performance. The proposed method is evaluated by ChatGPT and human on several language models.", "questions": "* Regarding the second point of weakness. For example, does the granularity of the score affect the performance? What's the effect of system prompts on final performance?\n* Regarding the third point of weakness. Does the reduced quantity of data have a detrimental effect on the robustness of tuned models?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "* The paper is generally well-written and clear.\n* This paper again demonstrates that the data quality in instruction tuning is more important than quantity. \n* Many ablation experiments are provided in Appendix.", "weaknesses": "* As the authors stated in the limitation part, only 7B and 13B models are verified. I understand that this is mainly due to the computational constraint, but one concern is that similar observations may not scale on larger models.\n* Knowing that the LLM models is sensitive to the prompt,  more prompt templates on data filtering are expected to be studied. \n* The evaluation is mostly based on ChatGPT as the judge and the \"Win-Tie-Lose\" metric. More evaluation results are expected, especially on benchmarks related to adversarial and out-of-domain robustness. Evaluating solely on the curated instruction set is not enough to demonstrate the effectiveness of the data filtering method on other dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)