What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
Authors: Wei Liu
Weihao Zeng
Keqing He
Yong Jiang
Junxian He
Venue: ICLR 2024 poster (2024)
Keywords: data selection
instruction tuning
large language models
Primary area: general machine learning (i.e., none of the above)
TL;DR: We perform a comprehensive study to understand the characteristics of data samples that are the most effective for alignment, and propose automatic data selection approaches that lead to data-efficient instruction tuning

Abstract: Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present Deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA models using data samples automatically selected with our proposed approach.  When assessed through both automatic metrics and human evaluation, Deita performs better or on par with the state-of-the-art open-source alignment models such as Vicuna and WizardLM with only 6K training data samples -- 10x less than the data used in the baselines. We anticipate this work to provide clear guidelines and tools on automatic data selection, aiding researchers and practitioners in achieving data-efficient alignment.

Published as a conference paper at ICLR 2024
WHAT MAKES GOOD DATA FOR ALIGNMENT? A COMPREHENSIVE STUDY OF AUTOMATIC DATA SELECTION IN INSTRUCTION TUNING
Wei Liu Weihao Zeng"* Keqing He
Yong Jiang' Junxian He
1 ShanghaiiTech University
Beijing University of Posts and Telecommunications The Hong Kong University of Science and Technology liuwe 4eshangaittehh edu.eddu.c.. on zengwhābupt. wwwp:hhhbpt.eeuu.... edu.cn
3 MMeituan
AAlibaba Group
junxianh& cse.
ABSTRACT
Instruction tuning is standard technique employed align large language mod- els to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engincering in instruction tuning when appropriately selected, only limited data is necessary to achieve superior perfor- mance. However, we still lack principled understanding of what makes good instruction tuning data for aligmment, and how we should select data automati- cally and effectively. this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present DEITA (short for Data-Ej Aafa-Efficient Instruction Tuning for Aligument), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, DEITA performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), DEITA-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT- and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently
INTRODUCTION
In the development of large language models (LLMs), aligning LLMs with human preferences is necessary step for the models accurately understand human instructions and generate relevant responses. Standard approaches for L.M alignment encompass instruction tuning and reinforcement learning with human feedback (RLHF) (Ouyang al.] 2022). Instruction tuning, or supervised fine-tuning (SFT), refines the pre-trained model using annotated instructional data, often serving as the foundational step before RLHF facilitate the model's initial alignment (Touvron al. 20236). RLHF, on the other hand, leverages reinforcement learning to train models based on annotated feedback on their generated responses. While RLHF has underpinned the development of ChatGPT (OpenAl 2022) and GPT-4 (OpenAl, 2023). recent researches suggest that instruction tuning in isolation can offer competitive results (Sun et al. 2024 Zhou al. 2023).
Differing from traditional task-specific fine-tuning where data quantity is paramount, past studies argue that almost all knowledge L.MMs is acquired during pretraining, and instruction tuning is to
"Fqual Contribution, Order determined by random dice rolling. Work done during WL. and WZ's visit to HKUST.
Model checkpoints and data resources available at http::/ggiiuhh.ommhksssnn/ttaatt

---

Published as a conference paper at ICLR 2024
Enhance Tnætracttion Complexity
Response Quality
untiomsd?
(MM
R
R!
R(
((M)
(()
R
Instruction Response
Instruction
Bank Scaze
wo
Score
M
g
g0)
Complexity Bcores
Quality
sort data pool
selected data
diversity-baser selection
(I.R)
Evol Complexity
Evol Quality
Data Selection
Illustration of the data selection approach. We measure data from three dimensions: complexity, quality, and diversity. and R represent instruction and response respectively. For EVOL COMPLEXITY and EvOL QUALITY, we first collect samples with varying complexities or qualities through adopting an evolution- based approach following Xu et al. (2023), then we ask ChatGPT (The term "ChatGPT" in this paper refers to spt-3.5-turho-06133 3) to rank and score the variants of the same data sample for a small seed dataset, and we train our own complexity and quality scorers based on these scores. In the last step, we utilize the trained scorers and detail in 3
Figure 1:
adopt first, diversity-awan approach select the "good" data samples,
align existing model abilities towards desired direction (Zouu al. 2023). As such, relatively small high-quality dataset has been shown to be sufficient to align LLMs well Taori et al. 2023
Wang et al. 2023 Zhou al. 2023 Lu et al. 2023), with dataset sizes spanning from hundreds of thousands to a mere 1000 examples. However, the construction these datasets in earlier research predominantly relies heuristic automation (e.g. distillation from ChatGPT) manual selection, and it remains unclear what defines good data examples for instruction tuning, and how to systematically curate effective dataset that ensures competitive performance with the least amount of data.
In this work, we seek to define the characteristics of "good data" for instruction tuning, based on which we aim to push data efficiency of instruction tuning further in an automatic manner. To this end, we first explore various methods to quantitatively assess data examples from three key dimensions: complexity, quality, and diversity. Our hypothesis posits that the most effective datasets for instruction tuning are those that are complex, of high quality, and diverse. We examine wide range of data measurement baselines and propose new metries, which show stronger correlation with alignment performance after instruction tuning. Drawing inspiration from Xu et al. (2023), we introduce EVOL COMPLEXITY and EVOL QUALITY, which evolve single data point to produce a series of examples varying in complexity quality, then rank and score small subset of them with ChatGPT and train a scorer to predict these scores, as illustrated in Figure Such an wvouution-based method automatically yields examples sorted based on complexity quality, enabling enhanced and finer- -eeee-g data measurements. Combined with our diversity metric via distance of model embeddings, we design a simple strategy to select the most effective data examples from a large data pool. Being simple and effective, our data selection strategy paves the way for a paradigm of data-efficient instruction tuning where fewer training samples can be automatically selected out to yield performance on par with, or even surpassing, models trained on significantly larger datasets.
We present DETTA (short for Eifti-Effiiintt Instruction Tuning for Aligument). a family of models fine- tuned from the LLaMA (Touvron et al., 2023a b) and Mistral (Jiang et al., (023) using our proposed technique to maximize data efficiency. AlpacaEval (Li et al. 2023c), and the Open LLM Leaderboard (Beeching et al. 2023) demonstrate that DEITA is able to outperform or be on par with state-of-the-an instruction following models such as Zephyr (Tunstall et al. 2023), Vicuna (Chiang et al. 2023) and WizardLM (Xu et al. 2023). while using over 10x fewer automatically selected data examples. For example,
experiments in terms of MT-hbench (Zeng al. (023),.
our model based on Mistral-7B achieves 7.22 MT-bench and 80.78% AlpacaEval when trained only 6K data samples with vanilla SET training. After equipped with direct preference optimization (DPO, Rafailov et al. (2023)). DEITA-Mistrl--7B DPO, trained with 6K SFT and 10K DPO samples,
2

---

Published as a conference paper at ICLR 2024
obtains 7.55 MT-Benndee and 90.06% AlpacaEval. In addition to the DEITA model checkpoints, release our light yet effective SFT datasets to facilitate alignment for future researches.
WHAT MAKES GOOD DATA FOR ALIGNMENT?
In this section, we present a comprehensive study of the characteristics of "good data" in instruction tuning. We start by defining the data selection problem ($2.1) under which analyze different characteristics data. Then we introduce the experiment setup ($2.2) explore various metries to
assess data and examine their effects in instruction tuning 2.3
2.5
2.1 THE DATA SELECTION PROBLEM
To investigate the characteristics of optimal data for alignment, we analyze within the framework of data selection. In this context, delve into various metrics evaluate data, and then employ these metrics to select subset of data samples for instruction tuning. The resulting alignment performance serves as an indicator of whether the target metric can identify effective instruction tuning data examples. Formally, given a large instruction tuning data pool, X = {o 22, In.. where I; represents an individual data sample in the form of instruction-response pair. We aim select a subset sQm) of size m from X, using a selection strategy denoted by π. m is the subset size and correlates proportionally with the computation consumed in instruction tuning, thus we also refer m as the data budget. Typically, we define a metric to assess data and select data samples based on the metric. Denote the alignment performance after instruction-tuning as Q, the optimal data selection strategy nt" with data budget m satisfies:
π = arg max (()()))
In our empirical study following, we will explore a diverse range of data evaluation metrics and their corresponding data selection strategies, by selecting s according to certain metrics and performing instruction tuning on it. Next, we detail our experimental setup.
2.2 EXPERIMENTAL SETUP
In this section, perform controlled studies single metric to evaluate data a time, and follow the procedure as (1) selecting subset S(() from a data pool based on a given metric, (2) instruction-tuning pre-trained model using s(=)) and (3) evaluate the instraction-followee following abilities of the obtained models. Given data measurement metric, in this paper, we keep our selection algorithm simple as possible to maintain its practicality (e.g., selecting the examples with the largest complexity scores), while we leave more advanced selection algorithms as future work.
To investigate data selection from large data pools, we construct two data pools with distinct properties to mimic different practical settings: (1) Xeotas which is constructed by ensembling the training datasets of the -tateoff-heear aligned LLMs. This represents the setting where data pool that is relatively complex, diverse, and of high-quality is available, and we aim to further improve the data efficiency in this case. Specifically, WizardLM (Alpaca), WizardLM (ShareGPT), UltraChat (Ding et al. (202)), and ShareGPT (Chiang al. 2023), ) resulting in dataset of 300K samples; and (2) Xbases which mimics the scenario where the available data pool is overall lower-quality and redundant. This may be closer to the actual setting that people would encounter in practice. For Xbase, we have utilized data from Alpaca (Taori et al. 2023). ), Dolly (Conover et al., 2023). OAssit (Köpf et al. 2023), and FLAN 2022 (Longpre et al.
Data Pools:
follow [u et al. (2023) combine the datasets
2023) to construct a dataset 100K samples. Such quality-
separation of these open-source instruction tuning datasets roughly aligns with the analysis results in Lu et al. 2023): Li et al. (2023b).
Statistics of the two data pools are summarized in Table
Training and Evaluation: In the study of this section, we fine-tune the LLaMA-1 I 13B on the instruction tuning dataset unless otherwise specified. We assume a data budget m of 6K. We use the same hyperparameters for all as detailed in Appendix A To evaluate alignment performance, we utilize MT-Bench, a challenging benchmark that commonly adopted to assess the instruction- following ability. MT-bench consists of multi-turn conversations across various domains such as
3

---

Published as a conference paper at ICLR 2024
Model
Nanta
Xinae
Random Selection
5.84
4.93
Data Pool
Dataset Source
Sample Size
Instruction Length
5.89
4,00
Perplexity
4.06
ShareGPT
58 K
IFD
5,91
2.46
X sata
Ultrahhat
105 K
Instag Complexity
6.18
4.98
WizardLM
143 K
Direct Scoring (PooI=S0K)
5.16
4.87
Alpaca
$2
Instruction Node (Pool=50K)
5.65
4.82
Dolly
15
EVOL COMPLEXITY (Pool=50K)
5.73
5.29
X base
OAssit
EVOL COMPLEXITY
6.27
5.57
FLAN 2022
23 K
Table 2:
MT-bench of different complexity metrics. All methods select 6K samples. "Pool=50K" denotes the data selection is conducted in 50K-sized subset due the cost of using ChatGPT to annotate the entire pool. We include the results of our method on the 50K data pool to make fair comparison with baselines.
Table I: Statistics of data pools Xsota and Xiuse- The Dataset Source indicates the source of the data used for sampling. The Sample Size represents the number of samples in the respective dataset.
writing, reasoning, math, and coding. GPT-4 is employed the judge to model responses in MT-bench, which was found to produce a high agreement with humans (Zheng et al. 2023). )
2.3 FROM THE COMPLEXITY PERSPECTIVE EVOL COMPLEXITY
It is often believed that long, difficult, and complex data samples more beneficial for instruction tuning (Zhao et al. 2023 Cao et al. 2023). For example, Xu et al. (2023) prompt ChatGPT to "evolve" data samples to deliberately increase their complexities, which leads the state-of-the-ar open-source alignment models WizardL.M. In this section, we systematically study various metrics to assess the complexity of data and aim to identify the notion of complexity that contributes the most to the instruction followiwn ability. Concretely, we only consider the complexity dimension and define the selection strategy Teamplezity as selecting the m examples with the highest complexity scores.
Baselines: We study several existing methods as our baselines for complexity metrics: (1) Random Selection selects examples randomly; (2) Instruction Length uses length of instructions as the metric for complexity; (3) Perplexity of the responses computed with the pre-trained model in zero-shot manner used as the metric, and a large perplexity score typically entails difficulty of the data sample; (4) Direct Scoring (Chen al. 2023) directly prompts ChatGPT to score the difficulty and complexity of instructions; (5) Instruction Node (Zhao et al. 2023) uses ChatGPT to transform instructions into semantic and then adopts the number of nodes in the tree the complexity measure; (6) Instag Complexity (Lu et al. 2023) first utilizes ChatGPT to tag the samples based on semantics and intentions, then trains LLaMA-based tagger the ChatGPT tags to tag data. They use the number of tags as a proxy for complexity. We adopt their public tagger that is LLaMA-2 7B model tag the data. (7) IFD (Li et al. 2023a) is a new complexity metric computed based on the response loss. We note that Direct Scoring and Instruction Node are not scalable since they require ChatGPT to annotate the entire data pool, which is expensive. To save cost, we randomly sample 50K examples first from each pool and apply these two baselines. The respective prompts of these baselines (if applicable) are shown Appendix E.T
Evol Complexity: Taking inspiration from Evol-tnstruct which utilizes ChatGPT to evolve exam- ples to become more complex, we propose EVOL COMPLEXITY, a complexity measure based on evolution. Specifically, we collect a small-scale seed dataset, D = [(T{)) )
(T (0)
R }}
where (1 R represents an instruction-respone pair. For each instruction sample (0) we use
the In -e-eetth Evolving Prompt (please refer to Appendix E.2) for the prompt) from Xu et al. (202))
to enhance the complexity through techniques such as adding constraints, deepening, concretizing and increasing reasoning steps. After M iterations, we obtain set of instructions across different
complexities for It. { ....,........ "..,,,,(M))) Here we set M as 5 to obtain variations in total.
As illustrated in the left part of Figure [ we then ask ChatGPT to rank and score these 6 samples (prompt in Appendix E.2). obtaining the complexity scores a corresponding to the instructions. We emphasize that, distinct from direct scoring, we give ChatGPT all 6 samples within one prompt t
InsTag
4

---

Published as a conference paper at ICLR 2024
samples represent different evolution stages of the same original sample and such a scoring scheme helps ChatGPT capture the small complexity differences among them, which leads to complexity scores to achieve finer-grained complexity differentiation among samples. We find that this is critical since otherwise, ChatGPT tends to assign similar scores to most examples, as we showcase in Appendix [B] After obtaining ChatGPT scores the small seed dataset, an 1 7B model to predict the complexity score given the input instruction. In multi-turn dialogues, we score each turn separately and use the sum of them as the final score. Across this paper, we use 2K examples randomly sampled from the Alpaca (Taori et al. 2023 as the seed dataset.
use the scores train
Table 2 presents the results of selecting the 6K data samples from x and Xbase using various complexity metrics. Our Evol-Complyxity leads to the best alignment performance. We observe that while Instag Complexity performs well on X its performance on Xbase is only slightly better than Random Selection. In contrast, EvOL. COMPLEXITY achieves superior performance on both datasets, indicating strong robustness across different dataset pools. Our method also has significant advantages compared to methods that heavily rely annotating with ChatGPT, such as Direct Scoring and Instruction Node. The results also suggest that instruction lengths are not good indicators of preferred data by alignment. Interestingly, perplexity, intuitive measure for complexity, produces much worse results than the random selection baseline. Through further investigation, found that samples with large perplexity typically exhibit very short responses.
Results:
2.4 FROM THE QUALITY PERSPECTIVE EVOL QUALITY
Generally, LLMs that deliver accurate, detailed, and helpful responses are favored by people, as indicated by Zheng et al. (2023). In this section, we carry out a controlled study to examine various metries used for evaluating the quality of samples. Drawing parallels with EVOL COMPLEXITY, we devise a selecting strategy Qquality based on quality, enabling us to select m samples with the highest quality scores according to different measurements. Following this, we will introduce the examined baseline and our newly proposed methods for assessing quality.
We examine the existing methods that serve as baselines for quality assessment: (1) Random Selection selects examples randomly. (2) Response Length employs the response length as the quality metric. (3) Direct Scoring (Chen et al., (023)) prompts ChatGPT to evaluate the accuracy of the response to the instruction directly. The associated prompt is displayed in Appendix E.3
Baselines:
Evol Quality: In a manner akin to EVOL COMPLEXITY, we introduce EVOL QUALITY to augment
the discernment of quality measurement. For given data sample (T(0) Re(") ), we prompt ChatGPT to clevate the quality of the response in an evolved way (Appendix (epeendixE.44foor [E.4 the prompt). This primarily
involves enhancing helpfulness, augmenting relevance, enriching depth, fostering creativity, and
supplying additional details. After M iterations, for the same instruction T((), we procure a set
of responses spanning various qualities for Rg, denoted {R.) COMPLEXITY we set M to 5.
(0) R R.) Similar to EvOL
As demonstrated in the middle part of Figure T we then instruct ChatGPT to rank and score these responses in terms of the response quality, thereby obtaining a quality score q corresponding to each response (refer the Appendix [E.4 for the prompt template). Similar to Evol Complexity, this scoring approach is able to aid ChatGPT in discerning subtle differences between responses of varying qualities, and thus offering more nuanced distinction of quality scores, as exemplified in Appendix B We utilize scores derived from the seed dataset to fine-tune LLaMA-1 7B, enabling it to predict quality scores based on the provided instruction- response pair. The seed dataset is the same as the one for EvoL COMPLEXITY with 2K random samples from the Alpaca dataset.
Results: Table 3 presents the experimental results of selecting the top 6K data from Xsata and X Xbase respectively. The proposed EvOL QUALITY approach consistently exhibits superior alignment performance. We notice that the pool with a higher quality variance, Xbase. is more influenced by quality metrics, which is intuitive since many low-quality examples are present in such pools and will hurt the performance significantly. This result implies that quality is a necessary dimension to consider especially when working with data pools that have a substantial number of low low-quality examples. We also observe that the response length positively correlates with the final alignment performance, yet its effect is not substantial for datasets with already high quality, such X sota
5

---

Published as a conference paper at ICLR 2024
Model
sota
Xbase
Random Selection
5.84
4.93
Response Length
5,94
5.65
Model
X
Xbase
Direct Scoring (Pno1=50K)
5.61
4.44
Random Selection
5.82
4.34
EVOL QUALITY (PooI=50K)
5,85
5,29
EVOL QUALITY
6.19
5.67
Instag Diversity
6.10
4.46
Repr Filter
6.17
4.68
Table 3: MT-bench different quality measurements. methods select 6K samples for training. "Pool=50K" de-
Table 4: MTThench scores of different diversity measurements. All methods select 6K samples for instruction tuning.
notes the data selection procedure is conducted in 50K- sized subset due to the cost of ChatGPT to annotate the
entire pool. We include the results of our method the
50K data pool to make a fair comparison with the baselines.
2.5
FROM THE DIVERSITY PERSPECTIVE AN AMBEDDING-BASER APPROACH
As general principle, advanced L.M should be adept at handling various requests from humans. Therefore, the data used for instruction tuning is desirable to maintain maximum diversity. However, real-world data often exhibits redundancy (Abbas al. 2023). In this study, we explore the impact of data diversity on alignment by conducting controlled experiment and then introduce a simple yet effective strategy Ediversity to maintain diversity and conciseness in the selected subset of data.
Setup: In this work, we propose an iterative method to ensure the diversity of selected data. The iterative method picks sample x; from the pool X one by one selected dataset when J; contributes diversity to S. The process continues until the budget m is reached or all x; in X have been enumerated. To clarify, we formulate the benefit of diversity brought by a newly considered
sample z; as an indicator function 1 [[,,,, S)] which is equal to I only if F(x,, 8) is True and 0 otherwise. F, which we will define later, is the function to assess whether x, exhibits diversity in relation to the selected dataset S. Only when [[F(2,, S)] equals to 1, will x; be added to S. Additional setup details are in Appendix
Baselines: In addition random selection, we further assess Instag diversity (Lu et al. 2023). which is designed iteratively to ensure diversity within the selected dataset. It utilizes the growth the tag set for S as a metric to define the function F. Specifically, formulates F, [T's Te, > [T's), where T's represents the set of all tags in S, and Ta, represents the tags associated with Ti.
We examine our embedding-bassee approach, which we refer as Repr Filter. Specif- ically, we take the distance between sample and its nearest neighbor in S as the metric to define F. We leverage the 1 13B model to encode the sentence and compute the cosine distance then F : d < T. where E (0, 1) is threshold hyperparameter. This means that we consider an example x; could increase the diversity of when the embedding distance between 2; and its nearest neighbor is smaller than threshold. During data selection, we first sort the data pool X according to complexity and quality scores that we will detail in $3.1 then we examine each sample one by one and put I; S if d < where S is initialized empty. This process stops until the size of S reaches the data budget m. We set threshold 7 as 0.9 in all the relevant experiments across this paper, while we provide analysis results on and different sentence representations in Appendix C.T
Repr Filter:
Results: Table presents the results across different diversity strategies in X and Xtase respectively. Comparing Random Selection with the other strategies ensuring diversity, the model trained with randomly selected data significantly underperforms others, which demonstrates the key role of diversity. And our approach able to outperform Instag Diversity on both data pools.
DEITA- DATA EFFICIENT INSTRUCTION TUNING FOR ALIGNMENT
Based on our exploration in [2 we propose simple approach to select data samples considering all the three dimensions, complexity, quality, and diversity. Utilizing the selected subset, we train our model DEITA, as an attempt towards extremely Data- Inta-Efficient Instruction Tuning for Alignment. Below, we detail our data selection method and models.
3.1 METHOD

---

Published as a conference paper at ICLR 2024
Algorithm 1 Score-First, Diversity-Aware Data Selection
1: Input: The data pool X, data budget m
2: Output: The selected subset S
3: Initialize Empty Dataset Stpers
4: Sorting X with the combined complexity score and quality score 8 =
5: Getting the sorted Pool X";
6: for Each Sample E x" do
d(x,, S) denotes the distance between æ nearest neighbor
d(xxx, Stipl < then
Stm) + -Sapum U (z)
10:
else
11:
Continue
12:
end if
13:
X
X [x)
14: if | equals then
Break
16: end if
17:
end for
Score-First, Diversity-Aware Data Selection: While there are various ways of combining com- plexity, quality, and diversity measures, we aim to keep it as simple as possible to be practical. Intuitively, desirable select samples with high complexity and quality scores but maintain the diversity of the set. To this end, we propose a score score-first, eeverse-aware data selection strategy, denoted as TDDITAA Our strategy incorporates new evol score s that combines complexity and quality by multiplying the complexity score with the quality score dialogues, we calculate this score for each turn, summing them obtain the final score for the entire conversation. Next, we sort all samples in X using s, yielding the sorted pool X (xf .... zt)) where z represents the sample with the highest evol score. Beginning with $ Trems = (zī), we iteratively select data from X /S redundant samples for S Smmm. By integrating the evol score and the REPR FILTER, our approach guarantees complexity, quality, and diversity in the resulting dataset. Our data selection approach is illustrated in the right part of Figure I summarized in Algorithm
C q. For multi-turn
by one, following the REPR FILTER strategy. and discard
Training DEITA: We train DEITA using the selected dataset with samples. We denote the resulting models as DEITAM- In this paper, we train DEITA models based on LLaMA-I-13B, LLaMA-2-13B, and Mistral-7B respectively, and the training details are described in Appendix
3.2 EXPERIMENTAL SETUP
We train the DEITA models with data budget of 6K and 10K examples respectively. We select data samples from the data pool X We adopt MT-Bench, AlpacaEval, and the Open LLM Leader- board as our benchmarks for automatic evaluation. The Open LLM Leaderboard consists of four classification tasks: ARC (Clark et al. 2018). HellaSwag (Zellers et al. 2019). MMLU (Hendrycks) et al. 202T). and TruthfulQA (Lin et al. (022). We also provide human evaluation results in Ap- pendix D We first take LLaMA-1-13B
the backbone and compare DELTA with other data selection approaches such as LIMA (Zhou et al. 2023), Alpagasus (Chen et al. 2023), and TAGLM (Lu et al. 2023). Then we compare DEITA models based on LLaMA-I-I3B, LLaMA-2-15 1-BBBBBBBBBB and Mistral-7B with other top-performing open-source models, such as Vicuna, WizardLM, Mistral-Instruct (Jiang al,, 2023), and Zephyr. this paper, we mainly focus on models aligned using SFT without preference training since our contributions lie on data selection in the SFT stage, while we also run direct preference optimization (DPO, Raaailov et al. (2023) on top of our best SFT model report stronger performance for reference points. For DPO training, randomly sample 10K comparison data pairs used in Zephyr that originally obtained from the UltraFeedback dataset (Cui et al. 2023).
3.3 RESULTS
Main Comparison: We first compare DEITA with other data selection approaches in Table 5 where DEITA-LLaMAIa clearly outperforms other approaches by large margin. In Table 6 train
7

---

Published as a conference paper at ICLR 2024
Model
Data Size
MT-Bench
AlpacaEval(S)
Random
6K
5.84
73.91
Alpagasus (Pool=50K)
6K
5.61
71.21
LIMA
4.29
41,98
TAGLM
6K
6.09
72.80
DETTA LLM-IIIS
6.46
77.08
Table 5: Comparison of different data selection approaches, the backbone is LLLMM-1-1330 For Alpagasus, we could not ChatGPT to score all the examples the pool due the cost, thus score 50K random examples and select. i denotes the results ohtained by using their released LLaMA-7B tagger model for fair comparison.
Model
Data Size Alignment
MT-Bench
AlpacaEval(S))
Proprictary Models
GPT-4
8.99
95.28
Claude-v2
8.06
91.36
gpt-3.5-turbo
7.90
89.37
Open-sourced Models based on LLaMA-1-13B
Alpaca-12 13B
SFT
4.53
WizardLM-13B
70K SFT
6.35
75.31
Acuna-133-vl..3
125K / SFT
6.39
82.11
TAGL.M-13B
6K / SFT
6.09
72.80
Random-Select
10K / SFT
6.03
71.52
DEITA-LLaMAI-13R. 1-13BB
6K / SFT
6.46
77.08
DEITA-LLaMAI-1330 --13--
1OK/ SFT
6.60
78.01
Ooun-sourced Models based on LLaMA-2-13B
LLMMA2-13B-CH
> 100K / SFT IM RL.F
6.65
81.09
Vicuna-13B-v1.5
125K / SET
6.57
78.80
TÜLÜ 13B
326K SET
78.90
TULÜ 2 DPO 13B
326K SFT 60K DPO
7.00
89.50
Random-Sclec
10K SFT
5.78
65.19
LATTALLLMM2--33Ba.
6K / SFT
6.65
80.75
DEITA-LLaMA2-138
10K SFT
6.79
81.09
Open-sourced Models based on Mistral-7B
Mistral-7B Nistral-7B-Insttuer--e Instruct--OO.
6.84
69.65
Mistral-7B- -nn-nrrrcctt-00
7.60
93.65
zephyr-bcta-sfft
200K SET 200K / SFT 60K DPO
5.32
75.
zephyr-beta
7.34
90.60
Candom-Select
10K SFT
5.89
$6.90
DEITA-Mistral-7BBa
6K / SFT
7.22
80.78
DEITA-Mistral-7B... 10
10K SFT
7.32
81.67
DEITA-Min SeErYA-Misraal-BB... DPO
6K SFT +1KK / DPO
7.55
90.06
Table 6: Results of different instruction models on MT-Bench and AlpacaEval. Best SFT-only numbers within the same hase model are bolded, while the overall best numbers are underlined. f denotes the results obtained by using their released I.L.MA-BB tagger model for fair comparison. Zephyr-beta-sft" the official checkpoint after the phase supervised fine-tuning (SFT). notice the performance this checkpoint lower than expected. We speculate the reason is that this checkpoint is not the best SFT checkpoint reported in their paper since the checkpoint is used for further DPO training.
1-MMA-1-13B, LLaMA-2-1328 and Mistral-7B 7B with our selected 6K and 10K data respectively, and compare with other state-of-the-an SFT models well as random data selection baselines (Random- Select). Across all three backbone models, the SFT aligned DEITA models outperform almost all other SFT-aligned models. DEITA models based LLaMA-2 even outperforms LLaMA2-13B-CCI undergoes RLHF training with carefully crafted human annotations. Notably, DEITA-M CostT--Msttaal-B... 10x based on Mistral-7B achieves a 7.32 MT-Bench score, which is the state-of-the-ur result among all open-source SFT models at 7B and 13B sizes. In the meanwhile, we note that the gains of DEITA on AlpacaEval are not apparently consistent with the gains on MT-Bench. To analyze such difference between MT-Bench and AlpacaEval, we plot the standard radar plot of the 8 substasks on MT-Bench, as shown in Figure 3 It clearly demonstrates that the DEITA Mistral models achieve high MT-Bench scores due to the enhanced performance advanced abilities such as coding, math, and reasoning, which are not prominent in AlpacaEval. When equipped with DPO training. our DEITA-Mistral- 7B +DPo variant achieves 7.55 MT-Bench and 90.06% AlpacaEval scores, which is comparable to zephyr-beta that trains on 30x more data, and slightly lags behind the recent Mistral-7B BB-I--TIuct-tt-OO model whose alignment approach and data are not public.
that
8

---

Published as a conference paper at ICLR 2024
Model
Data Size Aligument
ARC HellaSwag
MMLU
TruthfulQA
Average
Open-sourced Models based on LLaMA-1
LIMA
1K/ SFT
59.22
84,25
49.60
46,20
59.82
WizardL.M-33B
/ SFT
57.25
80.88
52.92
50.55
Micuna-13B-v/
125K SFT
54.61
80.41
52.88
52.14
60.01
Random-Select
10K / SFT
55,80
79.95
47.35
57.44
60.14
DEITA-LLaMAI-I
10K / SFT
59.47
82.01
60.60
55.03
64.27
Open-soureed Models based on LLaMA-2
Micnn--13Bvile
125K SFT
57.08 61.52
81.24
56.67
51.51
61.63
Random-Select
SFT
83.69
55.22
44.84
61.32
DEITA-LLaMA2-ISBuur
10K/ / SFT 58.87
82.08
55.33
54.57
62.71
Open-sourced Models based Mistral-7B
1
54.52
75.63
55,38
56.28
60.45
zephyr-beta-sft
200K SFT 200K SFT 60K DPO
57.68
81.98
61.04
43.00
60.93
zephyr-beta
62.03
84.52
61.44
57.44
66.36
Kandom-Select
/ SET
55.38
79.16
58.73
53.59
61.72
DEITA-Mistral-BBek
6K SFT
57.76
80.29
61.90
59.82
64.94
+DO
6K SFT +10K DPO
66.21
85.42
60.66
67.14
69.86
Table 7: Results the Open LLM Leaderboard. Data size default represents the number of examples in SFT
unless specified otherwise.
Willing
Random TAGLMM M
model
Mistrak- wwwraa/--aaardmmmmm Lnitt si Nax Defte-D00 (4iatrai777))
1k
6.k
12
24
All kl
Figure 2: Data Data-scaling results MT Bench.
Figure 3: Radar plot of detailed scores for DEITA models and major baselines on subtasks on MT-Bench.
The X-axis represents the samples used.
We compare DEITA to other SOTA models and the random data selection baselines Saaando--Selctt on Open LLM Leaderboard, results shown in Table 7 Our DEITA SFT models - while trained with only 6K or 10K data achieve the best average results among SFT-aligned models across different backbones. Further DPO training greatly boosts the performance of DEITA-Min Mistra--B by around points on average and helps surpass the Zephyr.
Results on Open L.M Leaderboard:
Data Scaling: In order to investigate the data scaling effects of different data selection strategies, we experiment with subsets of X Xsota with various data budget m. Figure illustrates that our DEITA models consistently deliver the best data selection performance across different data volumes. Remarkably, DEITA achieves comparable results to using all the 300K training samples with only 3K samples, 100x data reduction. Interestingly, the quantity of selected data increases, the final performance initially shows an upward trend but eventually declines. This suggests that even for a relatively complex. diverse, and high-quality data pool like Xeote the proportion of truly "good data for alignment" is limited. This phenomenon confirms that the performance alignment does not necessarily improve even though we add more data and use more computing, implying the importance of data selection.
find that under our data selection approach, as
CONCLUSION
In this paper, we thoroughly investigate the question what makes good data for alignment. Our research encompasses three controlled studies conducted across three dimensions: complexity, quality and diversity. Throughout these studies, we propose new methods for automatic data selection and train models, DEITA, the selected data. Experimental results demonstrate that DETTA is able to achieve superior or comparable performance the state-of-the-ar open-source models with 10x training samples. We release selected data to effectively align models more efficiently.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This project is partially supported by the WeiXin Group in Tencent.
REFERENCES
Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S. Morcos.
Semdedup: Data-efficient learning at weh-scale through semantic deduplication. In ICLR 2023
Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023. URL https
//openreview.oo hwtp:/iwwwwweeeeeeeeeemmmmeee wwtp:/rrumtettem//./mo udd2BB Shna
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https
// musginggface.co/spaeess httpee//ggggggggaaaeeeeepepeeee
Leaderboarrr Lee.eeeroeaneeederrrrrrrraaa 2023.
Kush Bhatia, Avanika Narayan, Christopher De Sa, and Christopher Ré. TART: A plug- and-play transformer module for task-agnostic reasoning.
Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. nips cetpiips..ccpppp__ffeeeeeeeeee http://leeeeeapee/////a/aaee
2023, 2023. URL lece 1ecor/00222288335022224444471122
html
Yihan Cao. Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models. ArXiv preprint, abs/2307.06290, 2023. URL https
7/arxiv. org/ abs 2210/3337.0000 06290
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. ArXiv preprint, hts/22377.0870.. 2023. URL https: :/tp::///arxiis ozg/ /btgg//b/// 2207 08701
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018.
Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng. Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, al. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
Tri Dao. FlashAttention Faster attention with better parallelism and work partitioning. ArXiv preprint, abs/2307.08691, 2023. URL http::tt:://at //axivv..v org/ aass//bbbaae 2307 08691
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conver- sations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pp. 3029-3051, Singapore, 2023. Association for Computational Linguistics. doi: 10. 100.18653/w/2223.com 2023. emnlp-nannee... 183
main. 183. URL
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representattons, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htRp:hhttt////oeeeeee ew. ww..et..weeeeeeeeee.eeeeeeeeee ftwee//ooummseeemm d=d 3Gmo.
10

---

Published as a conference paper at ICLR 2024
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversatios-ddemorrttinngg large language model alignment. ArXiv preprint, abs/2304.073277, 2023. htRp::
org/ abs / 2tt.//334..000 07327
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting IIm performance with self-guided data selection for instruction tuning. ArXiv preprint, abs/2308. 12032, 2023a. URL https?
7/arxiv. wttp://g////aeeoe /308 12032
Xian Li, Ping Yu. Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. ArXiv preprint, abs/2308 06659, 2023b. URL httphhtss///xi....rrrtree abs wttpp////wwrrr////...... 06259
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooinng models. https:: // //gi thub. com www.oo/tt/ttto///ooooooooo lab/a Sacca_aaeea _ 2023c.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18665/11/2222.acc- long.229. URL https:ttpp.... : / ac anthologyyy..o...ooooooooo www.org/200022 ac 1 long
229
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le. Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023. 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22631-22648. PMLR, 2023. URL http::tttssssseeeeeeeeeeeeeeeeee ngs html
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv e-prints, pp. arXiv-2308, 2023.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB:
Massive text em- bedding benchmark. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter the Association for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, 2023. Association for Computational Linguistics. doi: 18.18653/ v1/2023.eacl-main 148. URL https: http:///aaaaaahllaoooc wwg..orrg/02..... eacl ma in. 148.
OpenAl Chatgpt: Optimizing language models for dialogue. OpenAl Blog, 2022. URL https hoopenai.combblgg///eeooo httpl//ggchhtttpp///ho
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang. Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Offload: Democratizing Billion-Scall -calle model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
11

---

Published as a conference paper at ICLR 2024
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloving Ilama model. https://gitne: cow hanm/tttsullab/// wttpuaaaaaaaaaaaaaaaaae stanford alpaca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023a. URL https
//arxiv. otg/ abs/ 300.1397.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, fbss/200..09285 2023b. URL https: //arxiv
org/ hbr/abb//a 2307.300 09288
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. ArXiv preprint, abs/2212.035333, 2022. URL
2212. 03533
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self- generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484- 13508, Toronto, Canada, 2023, Association for Computational Linguistics. doi: 10. 18653/11/2023. acl-long.755 754. URL hattps://aclaatthloo....
/ttg////220232 acl-long. 754
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models follow complex instructions. ArXiv preprint, abs/2304. 12244, 2023. URL htttht/ss///axxt org. abs 2
Rowan Zellers, Ari Holtzman. Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.),
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ P19- 1472. CtlLhtts::// http::/:s///aaaaahooooossoo
wwg/plgg/i.0 P19 1472
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L Zhang, A preliminary study of the intrinsic relationship between complexity and alignment. ArXiv preprint, Abs/2308.05696 2023. /tt.sttss://arxivv.oo
abs /200. 05696
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging aud-ase--judge with mt me-bench and chatbot arena. ArXiv preprint, abs/2306. 056685 2023. URL
org/as.
2306. 05685]
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2024
A SETUP DETAILS
Diversity-based Data Selection: In the control experiment of 22.5
meticulously manage the
complexity and quality of the selected data to conduct a rigorous analysis of diversity's impact and facilitate fair comparison between various methods. This is achieved by ensuring that the product of the mean complexity score and quality score of selected data S closely align with the respective means of the complete data pool X. by limiting a deviation of a maximum value of 2 from the mean. Through this way, we first select subset of samples from the data pool to construct new pool, where the examples have similar xx q scores. Then we explore different diversity selection approaches.
Training DEITA: We utilize four/eight NVIDIA Tesla A 100 GPUs to train 7B/13B models. To facilitate parallel training, we employ DeepSpeed Zero Tero-Stage? (Ren et al. 202T] and FlashAttention- 2 (Dao) (023). For the integration of multi-turn conversations, we use the Vicuna-style template. In all experiments of this paper, the training parameters are set with maximum input length of 2048. In terms of DEITA models based LLaMA-1-I 1 1-B,, we set the batch size to 128, training epochs to 6, learning rates at Ie-5, and a warm ratio of 0.03. For DEITA models based on LLaMA-2-
we set the batch size to 128, training epochs to 6, learning rate to 2e-5 and warm ratio to 0.1 follow [u et al. (2023). Regarding DEITA models built on Mistral-7B, we follow the hyperparameters from Tunstall et al. (2023) with batch size 512, learning rate 2e-5, warm ratio of 0.1 using cosine warmup scheduler for SFT, and batch size 32, learning rate 5e-7, a warm ratio of 0. using linear warmup scheduler for DPO. Due the use of significantly less training data, ensure adequate training, we increase the epochs for SFT training to 6 and the epochs for DPO to 9.
B CASE STUDY
Table and Table display the instances of scoring complexity and quality using the Direct Scoring method and the Rank & Scoring method we proposed, respectively. An analysis of these instances indicates that Direct Scoring tends to assign similar and excessively high scores. contrast, our method is capable of discerning the subtle variations in complexity or quality among different samples, thereby providing scores with distinct differentiation.
C ANALYSIS
C.I ANALYSIS OF REPR FILTER
We conduct experiments using various thresholds (7) ranging from 0.8 to 0.9 and different sentence encoding methods to assess their impact on the Repr Filter. We explore two methods for encoding sentences: one involves using the representation from the model which will be trained after data selection, while the other employs well-trained sentence embedding models such as E5-Large-V2 (Wang al. 2022). We refer the former method as the Model-buxer approach and the latter the Comanticcbuseed method. We utilize 1LaMA-1 13B as the representation for the Model-based method and Largeee- to represent the Semantic-based method. For Semantic- based, encode one sentence by averaging all embeddings of its tokens due to the limited context length for the model.
Figure presents the results of our Repr Filter across different thresholds 7 and different sentence representations. The findings demonstrate the robustness of the Model-based encoding method across different thresholds, maintaining substantial margin over the baseline method. Conversely, the performance of the Semantic-haser method exhibits significant decline with adjustments to the superior performance of L.5-Large--V V2 in MTEB benchmark (Muennighoff al., (023), we speculate (Bhatia et al.) (203)) the representation from the Model-based method not only encodes semantic information within the sentences but also reflects certain characteristics of the data for subsequent instruction tuning, we prioritize its utilization.
Due
D HUMAN EVALUATION
Setup: Since it is difficult to conduct human evaluation on MT-Bench due to the challenging problems, utilize the LIMA test dataset and randomly sample 100 of them as evaluation
13

---

Published as a conference paper at ICLR 2024
Case A
Direct Scroing
Ranking & Scoring
Instruciton
Rate the given restaurant based your experience. Restaurant: Pizzeria Garden Using your experience as a guide, provide comprehensive evaluation of Pizzeria Garden, taking into account factors such as the quality of the food, the level of service provided, the ambiance of the restaurant, and any other relevant aspects that may contribute your overall impression. Please provide detailed analysis that requires multiple steps reasoning to arrive at final rating.
8
3
Based on your extensive knowledge and expertise, please conduct a thorough and intricate assessment of Pizzeria Garden, encompassing various elements such as the excellence of the cuisine, the degree of attentiveness exhibited by the staff, the atmosphere mood of establishment, and any other pertinent factors that could influence your overall perception. Your evaluation should entail a meticulous analysis that necessitates multiple layers of reasoning to arrive at a conclusive rating. In order provide thorough and nuanced evaluation of Pizzeria Garden, we require your extensive knowledge and specialized expertise. Your assessment should encompass wide range of factors, including but not limited to the quality and inventiveness of the cuisine, the level of attentiveness and professionalism exhibited by the staff, the ambiance and overall atmosphere of the establishment, well any other pertinent elements that may influence your overall perception. Your analysis must be meticulous and multifaceted, necessitating profound level of reasoning and critical thinking to arrive at a conclusive rating.
8
4
8
5
Case B
Direct Scroing
Ranking & Scoring
Instruction
Create birthday wish for somcone who loves animals.
8
Craft heartfelt birthday message for an individual who has deep affection for a specific type of animal, such as dogs, cats, birds, or reptiles. Incorporate their passion for this ereature into your well wishes to make their special day even memorable and meaningful.
Compose sincere and thoughtful birthday greeting for an individual who holds a profound fondness for particular species of fauna, be it canines, felines, avians, or boldeblooded creatures. Infuse their ardor for this living being into your felicitations to elevate their celebratory occasion to higher level of significance and remembrance.
7
4
Compose a sincere and introspective birthday greeting for an individual who harbors a profound and enduring fondness for particular species of fauna, be it domestic canines, felines, avians, reptiles. Integrate their ardor for this animate being into your felicitations to amplify the significance recollection of their festive occasion. Furthermore, contemplate incorporating individual reminiscences or minutiae about the selected genus to further individualize the message and showcase your comprehension and admiration of their passions.
8
6
Table 8: Cases of complexity scores generated by different methods, including Direct Scoring and the Rank Scoring method we proposed.
112
602
340
332
Figure 4: Results of setting differentt r and leveraging different sentence representations in Xaota
prompts. Initially, we attempt to publish our evaluation on MTurk with a strict selection setting, as deseribed Li et al. (2023b). However, we encountered challenges the answers from MTuurkers
14

---

Published as a conference paper at ICLR 2024
Instruction A Find the title of book about a political movement in the 1940s.
Direct Scroing
Ranking & Scoring
Response
The title a book about political movement in the 1940s is
"A Matter Of Resistance: The Struggle for Social Justice and Human Rights in India's ' 1940s".
4.5
2
1940s "A Matter Of Resistance: The Struggle for Social Justice Human Rights in India's 's 1940s". This book provides a detailed account of the struggle for social justice and human rights during that time period. sheds light the challenges faced the people and the efforts made to overcome them. The book is a valuable resource for anyone interested in understanding the political climate of the 1940s.
One book that delves into the political movement
4.5
3
One book that delves into the political movement of the 1940s is "The Struggle for Social Justice and Human Rights in India'," 1940s". This book offers a comprehensive account the challenges faced by the people during that time period and the efforts made to overcome them. It provides valuable insights into the political climate of the 1940s, making it an excellent resource for anyone interested in gaining deeper understanding of this The book is highly recommended for those who want to explore the political movement the 1940s greater depth.
4.5
4
The Struggle for Social Justice and Human Rights in India)?s 's 1940s" book that provides detailed account of the political movement during that time period. offers comprehensive analysis of the challenges faced by the people and the efforts made to overcome them. The book provides valuable insights into the political climate of the 1940s, making it excellent resource for anyone interested in gaining deeper understanding of this era. covers various aspects of the political movement, including the key players, their ideologies, and the impact their actions. Overall, "The Struggle for Social Justice and Human Rights in India 's 1940s" must-read for those who want explore the political movement of the 1940s in greater depth.
4.5
5
Instruction
Generate three new lines for incomplete poem.
Direct Scoring
Ranking Scoring
Response
That filled the morning air Spreading joy and cheer
4.5
2
As far one could see.
Here are three new lines to complete the poem:
The flowers bloomed in the meadow,
4.5
3
Their colors vibrant and bright. scene of bcauty,
delight.
Here are three new lines to complete the poem:
The birds chirped their melody
4.5
4
Welcoming the day with glee
Nature's beauty on full display.
The sun rose in the sky,
Its rays spreading warmth and light, A new day dawning, full of promise and delight. The birds joined with their sweet melody,
symphony nature's harmony,
4.5
5
Welcoming the day with joy and glee.
And the world awakens, we see,
Nature's beauty on full display, breathtaking sight, all see and say.
Table 9: Cases of quality scores generated by different methods, including Direct Scoring and the Rank & Scoring method proposed.
15

---

Published as a conference paper at ICLR 2024
DEITA-LLaMAlls vs.
Win
Tie
Lose
Vicuna
12%
77%
11%
Random Selection
34%
43%
23%
Table 10: Human evaluation results. Vicuna represents the Vicuna-13BB 138-v1.3 model. Both DEITA and Random Selection use 6K training samples, while Vicuna is trained on 125K samples. The backbone model is LLaMA1- 13B.
Koars Freds des secedaace Metu Mecuee Bomtty and they shace counee banks SaiLAA aad
coveto Gulersl Walle Banney pupttre of Brwoe tebrivere lares Maus
teeete can seeaee sdditsion tuy satte Cands
Grenk legrey and oxllury casties
crais
Sexect Ut stewe poidec
bener sipuitiinnll
Figure 5: One example of our annotation prompts and interface.
exhibited inconsistency, even with the strict setting in place. Moreover, we observe difficulties in MTurkers following instructions well. Consequently, we enlist a group of 4 colleague researchers to serve as annotators for our evaluation. To ensure impartiality and conceal model bias, each annotator was assigned samples without knowledge of the source of the two responses. Following the methodology outlined in Zhou et al. (2023), single response was generated for each question. Throughout the evaluation process, annotators were presented with two responses from different models side by side and asked to indicate their preferred answer from three options: (1) Answer is significantly better, (2) Answer B is significantly better, or (3) Neither is significantly better. conducted blindly, with the answer pairs randomly shuffled conceal their sources. To ensure the reliability of our human evaluation, we performed the inter-annotator agreement test. In line with the methodology outlined Zhou et al. (2023). we randomly selected 50 generation pairs from our evaluation and calculated the author annotator agreement score using reediscounted accuracy. Our findings yield an agreement score of 77%, which is reasonably high and aligns with the agreement levels reported in Zhou et al. (2023). The annotation interface and prompts are provided for reference in Figure 5
The pairwise comparisons
Results: We compare DEITA-LLaMAIA with the random selection baseline and Vicuna-13BB 13B-v1..... 1.3.
In Table I0] we observe that human preferences align closely with GPT-4 scoring evaluations on MT-Bench. Our data selection strategy demonstrates significant advantage over random selection in terms of human evaluation. DEITA-LLaMAlek performs on par with Vicuna- Bicuna-13B--. 13B-v1.3 in terms of human evaluation with most of the responses being considered tie by the annotators. However, note that DEITA-LLaMAL 6x is trained with 20x less data than Vicuna- -anna--1B--. 13B-v1.3 which utilizes 125K training samples.
16

---

Published as a conference paper at ICLR 2024
Prompt Template
We would like you to evaluate and rate the difficulty and complexity of the following question. You should give an overall score on scale of to 10, where higher score indicates higher difficulty and complexity. You just give without any other reasons.
Direct Scoring
Question: <Instruction>
Score:
You need to rewrite the following "instruction" to TREE through Semantic Parsin the natural language processing field and only count the total node number of the TREE. You must just give node number without any other reasons. Instruction: <Instruction> Node number:
Instruction Node
Table 11:
The corresponding Prompt Templates the Complexity baseline, including Direct Scoring and Instruction Node.
Prompt Templates
want you act as a Prompt Rewriter.
Your objective rewrite given prompt into more complex version to make those famous Al systems (e.g.. ChatGPT and GPT4) bit harder handle.
But rewritten prompt must be reasonable must be understood responded by humans.
Your rewriting cannot omit the non-text parts such as the table and code in
#Given Prompt#:. Also, please do not omit the input in #Given Prompt# You SHOULD complicate the given prompt using the following method: Please add one more constraints/reqiieeennt into #Given Prompt#
adding constraints
You should try your best not to make the #Rewritten Prompt# hecome verbose, #Rewritten Prompt# can only add 10 to 20 words into #Given Prompt#. '#Given Prompt#', *#Rewriteen Prompt#', 'given prompt' and 'rewritten prompt' are not allowed appear in #Rewritten Prompt#f #Given Promptif:
<Here is instruction>>
#Rewritten Prompt#:
want you act as a Prompt Rewriter.
Your objective to rewrite given prompt into a more complex version to make those famous Al systems (c.g., ChatGPT and GPT4) a bit harder to handle.
But the rewritten prompt must be reasonable and must be understood and responded by humans.
Your rewriting cannot omit the non-text parts such as the table and in #Given Prompt#:. Also, please do not omit the input in #Given Prompt#. You SHOULD complicate the given prompt using the following method: If #Given Prompt# contains inquiries about certain issues, the depth and breadth the inquiry can be increased. or You should try your best not to make the #Rewritten Prompt#f become verbose, #Rewritten Prompt# can only add 10 to 20 words into #Given Prompt#. '#Given Prompt#', '#Rewritten Prompt#', 'given prompt' and 'rewritten prompt' not allowed appear in #Rewritten Prompt# #Given Promptif:
deepening
<Hre is instruction
#Rewritten Prompti::
Table 12: Prompt used to increase complexity the instruction, including adding constraints and deepening.
E PROMPT EXAMPLES
E. COMPLEXITY BASELINE
Table II displays the prompts used by baselines such as Direct Scoring and Instruction Node as complexity metrics.
E.2 EVOL COMPLEXITY
17

---

Published as a conference paper at ICLR 2024
Prompt Templates
I want you act as a Prompt Rewriter.
Your objective is to rewrite given prompt into more complex version to make those famous AI systems (e.g.. ChatGPT and GPT4) bit harder to handle. But the rewritten prompt must be reasonable and must be understood and responded by humans.
Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompti:: Also, please do not omit the input in #Given Promptit. You SHOULD complicate the given prompt using the following method: Please replace general concepts with more specific concepts. or You should try your best not to make the #Rewritten Prompt# become verbose, #Rewritten Prompt# can only add 10 to 20 words into #Given Prompt#.
concretizing
#Given Prompt#', *#Rewriten Prompt#", "given prompt' and 'rewritten prompt' are not allowed to appear #Rewritten Prompt#
#Given Prompt#:
<Here is instruction>>
#Rewritten Prompt#f:
want you act as Prompt Rewriter.
Your objective is to rewrite a given prompt into a more complex version to make those famous AI systems (e.g.. ChatGPT and GPT4) bit harder to handle.
But the rewritten prompt must be reasonable and must be understood and responded by humans.
Your rewriting cannot omit the non-text parts such as the table and code #Given Prompt#: Also, please do not omit the input in #Given Prompt#. You SHOULD complicate the given prompt using the following method: If #Given Prompt#f can be solved with just a few simple thinking processes, you can rewrite it to explicitly request multiple-step reasoning. You should try your best not to make the #Rewritten Prompt# become verbose, #Rewritten Promptff can only
increasing reasoning steps
add 10 to 20 words into #Given Promptff.
*#ivven Prompt#', *#Rerrittnn Prompt#", 'given prompt' and 'rewritten prompt' are not allowed to appear #Rewritten Prompt#
#Given Prompt#:
< Here is instruction
#Rewritten Prompt#f:
Table 13: Prompt used to increase the complexity of the instruction, include concretizing and increasing reasoning steps.
Prompt Templates
Ranking the following questions according to the difficulty and complexity. Score 1-5. You can give score of if the question is too complex for you to answer it. You should respond with the format: ( [1] Score: I',n [2] Score: 2
[1] <Instruction 1>
Rank & Score
[2] <Instruction
[3] <Instruction 3>
[4] <Instruction 4>
[5] < Instruction 5>
Table 14: Prompt used to rank and score instructions of different complexitics.
Prompt for Enhancing Complexity Table and and Tabel display the prompts used to enhance the complexity such as adding constraints, deepening, concretizing and increasing reasoning steps..
Prompt for Ranking and Scoring of different complexities
Table 14 shows the prompts used to rank and score instructions
E.3
QUALITY BASELINE
Table 15 displays the prompts used by baselines such as Direct Scoring quality metrics.
18

---

Published as a conference paper at ICLR 2024
Prompt Templates
We would like to request your feedback on the performance of Al assistant in response to the given question displayed following.
##Tips:Please rate according the accuracy of the response the instruction and the input. Each assistant receives score scale of 0 where higher score indicates higher level accuracy. You must just give score without any other reasons.
Direct Scoring
##Question:
<Instruction>>
Response::
< <essonse>>>
##Score:
Table The corresponding Prompt Templates for the Quality baseline, including Direct Scoring.
E.4 EVOL QUALITY
Prompt for Enhancing Quality such as enhancing helpfulness, augmenting relevance, enriching depth, fostering creativity and supplying additional details.
Table 16 and Table 17 display the prompts to enhance the quality
Prompt for Ranking and Scoring of different qualities.
Table 18 shows the prompts used to rank and score instructions
19

---

Published as a conference paper at ICLR 2024
Prompt Templates
want you to act as a Response Rewriter
Your goal is to enhance the quality of the response given by an Al assistant to the #Given Prompt# through rewriting.
But the rewritten response must be reasonable and must be understood by humans. Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompt# and #Given Responsett Also, please do not omit the input in #Given Prompt#.
You Should enhance the quality of the response using the following method: Please make the Response more helpful to the user.
enhancing helpfulness
You should try your best not to make the #Reeritten Responseit become verbose, #Rewritten Responseft can only add 10 to 20 words into #Given Responsetf. #Giveen Response# '#Rewritten Response#", 'given response' and 'rewritten response"
are allowed appear in #Rewritten Responset
#Given Prompt#:
Give three tips for staying healthy.
#Given Responsett:
<Response>>
#Rewritten Response#:
want you to act as a Response Rewriter
Your goal is to enhance the quality of the response given by Al assistant
to the #Given Prompt# through rewriting.
But the rewritten response must be reasonable and must be understood by humans. Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompt# and #Given Response#t. Also, please do not omit the input in #Given Prompt#.
You Should enhance the quality of the response using the following method:
Please make the Response more relevant to #Given Promptit. You should try your best not to make the #Rewritten Responseit become verbose, #Rewritten Responselt can only add 10 to 20 words into #Given Responseli, #Given Response#", '#Rewritten Response#", 'given response' and 'rewritten response" are not allowed to appear in #Rewritten Responsc#
augmenting relevance
#Given Prompt#:
Give three tips for staying healthy.
#Given Responset::
<Response>>>
#Rewritten Response#:
want you to act as a Response Rewriter
Your goal is to enhance the quality of the response given by Al assistant
the #Given Prompt# through rewriting.
But the rewritten response must be reasonable and must be understood by humans. Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompttt and #Given Responselt. Also, please do not omit the input in #Given Prompt#.
You Should enhance the quality of the response using the following method: Please make the Response more in-depth
enriching depth
You should try your best not to make the #Rewritten Responseit become verbose, #Rewritten Response# can only add 10 to 20 words into #Given Responseli, *#Given Responselt' '#Rewritten Responsed", 'given response' and 'rewritten response" are not allowed to appear in #Rewritten Response#
#Given Prompt#:
Give three tips for staying healthy.
#Given Responset:
<Response>>
#Rewritten Response#:
Table 16: Prompt used to increase the quality of the response, include enhancing helpfulness, augmenting relevance and enriching depth.
20

---

Published as a conference paper at ICLR 2024
Prompt Templates
want you to act as Response Rewriter
Your goal is to enhance the quality of the response given by an Al assistant
to the #Given Promptf through rewriting.
But the rewritten response must be reasonable and must be understood by humans.
Your rewriting cannot omit the non-text parts such as the table and code #Given Prompt# and #Given Responsett. Also, please do not omit the input in #Given Promptii.
You Should enhance the quality of the response using the following method:
Please increase the creativity of the response
fostering creativity
You should try your best not to make the #Rewritten Response become verbose, #Rewritten Responsett can only add to 20 words into #Given Responsett. '#Given Responsef"' ##eewittenn Responseff' 'given response" and 'rewritten response' are not allowed to appear in #Rewritten Response#
#Given Prompt#:
Give three tips for staying healthy.
#Given Response#:
<< <Response>>
#Rewritten Responselt:
want you to act as Response Rewriter
Your goal is to enhance the quality of the response given by an Al assistant
to the #Given Prompti through rewriting.
But the rewritten response must be reasonable and must be understood by humans. Your rewriting cannot omit the non-text parts such as the table and code in #Given Prompt# and #Given Response#. Also, please do not omit the input in #Given Prompt
You Should enhance the quality of the response using the following method:
Please increase the detail level of Response
supplying additional details
You should try your best not to make the #Rewritten Responset become verbose, #Rewritten Responsett can only add to 20 words into #Given Responseit: '#Given Responseff', #Reeritten Responseff'. 'given response" and 'rewritten response" not allowed to appear #RRewitten Responsett
#Given Prompt#:
Give three tips for staying healthy.
#Given Response#:
<< <Response>>
#Rewritten Responsell:
Table 17:
Prompt used to increase the quality of the response, include fostering creativity and supplying additional details.
Prompt Templates
Rank the tollowing responses provided by different Al assistants to the user's 's question according to the quality of their response. Score each response from to with 6 reserved for responses that are already very well written and cannot be improved further. Your evaluation should consider factors such as helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response.
Use the following format:
[Response 1] Score:
Rank & Score
[Response 2] Score:
#Question#t: <IInstrccton>>>
#Respouse Listf::
[Response !| < Response 1>
[Response 2] Response 2>
Ressoone 3] < Response 3>
Ressoose 4] < Response 4>
[Response 5] < Resspons 5>
Table Prompt used rank and score responses of different qualities.
21


Reviews:
[{"review_id": "9675GHyKc4", "reviewer": "Reviewer_NSy4", "summary": "This paper presents a data selection algorithm for instruction tuning that selects data points where 1) the queries are complex (e.g.: in terms of the constraints in the request); 2) the responses are high quality (e.g.: helpfulness, creativity); and 3) the data points are diverse. For quantifying complexity and quality, a dataset is collected for each of the two aspects by prompting ChatGPT in the following way: a seed set of samples are taken from the original dataset, ChatGPT is prompted to iteratively improve the complexity or quality of each of those samples along relevant dimensions, ChatGPT is then asked to score these samples for complexity or quality, a separate model (Llama-7B) is trained to predict these scores and then used to score the entire instruction tuning dataset. For diversity, the selected pool (initialized to be empty) is iteratively grown by adding points only if they are beyond a certain distance to their nearest neighbors already in the pool.\n\nThe data selection procedure sorts the existing dataset by the product of quality and complexity scores, and uses the diversity based selection procedure to select the points from the sorted set to a prespecified size.\n\nThe algorithm is evaluated by comparing Llama-13B models trained using the data selection procedure against existing instruction tuned models (trained on other datasets), and random selection baselines (trained on the same datasets of the same sizes with points selected randomly). Comparisons are made in terms of AlpacaEval and MT-Bench, where GPT-4 evaluates the responses and human evaluation on a sample of 100 requests from LIMA's test set. The algorithm outperforms random selection, and also other instruction tuned models based on Llama-1 trained on more data.", "questions": "- Is it possible for the diversity criterion to end up selecting outliers in the datasets? How can this issue be fixed?\n- Do you need to use the representations from the Llama-13B model for computing distances as well? Can you use a different (smaller?) model for computing distances than the one you instruction-tune?\n- DEITA-6K loses more to the random selection baseline than to the Vicuna model according to the human evaluation results in Table 4. This seems surprising. How does the random selection model perform compared to the Vicuna model in this case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The procedure used to quantify complexity and quality is innovative and can be used for other hard to quantify aspects of data quality in future work.\n- The results from the experiments clearly show that the proposed method is indeed selecting important data points (i.e., better than random selection) at least for improving performance on AlpacaEval and MT-Bench.", "weaknesses": "The evaluation in this paper is limited and leaves some important questions unanswered:\n\n- The main evaluation is done in terms of AlpacaEval and MT-Bench alone. Since these are relatively small evaluation sets and it has been shown that GPT-4 evaluation can be biased (Wang et al., 2023; https://arxiv.org/abs/2306.04751), one wonders if the data selection does better than random only because it is aligned with those biases. Including further evaluation, possibly on targeted benchmarks covering abilities like reading comprehension, complex reasoning etc. can be helpful.\n- Related to the above point, since human evaluation was done on only 100 instances, it would be helpful to quantify the reliability of this assessment, e.g.: using inter-annotator agreement scores and significance, and perform a larger scale evaluation if needed.\n\nThe procedure used for quantifying complexity and quality can be validated and possibly improved further\n\n- Section 2.3 states that ChatGPT is shown multiple samples evolved from the same seed example are shown to ChatGPT at a time for scoring their complexity (and this is possibly true for quality as well). Are these scores comparable across evalved sets from different seed examples? This is necessary because all these data points are used together as training data for training the Llama-based scorer. If they are not comparable, it might help to randomize the sets shown to ChatGPT for scoring.\n- Relatedly, it would help to have humans also score data points in terms of complexity and quality and see if the scores correlate with  \nChatGPT's judgments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2oH2VRKSg", "reviewer": "Reviewer_zdke", "summary": "This work shows that complexity, quality, and diversity are all important for data selection in instruction tuning. Based on the studies, it proposes a score-first, diversity-aware approach called DEITA to select the “good” data samples. With their proposed learning-based enhanced data measures, DEITA shows better or on par performance with state-of-the-art LLMs with only 6K training samples.", "questions": "Why the complexity measurement is obtained from “instruction”, and the quality measurement is obtained from “response”? Is there any intuition of this?\n\nTypo: “large margine” should be“large margin” in the last sentence of the introduction.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper provides in-depth controlled studies to show the proposed scoring measurements are better than baselines in terms of complexity and quality.\n- With fine-tuned LLaMA-7B models on the 20K Alpaca dataset with the scores from GPT, the measurements can score unseen instructions at a cheap cost.\n- The proposed diversity-aware selection method is efficient and easy to implement.\n- Clear presentation and easy-to-follow writing.", "weaknesses": "Limited evaluation\n- No evaluation result on benchmarks like MMLU and Big-Bench-Hard, which allows verification of commonsense knowledge and reasoning with ground-truth answers. GPT4-based evaluation often includes errors or bias, so not enough.\n- Marginal performance improvement. In Table 3, DEITA is worse than Vicuna-13B on AlpacaEval dataset. Also, In Table 4 with human evaluation, Vicuna performs almost similar to DEITA.\n- Limited baselines; recent instruction selection works including LIMA and Alpagasus are missing in the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M95mCPFWRx", "reviewer": "Reviewer_RtKh", "summary": "This paper investigates the trendy problem of selecting high-quality instructions for fine-tuning pre-trained language models (PLMs). The paper aims to provide an automated pipeline for this selection problem. Following recent works, sample quality is evaluated on three dimensions: complexity, quality, and diversity where this evaluation is conducted by other LMs such as GPT-4. The paper validates the proposed approach with LLaMA-1 and shows that when selecting samples from a larger pool of lower-quality data, the proposed method, DEITA (Data-Efficient Instruction Tuning for Alignment), is able to match the performance of current open-source alignment models with a small fraction of fine-tuning data.", "questions": "If the authors wish to further develop this work toward publication at a future venue, I suggest the authors to \n\n1. spend significant effort in discussing the current research landscape and identify a valid (important and essential) research gap that this work will make a substantial and unique contribution to. This is my main concern for this line of work.\n\n2. improve the quality of the paper. If its extent of innovation is capped by the many other papers out there,  this work needs to have a high quality to be a valid contribution. This includes conducting more experiments and ablation studies and discussing design choices and novel findings.\n\n3. improve the technical approach. The current methods documented in this paper do not seem particularly novel compared to existing works and its technical contributions seem capped by the heavy reliance on GPTs. If the authors could design original technical approaches (new metrics, new evaluation methods, novel ways for combining different metrics), the paper could be made much more attractive.\n\n- Appendix should not be submitted under the main paper.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is, in general, an easy read. Its ideas are presented straightforwardly along with nice tables and illustrations.\n\nThe problem it investigates is trendy. Quality for each part of the work is generally fine–it is structurally complete, self-contained, and has a reasonable narrative.", "weaknesses": "This paper is \"ok\" but not particularly enticing. The topic is trendy but the approach is not technically challenging or particularly innovative.\n\nI have recently read a number of papers on this topic of instruction mining with LLaMA/Alpaca–to name a few\n\n- a. ALPAGASUS : TRAINING A BETTER ALPACA WITH FEWER DATA\n- b. Instruction Mining: High-Quality Instruction Data Selection for Large Language Models\n- c. InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4 \n\nThe technical body of these papers are uncomfortably similar. Combining a number of existing metrics (often in trivial ways) such as quality, diversity, etc. as a new evaluation metric and conducting the evaluations largely with the help of GPTs. And the end goals are also the same–to achieve comparable or better performance with fewer samples. I'm not sure how much \"research gap\" remains there and how individual works may continue to contribute to that–at least, this concern is still not resolved by this paper.\n\nIn terms of quality, the construction of this paper is simple. It does not have many ablation studies or insightful discussions of design choices/novel findings. No additional results or further discussions are provided in the Appendix.\n\nIt is an \"okay\" paper without much particular. Given its position in this apparently overly populated track, I would not be very interested in seeing it at the conference and vote against publishing it.\n\n- Reproducibility: Code, data, model checkpoints, or data selection tools are not provided during the reviewing phase.\n\n- Format: Appendix is not cut from the main paper. The PDF provided for the main paper is this 20-page document.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)