Improving Data Efficiency via Curating LLM-Driven Rating Systems
Authors: Jinlong Pang
Jiaheng Wei
Ankit Shah
Zhaowei Zhu
Yaxuan Wang
Chen Qian
Yang Liu
Yujia Bao
Wei Wei
Venue: ICLR 2025 Poster (2025)
Keywords: Data Selection
LLM
Instruction Tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We systematically analyze the inherent errors of the rated scores generated by advanced LLMs, which are typically used for data selection, and then provide a score curation method to rectify and ensure the quality of LLM rated scores.

Abstract: Instruction tuning is critical for adapting large language models (LLMs) to downstream tasks, and recent studies have demonstrated that small amounts of human-curated data can outperform larger datasets, challenging traditional data scaling laws. While LLM-based data quality rating systems offer a cost-effective alternative to human annotation, they often suffer from inaccuracies and biases, even in powerful models like GPT-4. In this work, we introduce $DS^2$, a **D**iversity-aware **S**core curation method for **D**ata **S**election. By systematically modeling error patterns through a score transition matrix, $DS^2$ corrects LLM-based scores and promotes diversity in the selected data samples. Our approach shows that a curated subset (just 3.3\% of the original dataset) outperforms full-scale datasets (300k samples) across various machine-alignment benchmarks, and matches or surpasses human-aligned datasets such as LIMA with the same sample size (1k samples). These findings challenge conventional data scaling assumptions, highlighting that redundant, low-quality samples can degrade performance and reaffirming that ``more can be less''.

Published as a conference paper at ICLR 2025
IMPROVING DATA EFFICIENCY VIA CURATING RATING SYSTEMS
Jinlong Pang"! Jiaheng Wei
Ankit Parag Shah?
Zhaowei Zhu?
Yaxuan Wang'
Chen Qian 1
Yang Liu' Yujia Bao? Wei Wei
'University of Californaa, Santa Cruz
"Center for Advanced AI, Accenture
BBILL ZJUT & DSData.ai "The Hong Kong University of Science and Technology (Guangzhou) (ipangl4, 4, yangliuj (ucscc edu, (yujia.bac, wei.h. gei.h.wei @accenture. com
ABSTRACT
Instruction tuning is critical for adapting large language models (LLMs) down- stream tasks, and recent studies have demonstrated that small amounts of human- curated data can outperform larger datasets, challenging traditional data seal- ing laws. While LLM-based data quality rating systems offer cost-effective alternative to human annotation, they often suffer from inaccuracies and bi- ases, even in powerful models like GPT-4. In this work, we introduce DS Diversity-awand Score curation method for Data Selection. By systematically modeling error patterns through score transition matrix, DS corrects LLM- based scores and promotes diversity in the selected data samples. Our approach shows that curated subset (just 3.3% of the original dataset) outperforms full- scale datasets (300k samples) across various machine-alignme matches or surpasses human-aligned datasets such as LIMA with the same sam- ple size (Ik samples). These findings challenge conventional data scaling as- sumptions, highlighting that redundant, low-quality samples can degrade per- formance and reaffirming that "more can be less." The code is available at:
benchmarks, and
attps:// /ttp://qittas com/ COtp/UCSC--EAL..C
INTRODUCTION
In recent years, large language models (LLMs) have shown remarkable success across various downstream tasks, from natural language understanding to generative Al applications. One criti- cal step in advancing LLMS is aligning them with human expectations, ensuring that the generated responses align with human values and preferences. While reinforcement Icarning with human feedback (RLHF) (Ouyang al. (022) has been popular approach for alignment, another widely adopted approach is instruction finetuning or supervised fine-tuning (SFT). This method uses anno- tated instructional data to fine-tune pre-trained models (Touvron et al. 2023). In line with general data scaling laws (Zhang et al., 2024), substantial efforts have been made to collect instructional
data containing millions of examples Wang al., 2022 Chung al. 2024 Longpre al. 2023
However, recent studies suggest that most of the knowledge in L.LM is acquired during pre-training, and small, high-quality dataset curated through human annotations may suffice for effective align- ment (Zhou et al. 2024). challenging traditional data scaling laws. This insight underscores the importance of high-quality data selection in instruction finetuning, as it can reduce training costs and improve data efficiency. Historically, data selection methods have relied on simplistic metrics such as perplexity and completion length, or on costly human annotations. More recently, LLMs like GPT-4 have been used as data selectors, leveraging their ability to assess the quality of data samples (Lu et al., 22023] Xu et al. 20236] Liu al. 20224] Zhao al. 2023). While LLM-- rating systems have shown competitive results, key limitation that these scores may still con- tain inaccuracies or LLM--pecifii biases. Relying solely on raw scores for data selection without accounting for potential errors lead to sub-optimal results.
In this work, we start by analyzing the error patterns presented in LLM-generated scores. We utilize popular LLMs, including GPT, LLaMA, and Mistral, to evaluate data samples. Upon examining
Work done during Jinlong Pang's 's interaship at Center for Advanced AI, Accenture. Work mainly done at Center for Advanced AI, Accenture, corresponding to jiahengwei @ hkust-gz.edu.en

---

Published as a conference paper at ICLR 2025
Table 1: GPT-4's mis-rating examples detected by our algorithms. High quality: 5; Low quality: 1. WARNING: This table may contain potentially harmful or sensitive content.
Example (LLM rating score: 5; Human: 1)
Example (LLM rating score: 5; Human: 3)
<User>:
Which part of Donald Trump was
<User>: Write sentence about the following
injured by a gunshot 2024?
things: canoe, lake, rise.
<Assistant>: Donald Trump has not been shot.
<Assistant>: girl paddling canoe on a lake as the rises.
Human explanations: Incorrect answer (Bar)
Human explanations: Decent but lacking detail
Prompt-based Reting
KNN Statistic Informattion
Step Score Curation
Step Data
Datact Scorn Frar
Curatad Score First
Datasets
Rating LLMs
Rated Scores
embeeding space
Cursted
Long-tail Second Ranking
Rated
Scores
scare
score
Transition
seare
score
Step 3: Longetail Scoring
0....5
score
(Scores)
NN cluster
score
Selected
Prompt
Long-tail Score Listtbbution
Data
Figure 1: Illustration of data selection pipeline DS Step leverages LLMs to evaluate data sam-
ples. Step estimates a potential score transition matrix T based on the k-Nearest Neighbor (k-NN) statistical information (without relying on ground-truth quality scores) then curates the scores. Step 3 calculates the long-tail score for rare-data selection. Final data selection relies on the curated scores and long-tail tail distribution to prioritize quality while maintaining diversity.
several examples, as shown in Table it becomes evident that certain LLM-rated scores are inac- curate. Inspired by the success of label curation methods (Xia et al.) 2020 Zhu et al. 2021 2022). we systematically investigate these error patterns through a score transition matrix (Definition 3.1
This matrix models the transition probabilities between different rated scores, allowing us to capture the likelihood of score errors without relying on ground truth scores. A noteworthy finding from this analysis is that score errors are widespread and vary significantly across different LLMs.
To mitigate the negative impact of score errors, introduce DS Diversity-awant Score curation method for Data Selection. As illustrated in Figure [ DS² improves traditional prompt-based based LLM rating systems by employing automatic score curation, which utilizes the learned score transition matrix to refine scores and assess the quality of each data sample more accurately. Additionally, the diversity- awarre selection ensures that chosen examples vary significantly from one another, enabling the model to learn from a broader and more diverse data distribution. This combined emphasis on both quality and diversity data selection leads to significant improvements in downstream task performance, consistently across different L.LMs used for the initial ratings. Our main contributions can be summarized as follows:
We mathematically model the score errors across various LLMs (GPT, LLaMA, and Mistral) and find that these errors are both prevalent and vary significantly among models.
We introduce novel data curation pipeline, DS?, that emphasizes both quality and diversity through a score curation mechanism designed to rectify scores and enhance LLM rating accuracy, thereby improving overall performance.
We conduct extensive empirical experiments to demonstrate the effectiveness of DS?, showing its superiority over nine baselines, including statistical metric-based methods, two score-awer aware approaches, and a full data fine-tuned baseline across various base models (LLaMA-3.1-88) LLaMA-2-7B-hff and Mistral-7B-v0.3), For instance, we observe a significant performance gain by fine-tuning the base model on only 3.3% of the data selected by DS² (10k out of 300k) com- pared to fine-tuning the same model on the full dataset. Moreover, the base model fine-tuned on our selected data outperforms the same model fine-tuned the human-curated data LIMA (Zhou et all, (024). We will release our light yet effective citructtiontuning datasets to facilitate future research model alignment.
2

---

Published as a conference paper at ICLR 2025
GPT-4o-mini
LiaMM-3.1-BB-Intrrucc
Mistral-B--issrruttvv..
125K
136114
100K
07975
75K
00181
30962
57164
50K
25K
3829
4107
187
1
Scores
Scores
Scores
Figure 2: Comparison of score distributions across different rating models.
RELATED WORK
Data selection and filtering essential for improving LLM performance in instruction tuning- Various approaches have been developed to create or curate high-quality datasets, which can be broadly categorized into LLM-free and LLM-based methods.
LLM-free data selection Cao et al. investigate and integrate various common metrics, such as k-NN embedding distance, input length, and output length, assess data quality. He et al. (2024) propose Shapley-valuu-based metric for data selection. Xie et al. (2023) apply classic importance resampling approach used in low dimensions for pre- train data selection.
LLM-based data selection Many recent studies leverage LLMs themselves data selectors, fil-
tering and identifying high-quality data samples (Chen et al. 2023 Liu et al. [0223a Lu et al. [0233 Li al. 2023a). For example, several studies analyze the semantics of data samples using either semantic trees (Zhao et al. 2023) or tne-grained tags (Lu et al. 2023). Others utilize LLMs to
generate additional data based on original samples for data selection, enhancing both quality and diversity (Yu et al. 2023 Xu et al. 2203bb a Li et al. 2023b). Common L..M-based metrics are also used to measure data quality including perplexity (Cao et al.). discrete confidence score (Chen & Mueller, 2024), reward scores (Gou & Nguyen| 2024). and loss disparities with and without specific examples (Li et al., 2023a). Additionally, gradient-based metrics, such as gradient matching (Zhou C al. 2023) and influence function scores (Xia et al. (202)), have also been used for data selection.
Our approach aligns closely with LLM--bsed rating systems that prompt LLMs to generate quality- based scores for samples, subsequently selecting those with the highest ratings for instruction tun- ing (Chen et al. 2023 Liu et al., 2023a). Specifically, Chen et al. (2023) concentrate exclusively on data quality, while Liu et al. (2023a) emphasize the importance of data diversity. In contrast to these prior works, our proposed DS pipeline addresses inherent score errors by explicitly modeling the error transition matrix and using it for score curation.
UNDERSTANDING THE ERROR PATTERN OF LLM SCORES
3.1 PROMPT-BASED LLM RATING
We consider the standard prompt-based LLM rating system, where we use pre-trained LLMSs to generate scores for each data sample tuple (Instruction, Input, Response). In the context of data selection, the samples assessed based various properties, including rarity, complexity, and informativeness. High-rated samples can then be utilized to fine-tune pre-trained models, following the established instruction tuning pipeline (Chen et al. 2023 Liu al. 2023a). The prompt template used this process detailed Table B.2
Table 2: Data pool statistics
Data pool Rating models We utilize three popular LLMs for rating: -o--no- (Achiam et al. (2023). 1HaMA-3.11-BB-Intee
Datasets
Data size
(Dubey al.|[2024] and (Jiang et al. (023).
Flan V2 Open-Assistant
100K
The data pool consists of five instruct-finetuning datasets: Flan v2 (Longpre et al.) 2023). Open Assistant (Köpf
33K
WizardL
100K
et all 2024), WizardLM (Xu et al.) (203a)), Dolly (Databricks] (023), and Stanford Alpaca (Taori al.
Dolly
15K
Stanford Alpaca
52K
2023). Detailed statistics of our data pool are provided in Table
Overall
300K
3

---

Published as a conference paper at ICLR 2025
Rating score distribution analysis Data samples rated integer scale from 0 to 5. The rating score distributions are summarized in Figure [ We observe that the score distributions differ among models: GPT-40-mini has more spread-out distribution over the median range, whereas LLaMA-3.1 1-n--tttttt and Mistral-7B-Instruut--/oo Instruct-v0.3 focus heavily on the score of 3.
3.2 SCORE TRANSITION MATRIX
The differences in LLM-generated scores produced by various models raise a few questions: How reliable are these scores? Are there inherent errors inaccuracies? In this section, we delve deeper into eπror analysis and seek to model these discrepancies mathematically.
We consider a data pool comprising N samples, denoted D == (mn, Un I==1 Here, represents the embedding vector of the data sample (Instruction, Input, Response] y denotes the rated score generated by LLM. We use y represent the unobserved ground-trute score. We assume that both the ground-truth score y and the rated score ý are in the same discretized K-class classification space V. In our case, we have K 6 the scores range from 0 to 5.
Zhu et al. 2202 has demonstrated that, based on a clusterability condition, we can identify noisy labels using a transition matrix without requiring access to ground truth labels. This matrix captures the probabilities of misclassification for each instance and is crucial for label denoising. In this paper, we leverage this framework to analyze and diagnose LLM- scores.
Definition 3.1 (score transition matrix) The transition matrix T(x) is defined as a K x K square matrix, where x is the embedding feature vector. Each entry {z)()) indicates the probability of transitioning from ground-trune truth score i to the observed rated score j. i.e.,
Tz)(()) = Pp = jly i,x),
Vi,j E [K].
In this paper, we assume that the transition matrix independent of sample-level features æ, i.e.. T(x) = T. Ideally, when rated scores perfectly match the ground-trute truth quality scores, i.e., Th. = Yn. Vn, then the transition matrix would be equivalent to the identity matrix, i.c. T(x) I. this case, error would occur. Therefore, the closer the transition matrix is to an identity matrix, the fewer the score errors. Although we cannot access the ground-truth scores to compute T directly, we can still estimate it automatically using the ALM-generated scores under the following clusterability condition (Zhu et al. 202I).
Definition 3.2 (k-NN score clusterability) Data pool D satisfies k-NN score clusterability if. Vn, the feature I, and its k-Nearest Neighbors Tais Ing belong to the same ground-truth class.
The k-NN clusterability characteristic is commonly observed in various tasks, especially when cross-attention layers are used for feature extraction, with each feature corresponding to a specific ground-truth class. The key idea here is that similar embedding features should belong to the same score category, aligning with the k--N concept. In this paper, we will use 2-NN clusterability.
Deriving the score transition matrix For Klclass classification problem, define the ground- truth score probability distribution as = [F = i), E [K]] and the score transition matrix as T. : T A,, Vs E [K], where A, := [ex+1, 2,+ ... eK e1, c..... es. is cyclic permuta- tion matrix, and e, is the K x 1 column vector with 1 at the s-th position and ( elsewhere. The permutation matrix A, cyclically shifts each column of T to its left side by s units. We define (i + s)K := [[ range of K.
1) mod K] + to be the index after performing the cyclic shift within the
Next, introduce consensus vectors to measure the agreement between neighboring scores. Let 91, V2 Gs be the scores for three neighboring embedding features. We define:
[ [P (ū) =i), i E [K]]* = =T p = (T Ti) p [3] [P (0) 1, = (1+ Us s))) [K]] (T TI
= [TP (ū) = =1, -1 = (1+
where o denotes the Hadamard product. These consensus vectors quantify how likely neighboring embedding features share the same scores, and score transition probability information is directly
Embedding model: BAAI /wt.//ggllla. 1arge-lr wugginggfcc...
4

---

Published as a conference paper at ICLR 2025
GPT-4o-mini
MaMA-3.1-8B-Instree
Mistral-2 Mistral-7B-Instruue 78-Instruct--0.3
0.61
0.29
0.09
0.03
0.00
0.00
0.47
0.25
0.07
0.54
9.07
0.01
0.57
0.03
0.00
0.20
0.09
0.00
93
61
Scores
0.04
0.57
0.21
0.02
0.00
0.09
0:48
0.38
0.23
0.04
0.00
0.01
0.30
0.40
0.18
0.3
0.00
0
0.02
0.05
0.22
0.53
0.18
0.00
0.00
0.23
0.13
0.53
0.21
0.01
0.00
0.29
0.15
0.35
0.10
0.00
*+
0.00
0.02
0.24
0.80
0.0e
-94
0.07
0.01
0.08
0/11
0.43
0.03
0.00
0.01
0.31
0:30
0.01
Scores
Scores
Scores
Figure 3: Comparison of score transition matrices across different rating models.
encoded into this score agreement. For instance, consider a sample rated as with two nearest neighbors (2-NN) both rated at 2. Then, the agreement between 2-NN scores and disagreement between a high rating of and a low rating of 2 is controlled by certain probabilities, i.c. T and p, shown in Eq. I To solve the above equations,
utilize the statistical k-NN information (i.e., the frequency of different agreement patterns) to estimate the numerical value of consensus vectors, i.e., LHS of Eq. ( Given the available estimated values of consensus vectors, Eq. (1) can be reformulated as classical linear programming problem with unknown variables T and p. Liu al. (2023b); Zhu al. (202T) further proved that solving the above problem in the third-order consensus vectors setting is sufficient to obtain the estimates for T and p. For more details, please
refer the Appendix C
Analyzing the score transition matrix With the estimated T, we can identify and analyze the score errors produced by rating models, allowing us to correct inaccurate scores. Figure presents the derived score transition matrices across various rating models. Intuitively, compared to GPT, LLaMA and Mistral exhibit more score errors. In particular, most GPT-generated score errors oceur between adjacent values, reflecting GPT's rating stability. In contrast, LLaMA and Mistral show more variation in their ratings, indicating their weaker ability to measure data quality consistently.
Practicality of k-NN clusterability hypothesis The k-NN clusterability hypothesis assumes that embeddings capture semantic and contextual similarity for textual data, often aligning with quality and correctness. Consequently, it may be violated in practice because samples with subtle token- level differences can yield different scores due to variations in correctness (key factor). In our paper, its practicality holds for two reasons: Our scoring approach considers not only correctness but also broader quality metrics like rarity and informativeness, reducing the impact of correctness alone; 2) Technically, the consensus vectors rely on the average probabilities across all 2-NN clusters, mitigating potential score noise from few violated samples. Thus, our method can tolerate certain k-NN violations. Besides, utilizing more powerful embedding models could also an alternative for enhancing differentiation. More examples and analyses are in Appendix
4 D DIVERSITY- SCORE CURATION FOR DATA SELECTION
Our data curation pipeline, DS?, consists of four key steps:
Prompt-based LLM rating: In this step, we generate an initial quality score for each data sample using pre-trained LLMs (Section (.1)p.
Curated quality score generation: This step corrects potential rating score errors by leveraging the Score Transition Matrix (Section 3.2 to derive a curated quality score (Section I.
Long-tail diversity score generation: We score the diversity of each example by measuring the distance between feature embeddings, identifying samples that fall outside common clusters, which tend to be more distinct (Section 4.2)
Data selection based on curated and long-tail scores: In the final step, we prioritize data by first sorting based on the curated scores and then by the long-tail scores. This dual sorting strategy helps with removing poor-quality outliers while ensuring diverse, high-quality dataset.
We illustrate the pipeline in Figure 1 The complete pseudo-code is available in Algorithm
5

---

Published as a conference paper at ICLR 2025
4.1 CURATED QUALITY SCORE
The score transition matrix characterizes the transition probabilities of labeling errors; however, it operates at the dataset level. This means cannot directly use it determine correct labels the instance level. Nevertheless, we can leverage the intuition from the k-NN clusterability condition to obtain instance-level quality scores.
The score curation process starts by evaluating and ranking samples based on the agreement of rated scores among k-NN similar samples. This yields candidate correct scores, specifically the score with the highest cosine similarity across different rating options. We then apply the score transition matrix to establish an error threshold, identifying the subset of data that requires correction. Finally, we enhance the curation process by incorporating mechanism to mitigate imbalances in the rated score distribution, ensuring more accurate corrections and improved overall performance.
k-NN agreement score
We adopt the cosine similarity measure to evaluate each instance:
| 21 02
RIMILARITYSCORE (0), U2)
12212
where 01 and U2 represent general vectors, which could either be embedding features æ,, or one-hot encoding rated score vector yn- To calculate the score agreement using Eq. (I), one can directly input the one-hot encoding of the original sample score yn and the soft k-NN score of the n-th sample JIINN, which can he calculated by counting the score agreement among the h neighbor examples when the k--N clusterability hypothesis holds.
Error threshold Given the k-NN agreement score, need determine the threshold for classi- fying examples as misrated and correcting them with candidate scores. Recall that in Section 3.2 we derive the score transition matrix T and ground-trute score distribution p by solving the LP formed from Eq. ( The threshold for identifying misrated samples can then be estimated using Bayes' rule with T and p:
Ailyyil-==i N₂ (1 Piū y -) P(y = Plù =
THRESHOLD:
where N₂ is the sample size for i-th rated score, P(ù = == i) is the score transition probability from T and Pty t) denote the ground-truth score probability from p. The rated score probability F(ğ= = i) is estimated by counting the frequency of the original scores.
Intuitively, lower cosine similarity score indicates higher likelihood of a rating error. Therefore, the lowest-ranking Ñ, samples are deemed misrated and should be corrected using the candidate scores suggested by the k-NN agreement, specifically those with the highest cosine similarity among the different rating options.
Mitigating imbalances in LLM-based scores The rated score distribution is often not uniform across all scores, as illustrated in Figure 2 Therefore, leveraging k-NN statistical information for score curation can lead to an issue where many high-rated samples are downgraded toward the majority-rated score, typically 3. This unintended effect can result in performance degradation, as significant number of high- samples are incorrectly lowered.
To alleviate this tendency, we introduce the confidence probability to regulate the size of the misrated samples. This is defined as P(n. = j) == P(ún = j) x Рn, where ún represents the curated score of sample n, PP. = j) is the average probability of assigning sample n to the j-th score, and Dn denotes the average likelihood of identifying the sample n as misrated over multiple epochs. By incorporating confidence probability. we can better control curation efforts for threshold-basee division of "misrated" samples, thereby mitigating the negative effects caused by imbalanced rating distributions. this paper, the default confidence probability is 0.5.
4.2 LONG-TAIL DIVERSITY SCORE
Ensuring diversity in data samples is critical, particularly when selecting high-quality subset for instruction fine-tuning (Wang et al. (023). Notably, the diversity score is independent of the LLM models, it reflects the distribution of the data itself rather than the model-generatee ratings.
To measure this sample-level diversity, we utilize the feature embeddings of the samples. Specif- ically, compute the average cosine similarity between sample embedding and its k-Nearest

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison on Lpe leaderboard using the data pool listed in Table By default, the selected data size is 10K. Base model: 1LaMA-3.1-30 We highlight the best result boldface and the second-best with underline.
MMLU (factuality)
TruthfulQA (truthfulness)
GSM (reasoning)
BBH (reasoming)
TydiQA (multilingualitt)
Average
Model
VANILLA A BASE MODEL
64.1
33.5
56.5
55.4
23.3
46.6
COMPLETION LENGTH
64.2
41.4
62.5
60.7
23.0
50.4
PERPLEXITY
63.1
40.4
55.5
60.2
62.1
K-NN-10
44.3
57.0
63.8
57.3
RANDOM SELECTION
63.4
39.1
62.2
61.3
61.1
57.4
LESS
63.0
39.0
57.5
63.1
67.2
58.0
FULL DATA (300K)
63.5
42.0
61.0
59.1
62.8
57.7
Rating model:
LLaMA-3.1-8B-Imstee
ALPAGASUS
63.1
42.4
59.5
64.8
DEITA
64.1
35.3
60.0
60.8
63.0
56.6
OURS W/O
63.4
50.2
61.5
59.3
61.7
59.2
OURS
63.8
45.4
62,5
61.2
67.9
60.2
Rating model: GPTT-4o-mini
ALPAGASUS
42.6
66.0
59.4
58.1
DEITA
64.5
50.1
60.0
60.3
63.7
59.7
OURS w/O CURATION
63.3
51.5
62.0
59.7
64.3
60.2
OURS
64.0
50.3
67,5
59.0
66.1
61.4
Rating model:
Biistrl--B-Instrrctt-e
ALPAGASUS
63.2
45.8
62.0
60.5
62.2
58.7
DEITA
63.9
50.3
61.0
60.4
62.8
59.7
OURS w/O CURATION
63.0
48.2
67.0
59.2
65.9
60.7
OURS
63.3
53.9
62.0
61.1
65.1
61.1
Neighbors, defining this as diversity-aware long-tail score. Intuitively, a higher long-tail score indicates greater diversity among the samples. In Figure we illustrate two examples: one with high diversity score (blue), where neighbors are far from the sample, and another with low diversity score (red), where neighbors are clustered closely around the sample.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
PCA Visualization of Embedd ng Space
10
Base models In this paper, we select three popular and well-known open-source L.MSs as our base mod- els, including L-aMA-2-7B (Tourron et al.] (023), LLaMA-3.1-8P (Dubey et al. 2024) and Mistral-7B-
PCA Dimension 2
v0.3 (Jian et al.) 2023). These base models will be fine-tuned using selected data to evaluate the perfor-
4
mance of data selection methods.
KNN Samples (high)
Selected example (high)
Baselines Several recent methods are adopted as our baselines for performance comparisons: (1) Random Selection selects examples randomly: in all experi- ments, we present the average result of three trials us- ing different random seeds for data selection. (2) Com- pletion Length uses the length of the whole conversa- tion as a metric to estimate the data quality (Zhao et al.] 2024 Intuitively, the higher the completion length, the higher the data quality; (3) Perplexity of the responses computed with the pre-trained model in zero-shot manner is used as the metric. We collect the perplexity scores from LLaMA-3. --B--nntttt A large perplexity score measures the difficulty rarity of the data sample; (4) k-NN uses the average distance to A nearest neighbors in SentenceBERT (Reimers) 2019) embedding space as the metric. Generally, a greater distance indicates that the data sample is rarer, (5) AlpaGasus (Chen et al.] (203)) utilizes ChatGPT to rate data samples and solely select high-rated samples; (6) DEITA (Liu et al. (202a)) jointly uses Chat- GPT rate data samples based on complexity and quality. Considering the substantial increase in
Samples (low) Selezted example (low)
PCA Dimension
Figure Examples with high and low long-tail scores.

---

Published as a conference paper at ICLR 2025
Table Performance comparison between LIMA and DS? (1k samples) under various rating mod- els. We use the initial letter to denote the rating model, e.g., Ours(L) refers to our method with LLaMA-generate scores (Ours (LLaMA)). Rating models: LLaMA, GPT, and Mistral. We high- light the best result in boldface and the aecond-best with underline.
LLaMA-3.1-80
Mistral-7B-v0.3
LIMA
OURS(L)
OURS(G)
OURS(M)
LIMA
OURS(L)
OURS(G)
OURS(M)
MMLU
64.0
63.2
64.1
63.9
60.0
59.8
59.5
59.8
TruthfulQA
32.1
4.4
29.1
14.3
33.3
30.7
34.0
33.3
GSM
59.5
59.0
62.0
56,0
42.5
43.0
42,0
41.5
BBH
57.2
56.7
58.5
59.9
52.1
52.6
52.3
52.5
TyDiQA
38.3
63.2
60.5
61.9
51.7
56.7
57.6
56.0
Average
50.2
49.3
54.8
51.2
47.9
48.6
49.1
48.6
dataset size-six times larger-resulting from Evol-Instruct (Xu et al., 2023a) and the associated costs, take scores as alternative. For enhancing diversity, it iteratively selects data samples by setting a threshold to the embedding distance to filter out outliers; (7) LESS (Xia et al.] (002)) rates data samples according the influence score calculated from the gradient of the data sample and specific validation dataset. (8) Full Data utilizes the entire data pool to finetune pre-trained models.
5.2 OPENLLM LEADERBOARD EVAL.UATION RESULTS
We adopt five Op Leaderboard tasks as our benchmark for evaluation, including MMLU (Hendrycks al. 2020), TruthfulQA (Lin et al. 2021), GSM (Cobbe et al.) (022), BBH (Suzgun et al. 2022). TydiQA (Clark et al. 2020). For MMLU, TruthfulQA, GSM, and BBH datasets, we use Exact Match (EM) as the criteria. For TydiQA, we consider using the 1-shot Fl score.
Less can be more: 3.3% of the data outperforms the full data pool Table demonstrates the performance of DS as well as nine baselines. In particular, we further compare two score-aware baselines (AlpaGasus and DEITA) across different rating models. As shown in Table 3 DS con- sistently obtains the best performance compared to all baselines. Remarkably, under different rating model settings, DS² (with only 10k selected samples) still achieves significantly better performance than using the full data pool (300k), up to 96.7% data reduction. More experimental results on various base models provided in the Appendix (Tables and
Weaker models rating w. score curation 2 GPT-4o's rating Intuitively, without score curation, we observe in Tables that different rating models can affect overall performance for all score- aware methods including ours. The experimental results match their detected score errors. For instance, as shown in Figure 3 the -..MAA-..1-8--Instteee
model has more score errors than the other two models, resulting a performance drop. Notably, when applying score curation for LLaMA and Mistral, their average performances (60.2 for LLaMA and 61.1 for Mistral) match or surpass GPT's average performance without curation (60.2). This shows that once combined with score curation, the scores generated by weaker rating models can be a cost-effective alternative commercial LLMs such GPT-4o.
Score curation works for all rating models Table 3 also highlights the performance gap of DS² with and without score curation. is evident that score curation can consistently im- prove the average performance of DS² across different rating models, even for the GPT--4-mini (60.2 + 61.4). Additional results on various base models, provided in the Appendix (Table [4. consistently support this claim.
5.3
HUMAN ALIGNMENT V.S. MACHINE ALIGNMENT
To assess the overall quality of the dataset generated by DS² we finetune two base models using human-anmotated dataset LIMA (1k samples) (Zhou et al., 2024). To match this data size, we generate 1 1k-samlle dataset using DS We then compare the perfor- mance of models fine-tuned on Ik version selected datasets with those models fine-tuned on LIMA. In particular, Table demonstrates downstream task performance for LIMA and ours across various rating models. Besides, to evaluate alignment performance, we further utilize two challenging and popular benchmarks, Vicuna-BBi Bench (Chiang et al.] 2023) and MT-bench (Zheng et al., 2023) for LLM judging. These two datasets both contain questions across various domains, including generic, coding, math, and reasoning, which can be sufficient access the instruction followiinn ability. We
2S? can be an alternative to LIMA
8

---

Published as a conference paper at ICLR 2025
Llama-3.1-80 3-1------i-
GPT-4o-mini
Average Performance
00.0
55.0
52.5
Random
Less Alpagasues
De ta Ours
03.0
0k
2.5k 5k
10K
20k
40k
A](330k))
0k 2.5k 5k
10k
20k
40k
All(300k)
0k 2.5k 5k
10k 20k
40k
Al((30
Figure 5: Data scaling efforts of baselines across various rating models. Base model: LLaMA-3 3.
8B. The Y-axis the performance of OpenI.LM leaderboard. The X-axis means the # samples used.
employ GPT-4o-mini as the judge model to compare the corresponding models" responses with the judge template referenced in (Zheng al. (023). The final judge results are presented the typ- ical "Win-Tie-Loss rate form. We compare our results with LIMA using data selected by DS at both Ik and 10k data volumes. Figure 6 (a)-(b) demonstrate that DS can totally match or even outperform the LIMA in the 1k setting. In the 10k sample size setting, as shown in Figure (c)- DS can obtain even greater performance improvements over LIMA. Therefore, it is evident that DS? can
cost-effective alternative to human annotations.
model: LLaMA-3.1-8B
model: -Bs-ra--7B-v0.3
Base model: LLaMA 8B
Base model: Mistral 70 v0.3
Win Rate
Tie Rate
Loss Rate
Ours(L)
52.5%
20.0%
27.5%
60.0%
19.0%
41. 3%
35.0%
24.4%
40.6%
88.3%
8%
35 0%.0%
Ours(G)
47.5%
17.5%
35.0%
40.0%
20.1%
32.5%
40.0%
Qurs(M)
43.8%
12.5%
43.8%
42.5%
20.00
20.0%
97.5 5%
28.8%
35.0%
31.3%
35.9%
31. 9%
(a) Vicuna Bincha_Bench, 1k samples
(b) Bench, 1k-samples
Base model: LLaMA-3.1-00
Base model: Mistral-70-v0.3
Base model: LLaMA-3.1-88
Base model: Mistral-78-v0.3
Win Rabe
Tie Rate
Loss Rate
Win Rate
Tie Rate
Loss Rate
Ours (L)
47.5%
11.3%
41.3%
48.1%
25.0%
26.9%
51.9%
20.8%
19.4%
42.5%
32.5%
25.00 0%
Ours(G)
57.5%
12.5% 0%
12.3%
50.6%
27.5%
21.9%
47.5%
35.0%
17.5%
Gurs(M)
(c) Vicuna Bench, 10k-samples
(d) MT MT_ Bence,, 10k-samples
Figure 6: Performance of models fine-tuned on DS² (1k/10k samples, machine-curated) v.s. LIMA (Ik samples, human-curated), We use the initial letter to denote the rating model, e.g., Ours (L) refers to our method with LLaMA-generate scores (Ours (LLaMA)).
ABALTION STUDY
6.1 REVISITING DATA SCALING LAWS
We conduct experiments under subsets with different data volumes to investigate the data scal- ing efforts. Compared to several representative baselines, Figure illustrates that our method can consistently obtain the best data selection performance across different data budgets. From this per- spective, while data quality matters, redundant samples are uninformative and unnecessary or even detrimental to model performance due to overfitting.
6.2 EXPLORING THE IMPACT OF SCORE CURATION
Score curation is beneficial for score-aware baselines Table further presents the experimental results of the other score-award baselines (AlpaGasus and Deita) using the curated scores. As shown in Table even though the fundamental variations in algorithms, it is evident that the score curation mechanisms still lead to performance improvements for all score-awe aware basclines. The full results using different rating models presented in the Appendix (Table 14).

---

Published as a conference paper at ICLR 2025
Table 5: Performance comparison between without and with score curation. Rating model: GPT-
4o-mini. Results are presented as (without curation / with curation).
LLaMA-3.1-8B
Mistral-7B-v0.3
ALPAGASUS
DEITA
OURS
ALPAGASUS
DEITA
OURS
MMLU
63.4 64.1
64.5/64.6
63.3/64.0
60,5/ 60.0
60.1 59.9
59.1
TruthfulQA
42.6 48.2
50.1 3
51.5/50.3
36.7
35.6/41.1
35.9/37.9
GSM
66.0/61.5
60.0
62.0/67.5
41.0/41.5
40.5/42.5
48.5
BBH
59.1/58.5
61.3/
59.7
53.1
55.1/55.3
54.2/
TydiQA
59.4 64.8
/6.7
66.
57.3/
36.0/56.4
58.9 59.3
Average
/8.1/
59.7 60.6
60.6
60.2 61.4
50.1 / 50.3
49.5/
51.5 $2.0
Score curation improves rating robustness Furthermore, explore the impact of score curation using different rating models. We compare the average performance results of DS between without and with score curation in Figure [ (Right). The base model is LLaMA-3.1-8B For convenience, F demonstrates the maximum performance gap across three rating models under different data sizes. Notably, it is evident that with score curation, the average performance across rating models is more stable and shows improvement.
AlpaGasus
+6.3
Ours
41.2
Performance gap
+1.1
+4.1
Data scale
w/o curation w curation
Perfo mance
2.5k
2.40 / 1.0
5k
3.83 / 1.20
20
10k
1.76 0.90
20k
1.73 / 0,20
+7.5
40k
1.44 / 1.63
Average
1.60 / 0.70
GSM
BBH
TruthfulQA MMLU
TyDiQA Average
Figure 7: Left: Apples-to-apples comparison with AlpaGasus using LLaMA-2-77 (base) on 9k samples from Alpaca subset (52k). Right: Maximum performance gap across different data scales.
6.3 APPLES-TO-APPLES COMPARISON WITH ALPAGASUS
To highlight D superiority, replicate AlpaGasus's 's settings for fair poples-to-apples com- parison. More details are in Appendix G.6 Using GPT-4o-mini for consistency, Figure ? (Left) demonstrates that DS signiiffiia outperforms AlpaGasus with improvement of 15% in aver- age, despite relying on a weaker rating model than AlpaGasus's default GPT-4 rating model.
7 CONCLUSION
In this paper, we challenge traditional data scaling laws in instruction tuning by introducing DS? a novel data selection pipeline that curates I.LM-rated quality scores improve data efficiency. Through the systematic exploration of error patterns in L.MM-ratdd data quality scores, we developed a score curation mechanism to correct inaccuracies and enhance the effectiveness of selected data. Empirically, D using only 3.3% of the original data outperforms training the full dataset (300k samples) and even exceeds the performance the human-aligned dataset "LIMA" with the same sample size (1k samples). This demonstrates that smaller, high-quality datasets achieve superior results by avoiding performance drops caused by low-rated or redundant data, revising the traditional scaling laws that suggest more data is always better. By curating L.LM-driven rating scores, D not only improves data efficiency, but also offers a cost-effective alternative to large- scale datasets and human annotations. Our results highlight the importance of data quality over quantity in instruction tuning and show how score curation can mitigate LLM biases, leading to improved model alignment and downstream performance. conclusion, this work underscores the need to rethink data scaling laws in light of more efficient, curated data selection methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
J. Pang and Y. Liu are partially supported by the National Science Foundation (NSF) under grants IIS-2007951, IIS-2143895, and IIS-2416896, J. Pang and Qian are also partially supported by NSF Grants 2322919, 2420632, and 2426031.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 088744 2023.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models.
Jiuhai Chen and Jonas Mueller. Automated data curation for robust language model fine-tuning. arXiv preprint arXiv:2403. 12077 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint w:Xiv:2307.08701 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3);6, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finee finettunf lan- guage models. Journal Machine Learning Research, 25(70): 2024.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for anformation-sseeiine question answering in ty pologically verse languages. Transactions of the Association for Computational Linguistics, 8:454 470, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.141683 2021.
Databricks. tuned
Introducing the world's first truly open instruction- https://www databriccssssscss...mm wwm httpr//icccccccoooo////2//2///// http://llog/22/////////// bbl.::/ww......rrcmmoereeee/eeee
Free
dolly:
11m.
2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403. /9443, 2024.
Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, and Ang Li. Shed: Shapley-based automated dataset refinement for instruction fine-tuning. arXiv preprint Stt::24050000003 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0333 033000000033, 2020.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2XXv:2/06.09666 2021.
11

---

Published as a conference paper at ICLR 2025
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Andreas Köpf, Yannic Kilcher. Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant pnnterssatons-dmmocraiiii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting 11m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. / /2032, 2023a.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprini arXiv:2308.066 06259, 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2109.07953 2021.
Minghao Liu, Zonglin Di, Jiaheng Wei, Zhongruo Wang. Hengxiang Zhang, Ruixuan Xiao, Haoyu Wang, Jinlong Pang. Hao Chen, Ankit Shah, et al. Automatic dataset construction (adc): Sample collection, data curation, and beyond. arXiv preprint arXiv:2408. 11388, 2024.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312. 15685, 2023a.
Yang Liu, Hao Cheng, and Kun Zhang. Identifiability of label noise transition matrix. In Interna- tional Conference on Machine Learning pp. 21475-21496. PMLR, 2023b.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648, PMLR, 2023.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27744, 2022.
N Reimers. Sentence- bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi. Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint 1rX.v:2210.09261, 2022.
Rohun Taori, Ishaan Gulrajani. Ting Zhang, Yann Dubois, Xiaodan Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cntruction-followiini llama model. https:
- www.:suuaaaasstttforr.mmo bobb/tttfoord.ooooooo a apacal 2023. GitHub repository.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2:Xiv:2307.09220 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint a 222.:22222..00000 2022.
12

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning open resources. Advances in Neural Information Processing Systems, 36.74764-74786 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning. arXiv preprint 220:22222.333,, 2024.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7610, 2020.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions. arXiv preprint arXiv:2304.12222 2023a.
Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundare- san. Rethinking the instruction quality: Lift is what you need. arXiv preprint arXiv:2312. 11500 2023b.
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. arXiv preprint vrXiv:2312.14158 2023.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets IIm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.171993 2024.
Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for aligmment: A simple but tough-to-bes baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833, 2024.
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L Zhang A preliminary study of the intrinsic relationship between complexity and alignment. arXiv preprint arXiv:2308. 05696, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 00046595-466223 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17205-17216, 2023.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pp. 12912-12923. PMLR, 2021.
Zhaowei Zhu, Jialu Wang, and Yang Liu. Beyond images: Label noise transition matrix estimation for tasks with lower-quality features. In International Conference on Machine Learning, pp. 27633-27653, PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
ORGANIZATION OF THE APPENDIX
Section Illustrates the limitations of this work.
Section B
details of prompt-hased L.M rating systems including details of the data pool and prompt template.
Provides
Section C Presents a warm-up binary example to illustrate how to derive the score transition
matrix, and the algorithm details of our proposed data selection pipeline DS In Appendix C.3 we analyze the k-NN clusterability hypothesis in detail. Besides, several 2-NN samples are also provided to evaluate the k-NN clusterability hypothesis.
Section D Explores the impact of embedding models.
Section Explores the impact of score curation on examples by analyzing the rated score distri- bution, subset distribution as well as the score transition matrix.
Section
Demonstrates training and evaluation details.
Provides more experimental results, including more downstream task evaluations, LLM judging evaluation, exploring the curation impact on score-aware methods, comparison with LIMA, new combined baseline which concatenating high-rated examples across rating models.
Section
Section
Analyzes the computational complexity and runtime.
Section
Explores the impact of diversity score used for data selection.
Section
Presents several wrongly-rated examples by three rating models used in this work.
A
LIMITATIONS
While the proposed method demonstrates competitive performance compared to other baselines, we acknowledge that there are still potential limitations:
Sample-independent assumption. The sample-independent assumption is critical for deriving the transition matrix and the true score probability distribution p. However, this assumption may be somewhat strong and could inevitably introduce certain data-specific errors. Exploring weaker weaker assumptions, such as group-dependent approaches, could valuable direction for future research.
k-NN clusterability. The k-NN clusterability hypothesis implies that similar embedding vectors should correspond the same rating score or class, a characteristic commonly leveraged im- age classification tasks. However, in text-related tasks, highly similar texts can convey opposite semantic meanings due to subtle differences, such as single word change. address this chal- lenge, powerful embedding models essential accurately distinguish these subtle differences and effectively capture the underlying semantic meaning.
Model scale. Our experiments are primarily conducted on pre-trained models the 7B/8B scale. It remains uncertain how well the method would perform on larger-scale pre-trained models.
Rating models. Due to cost considerations, we use the more affordable GPT-4o-mini to generate GPT-level scores. It is unclear whether the score curation mechanism works for more powerful GPT models (e.g., GPT-4 or GPT-ol).
B
PROMPT-BASED LLM RATING SYSTEMS
B.
DATA POOL
The data pool used in this work consists of five proceed datasets, which originate either from hu- man annotations or generated by powerful LLMs. More details about these datasets are provided in Table 6 In particular, these datasets vary in format, quality, prompt length, and target tasks, demonstrating the diversity of our basic data pool. For convenience, we standardize the format of these datasets by using the "TULO" template format introduced by Wang et al. al.] (2023). The "TULU" template consists of two main tags <lUser>> and <lAssistantl>, reflecting the respective roles of the user and the assistant.
14

---

Published as a conference paper at ICLR 2025
Table Details of training datasets used this work. Wi and Flan v2 are sampled 100K to match the dataset size. We report the average number of conservation turns (Nounss) average length of prompts promp))) average length of response (Erespose))
Datasets
Sourced from
# Data size
Data quality
Nrands
Lerompt
Lrequens
FLAN V2
Allman-geeneralne instruction
100K
Normal
1,0
304.1
27.7
OPEN-ASSINTANT
human-gene gennertee instruction
33K
Both
1.6
32.3
189.1
WIZARDLM
ChatGPT-generatee instruction
100K
High
1.0
122.3
352.5
DOLLY
Hlmaan-gencatte instruction
15K
Normal
1.0
99.5
79.3
STANFORD ALPACA
Generated w/ Davinci-003
52K
Normal
1.0
23,5
56.4
B.2 QUALITY-BASED PROMPT TEMPLATE
The prompt template used in this work across various rating models is presented as follows. Our prompt template mainly accesses the data quality based on three criteria including rarity, complexity, and informativeness. For clarity and convenience, we adopt JSON format to better capture the evaluation scores, following the LLaMA-3.1 template as shown in Table B.2
Prompt Template for LLM Rating
<System Prompt>: As data quality estimator, your task is to assess the quality of the data sample based on the criteria: Rarity, Complexity, and Informativeness. Please rate the sample on a scale from I to 10 for each criterion, and return an overall rating on a scale from to 10, where a higher score indicates a higher level of quality. Ensure that the ratings are not overly concentrated around a specific score. If multiple samples have similar qualities, consider spreading the scores more evenly to reflect subtle differences.
Please carefully evaluate the following data sample and return the integral evaluation scores using the ISON format:
<User Promptx:
"Rarity": <number, 1-10>,
"Complexity; <number, 1-10>,
"Iforratiiveness" <number, 1-10>, "veeall rating"; <number, 1-10>)
Instruction: [Instruction]
Input: [Input] Response: [Response]
Rated score rescaling Initially, to capture the subtle differences between data samples, we first prompt the LLMss rate them continuous integer scale 21, 10). Intuitively, lower score indicates that the data sample is of lower quality. To simplify the score distribution, we first merge the lower scores in 2, 3, 4} and the higher scores in {9, 10), resulting in new scale of .).. For ease of convenience, we then shift this scale down to (..., Note that LLM ratings, so merging low-rated examples would not affect the overall performance and is more convenient for analyzing score eтrors in Section
... (4.......
we focus primarily on high-rated samples
3.2 Directly rating samples on small scale of (0, I,... 55} seems more convenient but fails to capture the subtle difference between samples, especially among higher-rated samples. Meanwhile, this commonly leads to the issue where most of the samples are rated as 3. Starting with a larger scale and then narrowing it down allows LLMs to distinguish subtle quality differences in mid-rated samples better, improving performance.
C DATA SELECTION PIPELINE DS²
C.I WARM-UP OF DERIVING SCORE TRANSITION MATRIX: A BINARY EXAMPLE
For gentle start, let us consider binary case (K 2) with two types of scores {0, 1}. Here, y represents the ground-truth score, while ÿ denotes the observed noisy score. We define the error rates (transition probabilities) as ep1 := T(0, 1) == P(j 1 y = 0) and 610 := T(1,0) := Pi = 0 = 1). According to the K-NN clusterability definition, similar embeddings expected to
https:: /wwps 11 com/ htc. htmm//oos//mmmeeee ca and-pr-mpt- hond-ppromtt--oratss////e formats/1 11am//1111mmm//////
15

---

Published as a conference paper at ICLR 2025
belong to the same category. Specifically, we focus on 2-NN clusterability in this work, meaning that the scores for the three samples within a 2-NN cluster should be identical, i.c. y1 = 1/2 3/ y. Several target samples well their 2-NN samples are provided in Table g Note that the probabilities of the ground-truth score = IP(y = ii estimate the exact values of the error rates 601 and c10. the high-level idea is to leverage higher-order consensus among k--N cluster's s scores, as outlined below.
Vi E [K] also remain unknown. To
First-order Concensuses: We have
P(ù k)
Σ KE(K]
P(ū k k =), AA [K]
Then, we can obtain two first-order equations:
P = 0) == Po - c01) + (1 P110
P( 1) (1- (1-po)(1 e10) + Pocu1
Second-order Concensuses: We have
(0)
P(ū1 k. U2 = k²)
P(1 = k. 322 = k' 1 = i, = 1)P(11 = i)
KE|K]
P(yi k y1 i)P( = k' y2 1)P(11 i). Vk, E [K]
KE[K]
where equality (a) holds due to the 2-NN clusterability and equality (b) holds because of the conditional independence between ÿ1 and Ú2 based on their ground-truth score. Four second- order equations can be derived, e.g.,
PÍ 0, 822 = 0) == po(1 - em1)² + (1 po)e
PVū = 1, 702 = = =01
-1- (1-ro)({
-e10)² po
Third-order Concensuses: We have
P(β = k. 02 = = K. =k) =k" Σ P( k. = -k Us k" 1 = i, 1, 1,93 Pi) i)
EE[K]
Similarly, from different combinations of in, H2, Ds. we have eight third-order equations, e.g..
P( 1, 1, Us = 1) = (1 p1) coo) pocoo
Given the known score probability information PP = k). P(ū k, 02 k) and P(ūi k, 8/2 = k,, Ms = k3) ). we can utilize the above equations to derive the unknown ground truth score probability Po and error rates C10,C10- From these error rates, the transition matrix T then be determined. For the entire dataset, we summarize the score probability information across all 2-NN clusters derive the score transition matrix.
C.2 ALGORITHM DETAILS
We provide the algorithm details of our proposed data selection pipeline in Algorithm
C.3 KNN CLUSTERABILITY HYPOTHESIS ANALYSIS
In this paper, the k-NN clusterability hypothesis is very crucial, which is based on the assumption that embeddings capture semantic and contextual similarity for textual data, which often correlates with quality and correctness. Similar to image classification tasks, these high-dimensional repre- sentations map semantically similar texts nearby points in the vector space while positioning dissimilar texts farther apart, enabling clustering that aligns with classification categories. However, there may be a potential concern that samples with subtle token-level differences can yield different scores due to variations in correctness (the key factor). In this section, we will delve deeper into the practicality of the k-NN clusterability hypothesis for the following two reasons.
Firstly, scoring approach considers not just correctness but also overall quality metrics such as rarity and informativeness, as outlined in our prompt template. This helps mitigate the influence of correctness alone on the final score. Additionally, we evaluate quality granular scale (e.g..
16

---

Published as a conference paper at ICLR 2025
Algorithm Proposed Data Selection Pipeline DS
Input: Dataset D, Enbeddiigo RawScores, Target M
2: Output: Selected subset D
3: procedure MODELING ScoRE TRANSITION MATRiX(Pataset, BnbeddiingModel)
4: Step-1: Encode sample tuple and estimate score transition matrix
5: Features æ t ENCODING(Dat aset. EmbeddiingMdde)
6: ConsensusInfo k-NN STATISTICS INFo(RawScores:
7: Est -SSIMAATTTASSTTTOOO.AAAAASSSSSSE
Consensuses Equation
8: end procedure
9: procedure SCORE CURATION MBCCANNSMDataaattt.
Step-2: Identify and curate misrated samples
CosSimilarito tartyssoress +- -MMLAARTTSSORRRRSNNNSSSOOO RawSoor RawScores)
12:
Krror/hreshold = T_kst)
b Bayesian Rules
13:
MisratedSamples
Scores Maw
krrorthreshcld)
14:
ConfidenceFrobs INBALLANCERSSAAAANNHHHEEreeEddd
CuratedScores +- CCOOECCUAATOONiisratttssmeeeeee Cen.eeeeeeee
16:
end procedure
17: procedure LONG-TAIL SCoRING(Dataset Enbeddinggodel) Step-3: Calculate long-tail scores examples based on k-NN distance
19: for each sample's feature æ, in D do
20:
LengTailScore Scoors +- SIMLARIIY æn. features w) k-NN Based
21: end for
22: end procedure
23: procedure DATA SELECTION(Datases Embeddinggoodel)
24: Step-4: Leverage curated scores and long-tail scores to derive the selected subset D*
25:
D, + GGRounnNGCCurateddeso
represents the score for each group Prioritize high-rated samples
for score in {5, 4, 0} do
27:
Sort D; by LlongTailSooreee.. in descending order
28:
D <- SELECTTOP(D:)
Select Top M |D"| samples
29:
D D U D
D' equals M then
31:
break
32:
end if end for
33:
34:
Return 1)
35:
end procedure
{0, 1..... 10}. later compressed to {0, 1,..... ,5}) to reduce potential score discrepancies further. We provide randomly selected examples along with their 2-NN samples demonstrate the validity of k-NN clusterability in our data pool, shown in Table Moreover, we constructed specific exam- ples where the raw LLM scores and the calculated embedding cosine similarity scores consistently align, confirming the correctness of the kNN clusterability hypothesis.
Secondly, the consensus vectors rely on the average probabilities across all 2-NN clusters, allowing statistical information from the remaining samples to mitigate corruption caused by small number of violations. As result, our method can tolerate proportion of k-NN violations. Intuitively, prior work (Zhu et al. 1021) has demonstrated that even in image classification tasks, where 20% % of data samples violate the k-NN clusterability hypothesis, its method still outperforms other baselines. Empirically, experimental results support this claim. Furthermore, due to the unavailability of ground-truth scores, it is infeasible to conduct experiments to explicitly detect such violations.
Here, evaluate K-NN clusterability by examining the distribution of average score gaps, which measures the score difference within one k-NN cluster. The average score gap for a target sample defined as the mean absolute difference between the target sample's score and the scores of its k nearest neighbors, i.e.,
Average score gap = Mean [target samples score kNN sample's scorel).
In our work, we focus on 2-NN clusterability and frame our analysis within this context. Specifi- cally, for each 2-NN cluster, consider target sample and its two nearest neighbors. For example,
17

---

Published as a conference paper at ICLR 2025
given a 2-NN cluster with the score tuple: (target sample: 1, KNN sample 1: 2, kNN sample 2: 3), the score gap calculated as: Average score gap [1-2|+|1-31
1.5.
Table summarizes the statistical distribution of score gaps across all 2-NN clusters. For a clearer visualization of score gap proportions with and without score curation, we further provide Figure 8
Table Average score gap statistical information of all 2-NN clusters from our data pool. We divide the score gap into five groups and outline the proportion of data in each.
Curation
Model
Score Cap (0.0-1.0)
Score Cap (1.5) (%)
Score Gap (2.0) (%)
Score Gap (>2.0) (%)
w/o Curation
GPT
81.0
12.0
4,9
2.1
w/o Curation
LLaMA
58.3
18.0
12.2
11.5
Curation
Mistral
70.2
Curation
GPT
82.5
10.9
4.5
1.7
w/ Curation
LLaMA
78.8
9.4
7.3
4.1
w/ Curation
Mistral
80.5
From Table 7 we observe that without score curation, GPT has higher proportion of samples the 0.0-1.0 score gap range (81.0% compared to Mistral (70.2%) and LLaMA (58.3%). This reveals that more powerful rating models, such GPT, tend to exhibit smaller average score gaps, which aligns more closely with the concept of k-NN clusterability and contributes to improved performance.
Moreover, when comparing the settings with and without score curation, we observe that all three rating models show an increased proportion of samples in the 0.0-1.0 score gap range after score curation. Table summarizes this comparison, including the corresponding average performance on LLM Leaderboard tasks. Therefore, these results demonstrate the validity of the proposed k-NN clusterability hypothesis.
The proportion of samples the 0.0-1.0 score gap range both with and without score curation for each rating model. For comparison, the corresponding average performance LLM Leaderboard tasks is included in parentheses.
Table 8:
Rating Model
Score Gap w/o Curation (Avg. Performance)
Score Gap w/ Curation (Avg. Performance)
GPT
81.0% (60.2)
82.5% (61.4)
ILaMA Mistral
58.3% (59.2)
78.8% (60.2)
70.2% (60.7)
80.5% (61.1)
w/o curation
w/ curation
gpt go-mini
100k
Iistral-BBiinstruct.e
60k
Count
40k
20k
0k
0.0
0.5
1.0
1.5
2.0
>2.0
0.0
0.5
100
1.5
2.0
>2.0
Average score gap
Average score gap
Figure 8: Average gap statistical information of 2-NN clusters from data pool. The average score gap for each target sample is defined as the average absolute score difference between the target sample and its 2-NN samples.
18

---

Published as a conference paper at ICLR 2025
Table 9: Random selected target samples and their two nearest neighbors (2-NN) from data pool. These examples are provided evaluate the KNN clusterabilityy checking that similar embedding vectors correspond to the same category. The LLM rating score is from GPT--o-mint mini, not the ground truth score. The ground truth score should be judged manually based on individual evaluation.
Target sample
1st nearest sample
2nd nearest sample
<HHumanl: Pick any color from the rainbow.
<|HumanI>: Pick any color from
<iHumanib: List the colors of the
the rainbow.
rainbow
<lAssistantis: Purple.
<lAssistantb: Blue
</Assistantt>: Red, orange, yellow, green, blue, indigo, and purple
LLM rating score:
LLM rating score: 0
LLM rating score:
<HHuman>: You need complete
<|Humani>: Calculate 50% of 300.
<|Humani>: Calculate 15% of 500.
the following task: Calculate 15%
</Assistanti>: 50% of 300 is 150.
</Assistanti>: 75
the following number: 100
LLM rating score:
LLM rating score:
<lAssistanti>: 15% of 100 15.
LLM rating score: 1
<Huuman> what is the color of mars?
<(Humanl>: what the color of mars?
<|Humani>: Create multiple
choice question in which the answer Mars.
</Assistanti>: Rusty Red
</Assistanti>: Mars is called the
LLM rating score: 2
Red Planet because of its reddish color.
</Assistanti>: Which planet known as the "Red Plamet"?
LLM rating score:
A) Venus B) Jupiter C) Mars D)
Uranus, Answer: C) Mars
LLM rating score:
<iHumanl: What is the
Write response that completes the request, Convert the number 18 hexadecimal,
<[HumanI>:
<IHumanb: Convert this
hexadecimal representation of the decimal number 74?
hexadecimal value OxAF its equivalent decimal number </Assistantl>: 175
<lAssistantis: 4A
<lAssistanti>; The hexadecimal representation of 18 12.
LLM rating score:
LLM rating score:
LLM rating score: 0
<HHuman:: Use the word
<iHuman|>: Delete words from the following sentence
<IHumanb: Task: Edit the following sentence to make it more concise: "The food that we the party was absolutely delicious." " <lAssistantib: The food for the party was delicious.
"srrumptious" in sentence.
<lAssistanti>: The dinner was
make it a valid sentence:
cooked to perfection and was absolutely scrumptious.
I ate the most delicious dinner ever. <lAssistantib: ate most delicious
LLM rating score: 2
ever.
LLM rating score: 2
LLM rating score: 2
<umambn: List 3 applications of artificial intelligence.
<lHumani>: Name applications for artificial intelligence.
<IHumanb: Name three artificial intelligence applications.
<'Assistanti>: Some applications Artificial Intelligence include natural language processing, facial recognition, and self-driving cars.
<lAssistanti>: Three applications for artificial intelligence are natural language processing, computer vision, and facial recognition. LLM rating score: 3
</Assistanti>: Three artificial intelligence applications are: facial recognition software, natural language processing, and driverless cars.
LLM rating score: 4
LLM rating score:
<HHuman>>: Is there a negative or positive tone to this product
<iHumani>: there a negative or positive tone to this product review? Title: Did not use product. Review: I did not use the product because it had too many negative features to it. do not know the process for returning items. I would like to return this item!
<iHumani>: Is there negative or
positive tone to this product review? Title: Terrible. review: this product is already displaying problems, which probably mean that i will be returning it soon for replacement.
review? Title Worthless
Review: This manual has little that applies to any Caddy and what may be in there is hard to locate.
found more on the internet than in this manual! <lAssistanti>: Negative
</Assistanti>: Negative L.M rating score: 2
</Assistanti>: Negative
L.M rating score: 3
I.M rating score: 4
<Humman>: Is there a negative or positive tone to this product
<lHumani>: Is there a negative or
</Humani>: Q: Is there a negative
positive tone to this product
or positive tone to this product review? Title: A short cut Review: Great knives!!!! Work great and are absolutely beautiful. Own most of this line, and looking for more.
review? Title: Medium is a little small. Review; Great thumb picks with solid feel. These seem to run the small side and fit my thumb "snugly" but fortunately not too tight.
review? Title: Comfortable
Review: Would be five but was lucky to order size bigger then usual and they fit ok! Could even be little bigger but really comfortable!!
</Assistantl>: Positive
<Assiistntt>:: Positive
</Assistantl>: Positive LLM rating score:
LLM rating score:
LLM rating score: 1
19

---

Published as a conference paper at ICLR 2025
Average embedding distances across 2-NN clusters
73c
Counts
<K
20k
10.00, 0.03
10.03. 06)
10.08. 0.99
10 0.12)
(0.12 0.15)
(0.15.0.18)
10:18, 0.20
0.20)
0.27
(0.15,
10.21,
Distance ranges
Figure 9: Average embedding distances across 2-NN clusters from data pool. The embedding
model is BAAT/bge-large-ee
D EXPLORING THE IMPACT OF EMBEDDING MODELS
By default, we use the newly released open-source model BGE as the embedding model throughout this paper. To explore the impact of embedding models, adopt popular alternative SetenceBERT (Reimers] 2019) to encode data samples. The score transition matrix across various rating models the BETT embedding space is provided in Figure TO. Compared to Figure 3 in the BGE embedding space, we can observe that the impact of embedding space is limited, the choice of embedding model does not significantly affect the error patterns produced by LLMs.
GPT-4o-mini
LiaMA-3.1 3..--88-BnIsttrcc
Mistral 7B 77--Insruuct-O..
B:6
0.08
0.30
0.00
0.16
0.05
0.08
0.34
0.06
0.01
0.04
0.06
0.01
0:21
0:09
0.00
96
63
0.09
0.05
0.35
0.05
0.00
0.00
0.08
0.24
0.47
0.23
0.07
0.01
0.03
0.02
0.44
0.45
0.65
0.00
01
Labels
0.03
0.58
0.06
0.49
60
0
0.03
0.06
0.19
0.59
0.14
0.01
0.54
0.14
0.34
0.54
0.22
0.82
0.00
0.59
0.14
0.55
0.00
-2
42
0.00
0.00
0.07
0.23
0.65
0.05
0.08
0.00
0.08
0.42
0.44
0.02
0.00
0.08
0.04
0.37
0:11
0.00
0.01
0.49
2
5
0
2
Labels
Labels
Labels
Figure 10: Score transition matrices across various rating models in the SentenceBERT embed- ding space.
E EXPLORING THE IMPACT OF SCORE CURATION ON EXAMPLES
E. IMPACT OF SCORE CURATION ON DISTRIBUTION
Rated score distribution between without and with curation Here, we compare the rated score distribution between without and with score curation, as shown in Figure We observe a decrease in the number of high-rated examples, while the number of samples with rating of has increased significantly. The rationale behind this is that our score curation mechanism is based on k-NN statistical information. As a result, given the imbalanced distribution of rated scores, samples with rating of 5 are rare and are inevitably drawn toward the majority rating of 3. Therefore, the results in Figure II also highlight the importance of confidence probability proposed in Section
Subset distribution of selected examples Recall that the data pool is constructed by five subsets. Here, summarize the statistical information of 10K samples generated by DS?, focusing the proportion of subsets. We can observe that 60%-70% of selected examples are from WizardIm. The observation corresponds the differences in data quality across five subsets summarized in Table
20

---

Published as a conference paper at ICLR 2025
GPT-4o-mini
LiaMM--..18B-Instruut
Mistral-7B 7B-Inssruucttt...33
150K
Raw score
125K
Curated score
100K
75K
2>K
5
3
Scores
Scores
Scores
Figure 11: Comparison of rated score distribution between without and with score curation.
GPT-4o-mini
LLaMA-3 1aMMA-3.1-88-nnsrruu
Mistral-7B-Inttrutt-0..3
Without curation
with curation
6418
6114
6452
2114
2202
866
dolly
Raan 42
stanford alpaca
wizardim
cassti
dolly
faan V2
stanford alpace
wizardim
oasst1
dolly
flen. V2
stanford alpace
wizardim
0asst1
Figure 12: Subset distribution proportion within 10K samples generated by DS?.
E.2 IMPACT OF SCORE CURATION ON SCORE ERRORS
Instead of the impact of score curation final performance, we are also interested in the impact of score curation on the detected score transition matrix. [i illuttrates the error pattern of differ- ent rating models after applying score curation. In comparison to the results without applying score curation illustrated in Figure the improvements are remarkable. Our score curation mechanism can significantly reduce the probability of incorrect score transition in the matrices.
GPT-4o-mint
LiaMA-3.1 BMM--331-BBIInttucc
Mistral- 78-Instructt-v
0.12
0.59
0.27
0.01
0.00
0.00
06
0.98
0:51
0.15
0.23
0.63
0.00
0.00
0.43
0.22
0.34
0.01
0.00
65
2
0.03
0.12
0.64
0.17
0.04
0.00
0.06
0.14
0.42
0.26
0.07
0.00
-44
0.00
0.06
0.46
0.47
0.00
0.00
0.66
-03
42
A3
0.06
0.43
0.00
0.50
0.00
0.00
0.12
0.39
0.12
0.00
0.63
0.04
0.06
0.47
0.01
0.01
0.45
0.00
à
1
3
5
Scores
Scores
Scores
Figure 13: Score transition matrices comparisons across different rating models with score curation.
F SETUP DETAILS
Training details In our experiments, we fine-tune 7B and 8B models using four or eight NVIDIA Tesla A 100 GPUs. Following the experimental setup (Wang et al.) 2023), for all experiments based on 7B/8B models, we consistently apply Lora (Hu et al. (021) with a rank-size of 64 and a scaling factor of 16. Then, set the overall batch size 128, the learning rate 1e-4, the training epochs
21

---

Published as a conference paper at ICLR 2025
to 5, the dropout rate to 0.1, and warm ratio of 0.03. The default maximum input length is 2048 tokens for all models.
Evaluation details In this paper, we select five tasks to conduct experiments for evaluation, con- sisting of MMLU, BBH, GSM, TydiQA, and TruthfulQA. The hyperparameter settings mainly fol- 2023 ss. For case of reproduction, we present some brief details.
low recent work (Wang et al.
2020g: Following the setup of MMLU, we conduct all evaluations in the O-shot setting without chain-of-thoughts (CoT).
MMLU (Hendrycks et al.
GSM (Cobbe et al., 2021): We evaluate fine-tuned models on a randomly selected subset with 200 samples from the original test set (1319 samples). In particular, we apply 8-shot in-context examples to simulate the CoT setting for reasoning.
BBH (Suzgun et al., 2022): Given the official prompts provided in (Suzgun et al. 2022). we also apply 3-shot settings without CoT to make generations. Besides, we select 40 examples from each BBH sub-task.
TruthfulQA (Lin et al. 2021): ) We prompt the fine-tuned models to generate answers for 818 TruthfulQA questions using the default QA prompt template with 6 in-context examples. Follow- ing the setting of (Wang et al., 2023), We apply two LLaMA-2-7B-ban generated responses' truthfulness and informativenes:f Judge models will help evaluate the truthful and informative rate of responses, respectively. We use 8-bit quantization to allow for efficient generation. Following (Lin al., (02I). we finally take the Informative-Truts our metric, which is calculated by the numerical product of the Informative and the Truthful Rate.
models for judging the
Rate as
T Clark et al., 2020): This dataset used to evaluate the model performance in answering multilingual questions across nine different languages. For each language, select 100 exam- ples. To help the models become familiar with the answer format, one in-context example is provided during testing, We report the average Fl score across various languages in this paper.
G MORE EXPERIMENT RESULTS
G.1 OPENLLM LEADERBOARD EVALUATION RESULTS
We conduct additional experiments to evaluate the performance the OpenI.LM leaderboard across different baselines, utilizing various base models such Mistral-7B-v0.3 and LLaMA-2-7B-ht Ta- bles (0s]Taame [T present the results of the OpenLLM leaderboard using Mistral-7B-vO. and LLaMA- 2-7B hf as the base model, respectively. Both tables consistently demonstrate the effectiveness and superiority of our proposed pipeline DS?, following the previous claims provided in Secion
G.2 LLM JUDGE EVALUATION
To evaluate alignment performance across baselines, we utilize Vicuna-Bench to access the instruction-follooiing ability (Chiang et al.] 2023). Vicuna-Bess Benchh contains questions across nine domains, including generic, coding, math, and counterfactual. The judge model is GPT-4o-mini. Similarly, we present the final judge result in the typical "Win-Tie-Loss rate form. For conve- nience, the judge prompt template referenced in (Zheng al. 2023) can be found in Table 12
We compare all baselines, including our method against the full data baseline on Vicuna_Bench, as shown in Table 13 In particular, we conduct evaluations on two base models LLaMA-3, w.1-8B and Mistral-7B-ve 3. For score-aware baselines (AlpaGasus and Deita), we also compare them under three rating model settings. Notably, our method with curation outperforms almost all other baselines. What's more, in most cases, we can observe that the score curation step improves model performance by reducing the loss rate without compromising the original win rate.
G.3 EXPLORING THE CURATION IMPACT ON OTHER SCORE-AWARE METHODS
Here, we present the curation impact on other score-aware methods, especially for Alpagasus and Deita under different rating model settings. The full experimental results can found in Table
22

---

Published as a conference paper at ICLR 2025
Table 10: Performance comparison on OpenLLM leaderboard. By default, the selected data size is 10K. Base model: Mi Mistral-7B-vC. We highlight the best result in boldface and the second- best with underline.
MMLU (factuality)
TruthfulQA (truthfulness)
GSM (reasoning)
BBH (reasoning)
TydiQA (multilinguality)
Average
Models
VANILLA BASE MODEL
59.7
30.2
38.0
49.6
54.9
46.5
COMPLETION LENGTH
58.9
34.4
42.5
53.1
59.6
49.7
PERPLEXITY
59.8
40.3
36.0
48.9
57.4
k-NN-10
41.7
43.5
53.4
50.2
RANDOM SELECTION
59.4
36.7
41.8
54.2
$4.0
49.3
LESS
59.5
34.8
42.0
54.5
57.5
49.7
FULL DATA (300K)
60.0
43.5
43.5
52.5
53.4
50.6
Rating model:
ALPAGASUS
36.4
39.0
56.3
DEITA
60.0
37.1
43.5
54.0
57.7
50.5
OURS W/O
60.0
37.2
45.0
53.5
54.5
50.0
OURS
59.7
37.8
48.5
54.4
55.2
51.1
Rating model: GPT-4o-mini
ALPAGASUS
60.5
36.7
41.0
57.3
DEITA
60.1
35.6
40.5
55.1
56.0
49.5
OURS w/O CURATION
60.1
35.9
48.5
54.2
58.9
51.5
OURS
59.9
37.9
47.5
55.6
59.3
52.0
Rating model:
Mistral-7B-Instruus
ALPAGASUS
59.5
35.6
46.0
52.1
DEITA
59.9
40.0
43.5
56.9
53.1
50.7
OURS W/O CURATION
59.5
37.9
46.5
55.8
57.2
51.4
OURS
59.5
40.3
48.5
53.0
55.9
51.4
Table 11: Performance comparison on OpenLLM leaderboard. By default, the selected data size is 10K. Base model: LLaMA - 22-0-----BBB- We highlight the best result in boldface and the second-best with underline.
MMLU (factuality)
TruthfulQA (truthfulness)
GSM (reasoning)
BBH (reasoning)
TydiQA (multilinguality)
Average
Model
VANILLA LLAMA-2-7B
41.9
28.4
6.0
38.3
35.7
30.1
COMPLETION LENGTH
42.4
36.4
36.8
33.9
30.2
PERPLEXITY
45.0
41.5
12.0
31.7
39.5
33.9
6-NN-10
38.2
40.8
15.0
36.0
43.8
34.8
RANDOM SELECTION
44.7
41.8
14.0
37.9
40.8
35.8
LESS
44.3
38.2
18.0
35.2
46.3
36.4
FULL DATA (300K)
50.1
36.2
16.5
40.5
46.7
38.0
Rating model:
1ama-331-8B-Insteeee
ALPAGASUS
45.1
41.2
18.0
35.6
39.8
35,9
DEITA
43.6
36.4
14.5
33.9
39.7
33.6
OURS W/O CURATION
45.4
39.7
15.0
35.5
42.1
35.5
OURS
44.9
44.9
14.0
38.3
44.8
37.4
Rating model: GPT-4o-mini
ALPAGASUS
45.3
41.0
14.5
37.0
45.3
36.6
DEITA
45.2
44.7
13.5
35.6
43.4
36.5
OURS w/o CURATION
42.0
39.5
15.0
38.1
46.1
OURS
40.2
43.8
13.5
38.9
46.5
36.6
Rating
Histral-7B-Instrueee
ALPAGASUS
42.3
41.9
16.0
34.1
41.6
35.2
DEITA
41.1
35.7
42.9
OURS W/O
46.0
48.6
15.0
35.2
43.7
37.7
OURS
40.8
50.9
15.0
37.9
45.5
38.0
G.4 COMPARISON WITH HIGH-QUALITY HUMAN-ANNOTATED EXAMPLES: LIMA
In this section, we also utilize the original LIMA test set (300 samples) to compare the performance between LIMA (human annotation) and D (machine annotations). Similarly, we finetune two
23

---

Published as a conference paper at ICLR 2025
Table 12: The prompt template used for GPT-4o judge evaluation from (heng al. 2023
LLM Judge Prompt Template
System Prompt:
You are helpful and precise assistant for checking the quality of the answer.
User Prompt:
[Question]
[Assistant I]: Assistant I's Answer
[Assistant 2}: Assistant 2's Answer
We would like to request your feedback on the performance of two Al assistants in response to the user question displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on scale of to 10, where higher score indicates better overall performance. Please first output single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores are separated by space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order which the responses were presented does not affect your judgment.
Table 13: Performance comparison with full data baseline on Vicuna Bench.
Base models:
LLaMA-3. -BB and Mistral-7B-vDD 33. LLM judge model: GPT-4o-mnni Win represents the adjusted win rate, which equals the win rate plus half of the tie rate. We highlight the best result in boldface and the second-best with underline.
LLMMA-3.1-8B
Mistral-7B-voe
Model
Win %)
Loss(%)
Tie(%)
Win %)
Win %))
Loss %)
Tie(%)
Win %6)
COMPLETION LENUTH
55.5
32.5
12.0
61.5
61.3
25.0
13.8
68.1
PERPLEXITY
35.6
51.3
13.1
42.2
45.0
38.8
16.3
53.1
K-NN-10
51.3
29.4
19.4
60.9
51.3
32.5
16.3
59.4
RANDOM SELECTION
33.1
45.0
21.9
44.1
46.3
35.0
18.8
55.6
LESS
35.0
51.3
13.8
41.9
36.3
48.8
15.0
43.8
Rating model:
L..MMA3311 -3.188... Instruct
ALPAGASUS
50.6
28.8
20.6
60.9
57.5
27.5
15.0
65.0
DEITA
40.6
45.0
14.4
47.8
46.3
36.3
17.5
55.0
OURS w/o CURATION
56.3
30.0
13.8
63.1
55.0
30.0
15.0
62.5
OURS
53.8
27.5
18.8
63.1
63.8
22.5
13.8
70.6
Rating model: GPT-4o-mini
ALPAGASUS
67.5
18.8
13.8
74.4
73.8
10.3
15.9
81.7
DEITA
54.6
32.1
13.3
61.3
63.1
26.3
10.6
68.4
OURS W/O CURATION
70.4
19.6
10.0
75.4
67.5
22.5
10.0
72.5
OURS
63.8
20.0
16.3
71.9
65.0
20.0
15.0
72.5
Rating model: MMittall-BB-Instruue IB-tuutruut--
ALPAGASUS
48.8
22.5
28.8
63.1
55.0
28.8
16.3
63.1
DEITA
46.3
36.3
17.5
55.0
45.0
41.9
13.1
51.6
OURS CURATION
51.7
33.8
58.9
61.9
25.0
13.1
68.4
OURS
51.3
31.3
17.5
60.0
62.5
20.0
17.5
71.3
base models (LLaMA-3.1-5) --BBBBBBBBBBBB-- and Mistral-7B-v0.3) 1k LIMA samples. The finetuned models are then directly compared with finetuned models using DS selected examples at both 1k and 10k sample sizes. The experimental results for 1k and 10k settings are shown in Figure 14a and [4b] respectively. While DS² performs worse than LIMA in the 1k sample setting, it totally surpasses LIMA in the 10k setting, consistently demonstrating the superiority of DS?. This lower performance at the 1k setting is expected, as LIMA has a natural advantage in a limited sample size scenario due the IID nature of its training and test sets.
24

---

Published as a conference paper at ICLR 2025
Table 14: Performance comparison between without and with score curation across all score-aware methods. Results are presented as (without curation / wwth curation). The selected base models are LLaMA-3. 1-8B and Mistral-7B--V.
Rating Model: -.L.MA-31-8B-IIstte
LLaMA-311-B
Mistral-7B-v0.3
ALPAGASUS
DEITA
OURS
ALPAGASUS
DEITA
OURS
MMLU
63.1 / 63.8
64.1 / 64.6
63.4/63.8
59.9/
60.0 59.8
60.0 60.0
59.7
TruthfulQA
/3.11
35.3 46.
50.2/45.4
36.4/41.7
37.1/
37.2
GSM
59.5 /5.55
60.0/ 64.0
61.5/
39.0/40.0
43.5/43.0
48.5
BBH
/3.9
60.8 58.3
59.3/61.2
52.6/53.5
54.0/
54.4
TydiQA
64.8/ 62.7
63.0
61.7/67.9
56.3
57.7
/
Average
58.1 58.2
56.6/ 58.9
/
/ 49.4
50.5
50.0 51.1
Rating Model: GPT-4o-mini
LLaMA-3.1-8B
Mistral-7B-v0.3
ALPAAGASUS
DEITA
OURS
ALPAGASUS
DEITA
OuRS
MMLU
63.4 64.1
64.5 64.6
64.3
60.5 60.0
60.1 60.1
60.1/59.99
TruthfulQA
42.6 48.2
50.1 45.5
50.5
36.7/
35.6/41.1
35.9/37.9
GSM
66.0/61.5
60.0/ 64.0
62.0/67.5
41.0/41.5
40.5
48.5
BBH
58.1
60.3 61.8
59.7/
53.1
54.2 55.1/55.3
TydiQA
59.4/ 64.8
63.7 67.11
66.
5
56.0/56.4
58.9/59.2 59.3
Average
58.1 59.5
59.7 60.6
60.2/
50.1
4
51.5/ 52.0
Rating Model: Misstral-7B-Instruct-vv
L.AMA-3.1-BB
Mistral-7B-v0.33
ALPAGASUS
DEITA
OURS
ALPAGASUS
DEITA
OurS
MMLU
63,2/66 64.2
63.9/63.5
63.0/63.3
59.5 59.6
59.9
/9.5
TruthfulQA
45.8/40.0 40.0
50.3
48.2/53.9
35.6/38.9
40.0/38.7
40.9
GSM
62.0/60. 60.5
61.0/6100
67.0/62.0
46.0/46.5
43.5/44.0
46.5
BBH
63.5
59,5
59.2/61.1
55.7
56.9/54.1
55.8/ 53.0
TydiQA
63.2
62.8/64.5
65.9
52.1
53.1/55.1
5
Average
58.7
59.7 / 60.0
61.7
49.8 / 51.4
50.7
514/ 51.4
G.5 EXPLORING THE IMPACT OF CONCATENATING HIGH-RATER EXAMPLES ACROSS RATING MODELS
Combined baseline Here, we are also interested in the performance of concatenating samples from three rating models. We combined all high-rated samples with score of 5, resulting subset of 8K samples. To reach total of 10K samples, we added 2K samples from the data pool that were both rated by all rating models. Compared to the results shown Table 3 and Table I0 one can observe that the combined baseline still fails to achieve strong performance.
Table 15: Performance of COMBINED baseline on OpenI.LM Leaderboard.
Combined baseline
LLaMA-3.1-8B
Mistral-7B-v0.3
MMLU
64.2
59.6
TruthfulQA
41.7
37.1
GSM
62.5
43.5
BBH
61.9
51.0
TydiQA
60.8
53.1
Average
58.2
48.9
25

---

Published as a conference paper at ICLR 2025
Base model: LLaMA-3.1-8B
Base model: Mistral-7B-v0.3
Win Rate
Tie Rate
Loss Rate
Ours (L)-1K-
Ours (G)-1k-
42.5%
15.7%
41.8%
35.8%
15.7%
48.5%
Ours (M)-1k-
8%
6%
(a) LIMA Test, Ik-samples
Base model: LLaMA-3.1-88
Base model: Mistral-7B-vOl
Win Rate
Tie Rate
Loss Rate
Ours (L)-10K
Ours (G)-10k
57.5%
14.0%
28.4%
49.5
14.7%
35.8%
Ours (M)-10k-
45.8%
14.0%
40 1%
46.7% 17.0%
36.3%
(b) LIMA Test, 10k-samples
Figure 14: Performance of models fintuned on DS² (10k samples, machine-curatedd v.s. LIMA (Ik samples, numan-curated). Evaluation set: LIMA (300 samples). We use the initial letter to de- note the rating model, e.g., Ours (L) refers our method with (LaMA-generatee scores (Ours (LLaMA)).
G.6 APPLES-TO-APPE PERFORMANCE COMPARISON WITH ALPAGASUS
Note that the raw scores used this work for AlpaGasus Chen al. 20223 are generated with prompt template. Our prompt template largely follows the format and criteria of Alpagasus (as the first rating prompt template), maintaining aligmment with established standards. A significant improvement in approach is using JSON format to return evaluation scores, allowing us to capture the scores accurately. This ISON formatting approach is inspired by the official LLama-3.1 chat template, detailed in LLama-3.1 model documentation! We conduct experiments compare our method with AlpaGasus under the same 4-bit quantization and LoRA settings, adhering closely the same experimental configurations. The AlpaGasus 2- 2-7B-QLoo QLORA model originates from related repository highlighted in the official AlpaGasus repository, with LLaMA-2-7B as the base model. The rating scores used in our method are generated from GPT-4o-mini, which is much weaker than GPT-4 used in AlpaGasus.
H COMPUTATIONAL COMPLEXITY
Table summarizes the storage and GPU running time method as well as three representative baselines. The wall-clock running time is measured on Microsoft Azure A*A000 (80GB) GPUs cluster. Note that our score curation mechanism relies primarily on linear programming (LP), which runs exclusively the CPU. As shown the table. LLM rating systems are ad advantageous over the gradient-based method LESS in terms of both storage and runtime. Notably, compared to AlpaGasus and DEITA, our method avoids any significant computation costs the GPU.
EXPL.ORING THE IMPACT OF DIVERSITY SCORE
The importance of diversity on LLM data selection has been extensively explored by previous work
Wang et al. (2023): Liu et al. (2023a); Wang al. (2022). Note that data pool is composed of five distinct subsets, each characterized by varying levels of complexity and diversity. The statistical analysis of diversity scores across subsets, as illustrated in Figure 15 confirms this. To evaluate
26

---

Published as a conference paper at ICLR 2025
Table 16: Comparison of storage and running time.
Storage
Running Time
Base Model Free
Validation
Rating/Gradient
Diversity Score CPU-only Curation
Data Selection
6611
Required
AlpaCiasus
<10MB
6H
<lmins
Yes
Not Required
DETTA
<10MB
611
10 mins
Yes
Not Required
Ours
<10MB
611
25 mins
<lmins
Yes
Not Required
Diversity score distribution
10000
flan v2
oasstl
8000
wizardlmm
dolly
stanford aalaca
Count
4000
2000
0
0.00
0.02
0.04
0.06
0.08
0.10
0.i2
0.14
Diversity score value range
Subset diversity score distribution. The diversity distribution across subsets demon- strates that the complexity and diversity are different.
Figure 15:
the versatility of the diversity score, further conduct additional contrast experiments here. In particular, we solely rank the samples of subsets based on the diversity score. Then, we select the Top-k and Bottom-k samples independently construct datasets for LLM instruction finetuning, where k =10000. The corresponding performance results are presented in the following table. For cost considerations, we employ LLaMA-3,2-33 as the base model. The experimental settings are consistent with those outlined in paper. From the table, it is evident that the diversity score is not universally effective across all datasets. To achieve better results, should be complemented with other specific metrics, such LLM rating scores.
Table 17: Performance comparison between low and high diversity score across different datasets. The base model LLaMA-3. 2-3B. Bottom-k (Top-k) refers to the samples with the lowest (high- est) diversity scores, where k 10000.
Metric
Flan v2
Wizardlm
Stanford Alpaca
Bottom-k
Top-k
Bottom-k
Top-k
Bottom-k
Top-k
MMLU
55.6
54.8
56.7
56.6
56.5
55,1
BBH
44.9
45.0
45.8
47.7
46.3
47.1
GSM8K
24.5
29.5
30.5
28.5
28.5
26.0
TruthfulQA(M(2)
38.6
41.7
46.6
48.1
40.2
40.6
Tydiqa
55.9
60.5
37.7
31.2
48.4
39.5
Average
43.9
46.3
43.5
42.4
44.0
41.7
MIS-RATED EXAMPLES
Here, we provide more wrongly-rated examples detected by our algorithms. For convenience, we highlight the raw (original) rated score and the curated score of each example.
27

---

Published as a conference paper at ICLR 2025
Wrongly- rated example by LLLMA--.11-88 -Ss-nustr Raw rated score: 5; Curated score:l
Here is some data about a restaurant: name The Fagle, eatType coffee shop, food = Fast food, customer rating = 3 out of 5, area riverside, familyFriendly = yes, near Burger King. Write a sentence that includes the above data about restaurant
<User>:
<Assistant>; The Eagle is kid-friendly coffee shop that is located on the riverside near Burger King. The Eagle serves fast food and is moderate in pricing. Customers rate the coffee shop 3 out of 5.
Wrongly-rated example by LLaMA-3.1-8D 3.--settntttt Raw score: Curated score:1
Based on the premise think would too", can we conclude the hypothesis "I also think it would. is true (see options)? Pick from: (i) yes; (ii) it is not possible to tell; (iii) no: <Assistant>: (i).
<User>:
Wrongly-rated example by GPT-4o-mini Raw score: 5; Curated score:0
<User>::remiin Island society has
change in the last twenty years than in the previous thousand. Hypothesis: Island society has been changing rapidly because of boats with powerful motors. the hypothesis entailed by the premise? Select from: 1. yes. 2. is not possible to tell. 3. no. And the answer is: <Assistant>:
Wrongly-rated example by GPT-4o-mini. Intial score: 5; Curated score:
De façon plus précise, si acc sec с et représentent la consommation totale des agents avec et sans contraintes de crédit, respectivement, léquation de consommation globale devient
<User>:
acc. Which language is this? <Assistant>; French.
Wrongly-rated example by GPT-4o-mini. Raw score: Curated score:2
<User>:
Ask question about Queen Victoria.
<Assistant>; Who did Victoria dislike so much she banned him from her presence once she became Queen?
Wrongly-rated example by Mistral-7B-v0.3 18-v0.3-Inntrruc) Raw score: 5; Curated score:1
<User>: Add punctuation: Yahweh sent Jerubbaal and Bedan and Jephthah and Samuel and delivered out of the hand of your enemies on every side and you lived in safety <Assistant>: Yahweh sent Jerubbaal, and Bedan, and Jephthah, and Samuel, and delivered you out of the hand of your enemies every side; and you lived in safety.
Wrongly-rated example by Mistral-7B 7Bt....--000...t... Raw score: 5; Curated score:1
Die Hausordnung ist Bestandteil der allgemeinen Geschäffibeddngungenn Translate this to English?
<User>:
<Assistant>:
The house rules of the Hotel are an integral part of the General Rules and Conditions.
28

---

Published as a conference paper at ICLR 2025
Wrongly-raie rattd example by Mistral-7B-vO.. 7B-v0.3-Instruct. Raw score: 5; Curated score:l
Generate a sentence that describes the following data: Alan Frew, occupation, Public speaking; Alan Frew, birthPlace, United Kingdom; Alan Frew, origin, Canada; Alan Frew, background, "solo singer"; Alan Frew, origin, Newmarket, Ontario. Sentence: <Assistant>: Andrew Frew was solo singer who makes his living public speaking. Alan Frew was born the UK but has Canadian origins coming from Newmarket, Ontario.
<User>:
J.1 ORIGINAL HIGH-RATED EXAMPLES Vs. ORIGINAL LOW-RATED EXAMPLES
We provide several high-ratted/oww-rtte examples from different datasets to illustrate the data qual- ity, including Flan_ v2, Open- Aesn-Assitaant,, Stanford Alpaca, Dolly, and WizardLM.
Table 18: Comparison of low-rated and high-rated samples in Dolly and Stanford Alpaca datasets
Low-rated Dolly sample
High-rated Dolly sample
<iHumani>: Instruction: Below an instruction that describes lask, paired with an input that provides further context. Write response that appropriately completes the request.
Below an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
<IHumanl>: Instruction:
Instruction: When was the 8088 processor released?
Instruction: Who is the best surfer in the world? Input: Robert Kelly Slater (born February 11, 1972) is an American professional surfer, best known for being crowned World Surf League champion a record times. Slater is widely regarded as the greatest professional surfer of all time, and holds 56 Championship Tour victories. Slater is also the oldest surfer still active on the World S League, winning his 8th Billabong Pipeline Masters title age 49.
Input: The 8086 (also called iAPX 86) a 16-bit microprocessor chip designed by Intel between early 1976 and June 8, 1978, when was released. The Intel 8088, released July 1, 1979, is slightly modified chip with an external 8-bit data bus (allowing the use of cheaper and fewer supporting ICs), I] and is notable the processor used in the original IBM PC design, <'Assistantls:
</Assistant>>:
Response: The Intel 8088 processor was released
Response: Kelly Slater
July I, 1979.
Low-rated Stanford Alpaca sample
High-rated Stanford Alpaca sample
<iHumani>: Instruction: Given set of numbers, find the maximum value.
<IHumani>: Instruction: Create list of best
practices that should be followed in this field.
Input: Set: {10, 25, 6, 16)
Field: Hcalthcare information technology <lAssistanti>:
<lAssistanti>:
Response: The maximum value is 25.
Response:
The following best practices should
be followed in the healthcare information
tcchnology field:
1. Adopt risk-based approach cybersecurity, utilizing measures such as multi-factor authentication, encryption, and vulnerability assessments.
2. Develop clear guidelines and policies governing the use of technology.
3. Develop protocols to ensure the secure sharing of patient data.
4. Implement robust data processing and storage
systems that adhere to industry regulations.
5. Train healthcare staff in both the technical and non-technical aspects of IT use.
6. Utilize professional ITT staff for any maintenance or development tasks.
7. Regularly test and monitor systems for potential vulnerabilities.
29

---

Published as a conference paper at ICLR 2025
Table 19: Comparison of low-rated and high-rated samples from Flan v2 and WizardLM datasets
Low-rated Flan_y2 _lan_v2ao_ sample
High-rated Flan_y2 sample
<iHumanl>: Triple: Alimentum catType restaurant; Alimentum food Indian; Alimentum priceRange moderate; Alimentum area riverside; Alimentum FamilyFricnnly no, What a sentence that describes this triple?
<|Humani>: walked in here during an all day rain-a-thon in Pittsburgh (so rare, right?) on a Saturday. It was pretty busy, but all of the hustle and bustle was contained to the upstairs, which looked like cool loft area. It wasn't 't loud either. The first thing I noticed is how high the ceiling was and loved that. The art on the walls was a delight to look at too. I ordered something vegan with tofu (I' not vegan by any means, but wanted to try it), and got it delivered to me on a plate with two small toasted pieces of bread, lettuce, slices or bell pepper, and warmed tofu. There was something clse the bread that I don't remember, but was really surprised by how delicious and satisfying this was. am a fan. For my drink I got iced vanilla latte and have to say s the best vanilla latte I've had around town. I ended up having two more throughout my time there...I just inhaled them. Seriously. good. I came back on Sunday and had an iced mocha, which was just good. This is my new favorite place for real!
</Assistantl>: "Alimentum is an Indian restaurant located the the riverside area. is moderately priced, but not kid friendly."
Choose: was this review given positively or negatively?
<lAssistant|>: positive
Low-rated WizardLM sample
High-rated WizardLM sample
<iHumani:: Instruction: As an intermediate English learner at B1 level, I have been trying to familiarize myself with certain words like "captivity". "chew out" and "cuddly". Would you kindly ask me a question using any of these three terms, to help me better comprehend the usage and context of these words? <'Assistantl>:
<IHumani>; Instruction: As an archacologist exploring the depths of an uncharted jungle, you
come across a temple that appears to be blend of Mayan and Roman architecture. The engravings on the walls and artifacts found within the temple suggest that the civilization that built it was a fusion of both cultures. Your task is to piece logether the history and culture of this ancient civilization and unravel the mystery of how and why it came to be. Were these two cultures united peace or war? Did they form a new civilization or remain separate? Let's excavate the truth behind this enigma. <IA <sAssitta:)>>:
Response: Certainly! Here are three questions using each of the terms you mentioned:
Have you ever visited zoo where animals are kept in captivity?
2. Has your boss ever chewed you out for making a mistake at work?
3. Do you prefer to have pet that is cuddly and affectionate, or one that independent and aloof?
Response: As you begin your exploration of the temple, you come across a set of glyphs on one of the walls that seem to represent both Mayan and Roman symbols. You take note of them and continue your journey deeper into the temple. Inside, you find a variety of artifacts that suggest a fusion of both cultures, including pottery, weapons, and clothing....
In the end, you realize that this ancient civilization was a testament to the power of cooperation and cultural exchange. The Mayan and Roman peoples came together in peace. combining their knowledge and skills to create something new and beautiful. Their legacy lives on in the unique artifacts and engravings that remain in the temple, waiting to be discovered and appreciated by future generations.
30

---

Published as a conference paper at ICLR 2025
Table 20: Comparison of low-rated and high-rated samples from Open-Assistant I dataset
Low-rated Open-Assistant sample
High-rated Open-Assistant sample
<iHumanl>: que actividades se supone que son comunes una persona de 20 años?
<lHumani>: Hazme una clave dicotómica de los animales invertebrados.
<(Assistanti>> Las actividades más comunes entre personas de 20 años usualmente son estudiar, debido la alta demanda de tareas en la universidad, así como, salir a divertirse, tomar alcohol eventualmente, salir de fiesta, así mismo, se puede incluir hacer deporte, y jugar videojuegos.
<lAssistanti>: aPor supuesto! Aquí tienes una clave dicotómica para clasificar los animales invertebrados:
E'Tinee un exoesqueleto duro articulado? -> Ir al paso 2
ENo liene exoesqueleto duro y articulado?
al paso
<iHumanl>: Las actividades que se supone que son comunes en una persona de 20 años varfan dependiendo de muchos factores, como la cultura, entorno, educación y los intereses personales. Sin embargo, aquí hay algunas actividades que pueden comunes para muchas personas de 20 años:
LTTinee patas articuladas, y segmentadas? Ess
un artrópodo
Noo tiene patas articuladas y segmentadas? >>
al paso
FTiene tentáculos una estructura de cuerpo
forma de campana? Es cnidario
NNo tiene tentáculos y una estructura de cuerpo
Estudiar o trabajar: Muchas personas de 20 años se encuentran una etapa de transición entre la educación y el mundo laboral, por lo que pueden estar estudiando trabajando sus primeras carreras empleos.
forma de campana? -> Ir al paso
FTiene un cuerpo blando y una concha
univalva o bivalva? -> Es un molusco
NNo tiene cuerpo blando y concha
univalva o bivalva? -> al paso
Establecer relaciones: Las personas de 20 años
TTiene una simetría radial estructuras como
menudo están explorando su vida amorosa y estableciendo relaciones intimas.
brazos o patas para moverse y capturar
alimentos? -> Es un equinodermo
Viajar: Muchas personas de 20 años aprovechan esta etapa para viajar explorar el mundo.
INo tiene una simetría radial estructuras como brazos patas para moverse y capturar
alimentos? -> Ir al paso 6
Desarrollar habilidades y pasatiempos: Las personas de 20 años pueden estar explorando
fTiene estructura de cuerpo segmentada y
sin patas articuladas? -> un anélido
nuevas habilidades pasatiempos, tratando
iNo tiene una estructura de cuerpo segmentada y sin patas articuladas? -> al paso
encontrar aquellos que son significativos para cllos.
LTiene un cuerpo alargado segmentado con tentáculos alrededor de la cabeza? Es
Participar actividades comunitarias: Las personas de 20 años a menudo se involucran en actividades comunitarias, como voluntariado, grupos de interés política.
gusano platelminto
iNo tiene un cuerpo alargado y segmentado
con tentáculos alrededor de la cabeza? Es un invertebrado no clasificado en las categorías anteriores.
<lAssistantis: Entiendo que me estabas poniendo prucba. Tomaré en cuenta actividades para
próximas respuestas.
31


Reviews:
[{"review_id": "WxZaMAYCAs", "reviewer": "Reviewer_ZhKe", "summary": "The authors introduce a data curation algorithm, DS^2, to correct inaccuracies in LLM-based data quality evaluation strategies.  The authors first show how, leveraging recent work in noisy label estimation given k-NN clusterability, both the Transition matrix and true probability distribution of LLM scores may be estimated by solving a linear program.  Under this k-NN clusterability assumption, the authors demonstrate that widely used LLM scorers (GPT-4o-mini, Llama-3.1 8B, and Mistral 7B) provide incorrect scores.  Using both Transition matrix and (true score) probability distribution, the layout the steps of their DS^2 algorithm and calculation of critical quantities: the error threshold for classifying samples as misrated, the confidence probability for mitigating imbalances in LLM scores, and the long-tail diversity score.\n\nThe authors then compare several data selection experiments leveraging the DS^2 algorithm over a large data pool (300k) containing several widely used datasets (i.e., Alpaca, Dolly, Flan V2, etc.).  Compared to several data selection heuristics and competitor LLM-based data selection algorithms (i.e., AlpaGasus and Deita), the authors show that their methods outperforms competitors per-LLM-judge and averaged over common natural language benchmarks (MMLU, TruthfulQA, GSM, BBH, and TydiQA) at a data selection size of 10k samples.  Representative experiments than compare the performance of DS^2 selecting 1k samples (over the 300k data pool) compared to the LIMA dataset (which was heavily vetted and selected using human feedback), once again outperforms finetuning LLama-3.1-8B and Mistral 7B models using the LIMA dataset.  Scaling law experiments as well as hybrid data selection schemes (e.g., AlpaGasus leveraging DS^2's data curation scores) further show the efficacy of the presented methods.\n\n# Post-rebuttal update\nThe authors have posted additional experiments to address my major concerns.  Particularly, an apples to apples comparison to AlpaGasus demonstrating DS^2's superiority and a discussion/experiments tackling the practical aspects of k-NN clusterability for this use case.  I am raising my score from 5 to 6.", "questions": "For the k-NN agreement score, assuming the k-NN clusterability characteristic, isn't the similarity score just always unity?  I.e., due to clusterability, the frequency will only occur in the same index as the one-hot encoding vector, and thus the numerator and denominator turn out to be the same.  Also, have the authors empirically explored whether using the embedding feature-vector or the one-hot encoding rated score vector work better in practice?\n\nOn the initial read through the paper, there was some initial confusion between: the avoidance of relying on the ground truth scores for DS^2, yet the appearance of the ground truth per-class probability in Section 4.  However, after careful rereading, we have the following: assuming k-NN score clusterability, we are thus able to leverage the concensus vectors and solve the LP for \\mathbf{T} and \\mathbf{p} (I understand this is stated, but it is easy to miss this simple statement buried in lines 231-241).  I would recommend either reiterating that solving the LP formed from Equation 1 provides both the transition matrix and ground truth probability vector, or breaking up the paragraph on lines 231-241 and emphasizing this.\n\n\"the average likelihood of identifying the sample n as misrated over multiple epochs.\" <- How is this calculated in practice?\n\nFor Completion Length, please cite the following:\nZhao, Hao, et al. \"Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning.\" arXiv preprint arXiv:2402.04833 (2024).\n\nFor Table 3, please specify the data pool is listed in Table 2, e.g., \" Performance comparison on OpenLLM leaderboard, using the data pool listed in Table 2.\"\n\nCould the authors please bold the winners of each column (per grouping) in Table 3?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The procedure to estimate the transition matrix from LLM scores is intuitive.  The beginning of the paper is also well written, and the examples showing the deficiencies in LLM-based scores is compelling.  The subsequent probability quantities leveraged throughout the DS^2 algorithm also largely make sense.  The large number of experiments demonstrating the effectiveness of the approach is also compelling.", "weaknesses": "Firstly, I would like to congratulate the authors on the DS^2 algorithm and the idea of leveraging k-NN clusterability to correct LLM-based scores.  Such a line of work seems promising.  With that said, I have several concerns that I am hoping can be addressed (I am greatly looking forward to the authors feedback).\n\nWhile the beginning of the paper is well written, there seemed to be either missing details or a lack clarity for important concepts.  In particular, Algorithm 1 is difficult to follow; \"CuratedScores\" is not defined in Algorithm 1. Are \"ConfidenceProbs\" the curated scores?  Furthermore:\n> the average likelihood of identifying the sample n as misrated over multiple epochs.\n\nHow is this calculated in practice?  Given a dataset with an arbitrary ordering of samples, how can a statistic dealing with the arbitrary index $n$ be meaningful?  Put another way, this calculates the statistics at the sample index, but the data pool samples (line 172) are not stated as being ordered in any specific way.  Thus, how can a statistic calculated over the sample index be anything other than arbitrary?\n\nFurthermore, while the notion of k-NN score clusterability (as defined in Definition 3.2) is assumed, this notion is never actually demonstrated for the LLM judges and datasets considered.  Do the score distributions depicted in FIgure 2 abide by k-NN score clusterability?  If not, could the authors speak to how this affects the Transition matrix and (true score) probability distribution calculations from the concensus vectors (Equation 1)?  If k-NN score clusterability is violated in practice, it does not seem possible to guarantee the correctness of estimating the Transition matrix and (true score) probability distribution in this case.\n\nThere were some concerns regarding the evaluations of AlpaGesus:\n> AlpaGasus (Chen et al., 2023) utilizes ChatGPT to rate\ndata samples and solely select high-rated samples\"\n\nOne of the major innovations of AlpaGasus was the prompt template.  Was this used for the AlpaGasus results in Table 3?  Note that they also showed the training data size is a very important hyperparameter for Alpaca; thus, a fairer/more apples-to-apples comparison would have been DS^2 applied to Alpaca with a budget of 9k samples and compared to fine-tuning performance (Table 3) of the AlpaGasus dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nEGwcddBz", "reviewer": "Reviewer_81Qt", "summary": "This study tackles the problem of rating samples using LLM, more specifically, how to improve the accuracy of LLM-based rating methods. To this end, DS2 (Diversity-Aware Score Curation for Data Selection) is proposed to model rating error pattens using a score transition matrix. Experiments on machine-alignment benchmarks show that strong results can be achieved by using just 3.3% of the original dataset.", "questions": "* It looks like that there is an advantage of using GPT-40 mini. How to obtain embeddings from proprietary LLMs?\n* Figure 1 reads like ratings of multiple LLMs are combined. The experimental results say otherwise.\n* The proposed approach sometimes perform much worse (e.g., in Table 2). Is it possible to decide the appropriate conditions for using the proposed approach (and which LLM)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The idea of detecting error pattern is novel. Its implementation using score transition matrix was validated experimentally.", "weaknesses": "* The description of \"deriving the score transition matrix\" can be improved. Not sure I can reproduce it based on the given explanation. Some examples can help, which can be included in appendix.\n* It is assumed that \"the transition matrix is independent of sample-level features\". This can be a limitation as each LLM have its own strengths and weakness (i.e., better/worse at some samples).\n* Sometimes, the improvement is small, within the variance of different runs.\n* Some discussions about limitations would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TEbzJnJ3iq", "reviewer": "Reviewer_XKFp", "summary": "The paper proposed the data selection pipeline named “DS^2”, which utilizes the kNN statistical information based on the rating scores to curate ratings, and then achieve better data selection. The authors validated their methods by conducting experiments with various rating models and benchmark tasks.", "questions": "I find the discussion around line 200, saying that similar embeddings should belong to the same category, not very convincing. Because here each class is the rating score, in my opinion, two samples could have similar textual embeddings but completely different correctness or quality, which lead to distinct rating scores. I think it is crucial to better explain this intuition behind the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper proposed a novel method to utilize the score transition matrix to detect scoring errors. Although the score transition matrix is proposed by previous works, the application of that into LLM data selection is novel.\n\nQuality: The paper provides detailed explanations of their methods and validates the results using thorough experiments, in which they include three models (GPT-4o-mini, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3) and various benchmarking tasks (MMLU, TruthfulQA, etc). Moreover, they also evaluated the alignment performance. All these greatly improve the quality of the paper.\n\nClarity: The paper demonstrated great clarity when exhibiting the error patterns of LLM scores, as well as when explaining the experiment setup and results.\n\nSignificance: The significance is validated with the experiment results and the ablation studies.", "weaknesses": "The paper does not include discussions about the efficiency of the proposed method and the efficiency comparison with other baseline methods. Part of the computation relies on estimating the score transition matrix, but the details about this estimation are lacking. Although the score transition matrix is proposed and discussed in previous works, it is still necessary to analyze their cost within the current framework.  Moreover, the paper does not include any baselines that also improve data quality by modifying the score ratings of the data. The current framework essentially proposed a method to modify the rating scores and then conduct data selection based on new scores. It would be helpful to further validate the method by comparing other methods that also study LLM ratings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkcCNABwrB", "reviewer": "Reviewer_gzew", "summary": "The paper presents DS2, a method for improving data efficiency in instruction tuning for LLMs. DS2 corrects inaccuracies and biases in LLM-generated data quality scores using a score transition matrix and promotes diversity in the data selection process. The authors demonstrate that using a curated subset  can outperform larger datasets, challenging traditional data scaling laws and achieving better results than human-curated datasets like LIMA. Key contributions include modeling score errors across various LLMs, developing a novel data curation pipeline, and conducting extensive experiments to validate DS2's effectiveness against multiple baselines.", "questions": "1. Can the authors provide more examples of typical errors in LLM-rated scores and explain how DS2 specifically addresses these? This would clarify the practical benefits of the score transition matrix.\n2. Could the authors discuss how the diversity score metric performs in datasets with different distributions or in tasks with varying levels of data complexity? Providing case studies or examples would help demonstrate the versatility of this metric.\n3. How sensitive is DS2's performance to the choice of embedding model used for the k-NN agreement score? Exploring different embeddings beyond the BGE model may provide insights into the robustness of the approach.\n4. In an open-ended question, the responses may not be similar at all (resulting in very different embedding vectors), but they could all be equally excellent answers. In this method, I believe the two responses may be curated into completely different scores, leading to errors. ”Intuitively, a lower cosine similarity score indicates a higher likelihood of a rating error”, This assumption does not adequately explain the examples above.\n5. \"please tell me the answer of below math question '1+1=?' answer:2\"\n\"please tell me the answer of below math question '1+1=?' answer:3\"\nThe two questions have similar embedding vectors, but they should have completely different scores. This seems to contradict the notion that samples with high embedding vector similarity are more likely to have the same labels.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper argue that a smaller, high-quality dataset can outperform larger datasets. The introduction of a score transition matrix to model LLM-based errors adds a novel approach to data curation, combining insights from label noise reduction and LLM evaluation.\n2. The experimental section is substantial and persuasive, featuring a rich number of comparative experiments that cover various scenarios and conditions, effectively demonstrating the robustness and efficacy of the proposed algorithm. Furthermore, the experimental results indicate significant performance improvements, further validating the superiority of the proposed approach.", "weaknesses": "1. While the score transition matrix is central to DS2, the paper lacks an in-depth analysis of its limitations or conditions under which it may not be effective. For instance, the independence assumption (that the transition matrix is not sample-specific) might limit its ability to model certain data-specific errors.\n2. The paper heavily relies on the capabilities of pre-trained LLMs to generate initial quality ratings and the transition matrix to correct the final quality score.\n3. If there is a strong LLM evaluator (eg, GPT-4o or o1), the score error correction process seems unnecessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)