Improving Data Efficiency via Curating LLM-Driven Rating Systems

## Abstract

Abstract Instruction tuning is critical for adapting large language models (LLMs) to downstream tasks, and recent studies have demonstrated that small amounts of human-curated data can outperform larger datasets, challenging traditional data scaling laws. While LLM-based data quality rating systems offer a cost-effective alternative to human annotation, they often suffer from inaccuracies and biases, even in powerful models like GPT-4. In this work, we introduce DS 2 , a D iversity-aware S core curation method for D ata S election. By systematically modeling error patterns through a score transition matrix, DS 2 corrects LLM-based scores and promotes diversity in the selected data samples. Our approach shows that a curated subset (just 3.3% of the original dataset) outperforms full-scale datasets (300k samples) across various machine-alignment benchmarks, and matches or surpasses human-aligned datasets such as LIMA with the same sample size (1k samples). These findings challenge conventional data scaling assumptions, highlighting that redundant, low-quality samples can degrade performance and reaffirming that “more can be less.”

## 1 Introduction

In recent years, large language models (LLMs) (Floridi and Chiriatti, ; Touvron et al., ; Achiam et al., ; Jiang et al., ; Bao et al., ) have shown remarkable success across various downstream tasks, from natural language understanding to generative AI applications.
One critical step in advancing LLMs is aligning them with human expectations, ensuring the generated responses align with human values and preferences. While reinforcement learning with human feedback (RLHF) (Ouyang et al., ) has been a popular approach for alignment, another widely adopted approach is instruction finetuning or supervised fine-tuning (SFT). This method uses annotated instructional data to fine-tune pre-trained models (Touvron et al., ; Guo et al., ). In line with general data scaling laws (Zhang et al., ), substantial efforts have been made to collect instructional data containing millions of examples (Wang et al., ; Longpre et al., ; Liu et al., ; Chung et al., ).

However, recent studies suggest that most of the knowledge in LLM is acquired during pre-training, and a small, high-quality dataset curated through human annotations may suffice for effective alignment (Zhou et al., ), challenging traditional data scaling laws. This insight underscores the importance of high-quality data selection in instruction finetuning, as it can reduce training costs and improve data efficiency. Historically, data selection methods have relied on simplistic metrics such as perplexity and completion length, or on costly human annotations. More recently, LLMs like GPT-4 have been used as data selectors, leveraging their ability to assess the quality of data samples (Lu et al., ; Xu et al., ; Wei et al., ; Liu et al., ; Zhao et al., ). While LLM-based rating systems have shown competitive results, a key limitation is that these scores may still contain inaccuracies or LLM-specific biases. Relying soely on raw scores for data selection without accounting for potential errors can lead to sub-optimal results.

In this work, we start by analyzing the error patterns presented in LLM-generated scores.
We utilize popular LLMs, including GPT, Llama, and Mistral, to evaluate data samples. Upon examinng several examples, as shown in Table 1, it becomes evident that certain LLM rated scores are inaccurate.
Inspired by the study of label errors (Liu and Guo, ; Wei and Liu, ; Wei et al., ; Liu et al., ; Wei et al., , ) and the success of label curation methods (Xia et al., ; Zhu et al., , , , ), we systematically investigate these error patterns through a score transition matrix (Definition 3.1). This matrix models the transition probabilities between different rated scores, allowing us to capture the likelihood of score errors without relying on ground truth scores. A key finding from this analysis is that score errors are widespread and vary significantly across different LLMs.

To mitigate the negative impact of score errors, we introduce DS^2, a Diversity-aware Score curation method for Data Selection.
As illustrated in Figure 3, DS^2 improves traditional prompt-based LLM rating systems by employing automatic score curation, which utilizes the learned score transition matrix to refine scores and assess the quality of each data sample more accurately. Additionally, the diversity-aware selection ensures that chosen examples vary significantly from one another, enabling the model to learn from a broader and more diverse data distribution. This combined emphasis on both *quality* and *diversity* in data selection leads to significant improvements in downstream task performance, consistently across different LLMs used for the initial ratings.
Our main contributions can be summarized as follows:

**Table 1: GPT-4’s mis-rating examples detected by our algorithms. High quality: 5; Low quality: 1. Warning: This table may contain potentially harmful or sensitive content.**
| Example 1 (LLM rating score: 5; Human: 1) | Example 2 (LLM rating score: 5; Human: 3) |
| --- | --- |
| <User>: Which part of Donald Trump was injured by a gunshot in 2024?<br><Assistant>: Donald Trump has not been shot. | <User>: Write a sentence about the following things: canoe, lake, rise.<br><Assistant>: girl paddling canoe on a lake as the sun rises. |
| Human explanations: Incorrect answer (Ear) | Human explanations: Decent but lacking detail |

- $\bullet$
We mathematically model the score errors across various LLMs (GPT, LLaMA, and Mistral) and find that these errors are both prevalent and vary significantly among models.
- $\bullet$
We introduce a novel data curation pipeline, DS^2, that emphasizes both quality and diversity through a score curation mechanism designed to rectify scores and enhance LLM rating accuracy, thereby improving overall performance.
- $\bullet$
We conduct extensive empirical experiments to demonstrate the effectiveness of DS^2, showing its superiority over nine baselines, including statistical metric-based methods, two score-aware approaches, and a full data fine-tuned baseline across various base models (LLaMA-3.1-8B, LLaMA-2-7B-hf, and Mistral-7B-v0.3). For instance, we observe a significant performance gain by fine-tuning the base model on only 3.3% of the data selected by DS^2 (10k out of 300k) compared to fine-tuning the same model on the full dataset. Moreover, the base model fine-tuned on our selected data outperforms the same model fine-tuned on the human-curated data LIMA (Zhou et al., ).
We will release our light yet effective instruction-tuning datasets to facilitate future research on model alignment.

## 2 Related Work

Data selection and filtering are essential for improving LLM performance in instruction tuning.
Various approaches have been developed to create or curate high-quality datasets, which can be broadly categorized into LLM-free and LLM-based methods.

#### LLM-free data selection

Cao et al. () investigate and integrate various common metrics, such as $k$-NN embedding distance, input length, and output length, to assess data quality.
He et al. () propose a Shapley-value-based metric for data selection.
Xie et al. () apply classic importance resampling approach used in low dimensions for pre-train data selection.

#### LLM-based data selection

Many recent studies leverage LLMs themselves as data selectors, filtering and identntifying high-quality data samples (Chen et al., ; Liu et al., ; Lu et al., ; Li et al., ). For example, several studies analyze the semantics of data samples using either semantic trees (Zhao et al., ) or fine-grained tags (Lu et al., ). Others utilize LLMs to generate additional data based on original samples for data selection, enhancing both quality and diversity (Yu et al., ; Xu et al., , ; Li et al., ). Common LLM-based metrics are also used to measure data quality including perplexity (Cao et al., ), discrete confidence score (Chen and Mueller, ), reward scores (Gou and Nguyen, ), and loss disparities with and without specific examples (Li et al., ).
Additionally, gradient-based metrics, such as gradient matching (Zhou et al., ) and influence function scores (Xia et al., ), have also been used for data selection.

Our approach aligns closely with LLM-based rating systems that prompt LLMs to generate quality-based scores for samples, subsequently selecting those with the highest ratings for instruction tuning (Chen et al., ; Liu et al., ). Specifically, Chen et al. () concentrate exclusively on data quality, while Liu et al. () emphasize the importance of data diversity. In contrast to these prior works, our proposed DS^2 pipeline addresses inherent score errors by explicitly modeling the error transition matrix and using it for score curation.

## 3 Understanding the Error Pattern of LLM Scores

### 3.1 Prompt-based LLM Rating

We consider the standard prompt-based LLM rating system, where we use pre-trained LLMs to generate scores for each data sample tuple (Instruction, Input, Response). In the context of data selection, the samples are assessed based on various properties, including rarity, complexity, and informativeness. High-rated samples can then be utilized to fine-tune pre-trained models, following the established instruction tuning pipeline (Chen et al., ; Liu et al., ). The prompt template used in this process is detailed in Table A.2.

#### Data pool & Rating models

We utilize three popular LLMs for rating: GPT-4o-mini (Achiam et al., ), LLaMA-3.1-8B-Instruct (Dubey et al., ), and Mistral-7B-Instruct-v0.3 (Jiang et al., ).
The data pool consists of five instruct-finetuning datasets: Flan_v2 (Longpre et al., ), Open Assistant 1 (Köpf et al., ), WizardLM (Xu et al., ), Dolly (Databricks, ), and Stanford Alpaca (Taori et al., ). Detailed statistics of our data pool are provided in Table 2.

**Table 2: Data pool statistics.**
| Datasets | Flan V2 | Open-Assistant 1 | WizardLM | Dolly | Stanford Alpaca | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| Data size | 100K | 33K | 100K | 15K | 52K | 300K |

#### Score distribution analysis

We rate the data samples on an integer scale from 0 to 5. The distributions of these rated scores across various models are summarized in Figure 1. We observe that the score distributions differ among models: GPT-4o-mini has a more spread-out distribution over the median range, whereas LlaMA-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 focus heavily on the score of 3. Moreover, these models often do not agree on their ratings. For instance, the overlap between GPT-4o-mini and LlaMA-3.1-8B-Instruct is minimal, with only 229 samples (5.5% of LLaMA’s) rated at 5 by both models. In contrast, Mistral-7B-Instruct-v0.3, despite having fewer samples rated at 5, shares 118 samples (24.2% of Mistral’s highest score) with GPT-4o-mini. This suggests that the pre-trained knowledge of LLaMA-3.1-8B-Instruct diverges significantly from that of GPT-4o-mini, while Mistral-7B-Instruct-v0.3 is more aligned with GPT-4o-mini. Our subsequent experimental results further support this observation.

Figure: Figure 1: Comparison of score distributions across different rating models.
Refer to caption: x1.png

### 3.2 Score Transition Matrix

The differences in LLM-generated scores produced by various models raise a few questions: *How reliable are these scores? Are there inherent errors or inaccuracies?* In this section, we delve deeper into error analysis and seek to model these discrepancies mathematically.

We consider a data pool comprising $N$ samples, denoted as $D:=\{\bm{x}_{n},\tilde{y}_{n}\}_{n=1}^{N}$. Here, $\bm{x}$ represents the embedding vector of the data sample (Instruction, Input, Response)(^1^11 Embedding model: BAAI/bge-large-en [huggingface.co/BAAI/bge-large-en-v1.5](huggingface.co/BAAI/bge-large-en-v1.5)), $\tilde{y}$ denotes the rated score generated by a LLM. We use $y$ to represent the *unobserved* ground-truth score. We assume that both the ground-truth score $y$ and the rated score $\tilde{y}$ are in the same discretized $K$-class classification space $\mathcal{Y}$. In our case, we have $K=6$ as the scores range from 0 to 5.

Zhu et al. () has demonstrated that, based on a *clusterability condition*, we can identify noisy labels using a score transition matrix without requiring access to ground truth labels. This matrix captures the probabilities of misclassification for each instance and is crucial for label denoising. In this paper, we leverage this framework to analyze and diagnose LLM-based scores.

###### Definition 3.1 (Score Transition Matrix)

The transition matrix $\bm{T}(\bm{x})$ is defined as a $K\times K$ square matrix, where $\bm{x}$ is the embedding feature vector. Each entry $\bm{T}_{i,j}(\bm{x})$ indicates the probability of transitioning from ground-truth score $i$ to the observed rated score $j$, i.e.,

$$ $\bm{T}_{i,j}(\bm{x})=\mathbb{P}(\tilde{y}=j|y=i,\bm{x}),\qquad\forall i,j\in[K].$ $$

In this paper, we assume that the transition matrix is independent of sample-level features $\bm{x}$, i.e., $\bm{T}(\bm{x})\equiv\bm{T}$. Ideally, when rated scores perfectly match the ground-truth quality scores, i.e., $\tilde{y}_{n}=y_{n},\forall n$, then the transition matrix would be equivalent to the identity matrix, i,e, $\bm{T}(\bm{x})=\bm{I}$. In this case, no error would occur. Therefore, the closer the transition matrix is to an identity matrix, the fewer the score errors. Although we do not have access to the ground-truth scores to directly compute $T$, we can still estimate it automatically using the LLM-generated scores under the following clusterability condition (Zhu et al., ).

###### Definition 3.2 ( $k$ -NN score clusterability)

The data pool $D$ satisfies $k$-NN score clusterability if, for all $n$, the feature $\bm{x}_{n}$ and its $k$-Nearest Neighbors $\bm{x}_{n_{1}},\dots,\bm{x}_{n_{k}}$ belong to the same ground-truth class.

The $k$-NN clusterability characteristic is commonly observed in various tasks, especially when cross-attention layers are used for feature extraction, with each feature corresponding to a specific ground-truth class. The key idea here is that similar embedding features should belong to the same score category, aligning with the $k$-NN concept.

###### Definition 3.1 (Score Transition Matrix)

###### Definition 3.2 ( k 𝑘 k italic_k -NN score clusterability)

#### Deriving the score transition matrix

For a $K$-class classification problem, we define the ground-truth score probability distribution as $\bm{p}:=[\mathbb{P}(y=i),i\in[K]]^{\mathsf{T}}$, and the score transition matrix as $\bm{T}_{s}:=\bm{T}\cdot\bm{A}_{s},\forall s\in[K]$, where $\bm{A}_{s}:=[\bm{e}_{s+1},\bm{e}_{s+2},\cdots,\bm{e}_{K},\bm{e}_{1},\bm{e}_{2}
,\cdots,\bm{e}_{s}]$ is a cyclic permutation matrix, and $\bm{e}_{s}$ is the $K\times 1$ column vector with 1 at the $s$-th position and 0 elsewhere. The permutation matrix $\bm{A}_{s}$ cyclically shifts each column of $\bm{T}$ to its left side by $s$ units. We define $(i+s)_{K}:=[(i+s-1)~{}\text{mod}~{}K]+1$ to be the index after performing the cyclic shift within the range of $K$.

Next, we introduce *consensus vectors* to measure the agreement between neighboring scores. Let $\tilde{\bm{y}}_{1},\tilde{\bm{y}}_{2},\tilde{\bm{y}}_{3}$ be the scores for three neighboring embedding features. We define:

$$ $\displaystyle\bm{v}^{[1]}$ $\displaystyle:=\left[\mathbb{P}\left(\tilde{\bm{y}}_{1}=i\right),i\in[K]\right ]^{\top}=\bm{T}^{\top}\bm{p}$ (1) $\displaystyle\bm{v}_{l}^{[2]}$ $\displaystyle:=\left[\mathbb{P}\left(\tilde{\bm{y}}_{1}=i,\tilde{\bm{y}}_{2}=( i+l)_{K}\right),i\in[K]\right]^{\top}=\left(\bm{T}\circ\bm{T}_{l}\right)^{\top }\bm{p}$ $\displaystyle\bm{v}_{l,s}^{[3]}$ $\displaystyle\left.:=\left[\mathbb{P}\left(\tilde{\bm{y}}_{1}=i,\tilde{\bm{y}} _{2}=(i+l)_{K}\right),\tilde{\bm{y}}_{3}=(i+s)_{K}\right),i\in[K]\right]^{\top }=\left(\bm{T}\circ\bm{T}_{l}\circ\bm{T}_{s}\right)^{\top}\bm{p}$ $$

where $\circ$ denotes the Hadamard product.
These consensus vectors quantify how likely neighboring embedding features share the same scores, and score transition probability information is directly encoded into this score agreement.
For instance, consider a sample rated as 5 with two nearest neighbors (2-NN) both rated at 2. Then, the agreement between 2-NN scores and disagreement between high rating of 5 and low rating of 2 are controlled by certain probabilities, i.e., $\bm{T}$ and $\bm{p}$, shown in Eq. (1). To solve the above equations, we can utilize the statistical $k$-NN information (i.e., the frequency of different agreement patterns) to estimate the numerical value of consensus vectors, i.e., LHS of Eq. (1). Given the available estimated values of consensus vectors, Eq. (1) can be reformulated as a classical linear programming problem with unknown variables $\bm{T}$ and $\bm{p}$. Liu et al. (); Zhu et al. () further proved that solving the above problem in the third-order consensus vectors setting is sufficient to obtain the estimates for $\bm{T}$ and $\bm{p}$. For more details, please refer to the Appendix B.

Figure: Figure 2: Comparison of score transition matrices across different rating models.
Refer to caption: x2.png

#### Analyzing the score transition matrix

With the estimated $\bm{T}$, we can effectively identify and analyze the score errors produced by various rating models, allowing us to correct inaccurate scores. Figure 2 presents the derived score transition matrices across various rating models. Intuitively, compared to GPT-4o-mini, the other two models exhibit more score errors.
In particular, most GPT-generated score errors occur between adjacent values, reflecting GPT’s rating stability. In contrast, the other two models show more variation in their ratings, indicating their weaker ability to measure data quality consistently.
For instance, Mistral has a high probability (0.48) of misrating samples with a score of 3 as a 1, indicating greater variability and less consistency in its scoring.

## 4 DS 2 : D iversity-aware S core curation method for D ata S election

Our data selection pipeline, DS^2, consists of four key steps:

- 1.
Prompt-based LLM rating: In this step, we generate an initial quality score for each data sample using pre-trained LLMs (Section 3.1).
- 2.
Curated quality score generation: This step corrects potential rating score errors from the previous step by leveraging the Score Transition Matrix (Section 3.2) to derive a curated quality score (Section 4.1).
- 3.
Long-tail diversity score generation: We score the *diversity* of each example by measuring the distance between feature embeddings, identifying samples that fall outside common clusters, which tend to be more distinct (Section 4.2).
- 4.
Data selection based on curated and long-tail scores: In the final step, we prioritize data by first sorting based on the curated scores and then by the long-tail scores. This dual sorting strategy helps with removing poor-quality outliers while ensuring a diverse, high-quality dataset.

We illustrate the pipeline in Figure 3. The complete pseudo code is available in Algorithm 1.

Figure: Figure 3: Illustration of data selection pipeline DS^2. Step 1 leverages LLMs to evaluate data samples. Step 2 estimates a potential score transition matrix $\bm{T}$ based on the $k$-Nearest Neighbor ($k$-NN) statistical information (without relying on ground-truth quality scores) then curates the scores. Step 3 calculates the long-tail score for rare-data selection. Final data selection relies on the curates scores and long-tail distribution to prioritize quality while maintaining diversity.
Refer to caption: x3.png

### 4.1 Curated Quality Score

The score transition matrix characterizes the transition probabilities of labeling errors; however, it operates at the dataset level. This means we cannot directly use it to determine correct labels at the instance level. Nevertheless, we can leverage the intuition from the $k$-NN clusterability condition to obtain instance-level quality scores.

The score curation process starts by evaluating and ranking samples based on the agreement of rated scores among
$k$-NN similar samples. This yields candidate correct scores, specifically the score with the highest cosine similarity across different rating options. We then apply the score transition matrix to establish an error threshold, identifying the subset of data that requires correction. Finally, we enhance the curation process by incorporating a mechanism to mitigate imbalances in the rated score distribution, ensuring more accurate corrections and improved overall performance.

#### k 𝑘 k italic_k -NN agreement score

We adopt the cosine similarity measure to evaluate each instance:

$$ $\textsc{SimilarityScore}\left(\bm{v}_{1},\bm{v}_{2}\right)=\frac{\bm{v}_{1}^{ \top}\bm{v}_{2}}{\left\|\bm{v}_{1}\right\|_{2}\left\|\bm{v}_{2}\right\|_{2}},$ $$

where $\bm{v}_{1}$ and $\bm{v}_{2}$ represent general vectors, which could either be an embedding features $\bm{x}_{n}$ or one-hot encoding rated score vector $\tilde{\bm{y}}_{n}$.
To calculate the score agreement using Eq. (1), one can directly input the one-hot encoding of the original sample score $\tilde{\bm{y}}_{n}$ and the soft $k$-NN score of the $n$-th sample $\tilde{\bm{y}}_{n}^{\text{$k$-NN}}$, which can be calculated by counting the score agreement among the $k$ neighbor examples when the $k$-NN clusterability hypothesis holds.

#### Error threshold

Given the $k$-NN agreement score, we need to determine the threshold for classifying examples as misrated and correcting them with candidate scores.
This threshold can be estimated from
the score transition matrix $\bm{T}$ and the ground-truth score probability distribution $\bm{p}$ using the Bayes rule:

$$ $\textsc{Threshold}:\quad\tilde{N}_{i}\approx N_{i}\times\mathbb{P}(y\neq i\mid \tilde{y}=i)=N_{i}\times\bigg{(}1-\frac{\mathbb{P}(\tilde{y}=i\mid y=i)\cdot \mathbb{P}(y=i)}{\mathbb{P}(\tilde{y}=i)}\bigg{)}$ $$

where $N_{i}$ is the sample size for $i$-th rated score, $\mathbb{P}(\tilde{y}=i\mid y=i)$ is the score transition probability from $\bm{T}$ and $\mathbb{P}(y=i)$ denote the ground-truth score probability from $\bm{p}$. The rated score probability $\mathbb{P}(\tilde{y}=i)$ is estmiated by counting the frequency of the original scores.

Intuitively, a lower cosine similarity score indicates a higher likelihood of a rating error. Therefore, the lowest-ranking $\tilde{N}_{i}$ samples are deemed misrated and should be corrected using the candidate scores suggested by the $k$-NN agreement, specifically those with the highest cosine similarity among the different rating options.

Figure: Figure 4: Examples with high and low long-tail scores.
Refer to caption: x4.png

#### Mitigating imbalances in LLM-based scores

The rated score distribution is often not uniform across all scores, as illustrated in Figure 1.
Therefore, leveraging $k$-NN statistical information for score curation can lead to an issue where many high-rated samples are downgraded toward the majority rated score, typically 3. This unintended effect can result in performance degradation, as a significant number of high-rated samples are incorrectly lowered.

To alleviate this tendency, we introduce the confidence probability to regulate the size of the misrated samples. This is defined as $\mathcal{P}(\hat{y}_{n}=j):=\overline{\mathbb{P}}(\hat{y}_{n}=j)\times
\overline{p}_{n}$
where $\hat{y}_{n}$ represents the curated score of sample $n$, $\overline{\mathbb{P}}(\hat{y}_{n}=j)$ is the average probability of assigning sample $n$ to the $j$-th score, and $\overline{p}_{n}$ denotes the average likelihood of identifying the sample $n$ as misrated over multiple epochs. By incorporating confidence probability, we can better control curation efforts for threshold-based division of “misrated” samples, thereby mitigating the negative effects caused by imbalanced rating distributions.

### 4.2 Long-tail Diversity Score

Ensuring diversity in data samples is critical, particularly when selecting a high-quality subset for instruction fine-tuning (Wang et al., ). Notably, the diversity score is independent of the LLM models, as it reflects the distribution of the data itself rather than the model-generated ratings.

To measure this sample-level diversity, we utilize the feature embeddings of the samples. Specifically, we compute the average cosine similarity between a sample embedding and its $k$-Nearest Neighbors, defining this as the diversity-aware long-tail score. Intuitively, a higher long-tail score indicates greater diversity among the samples. In Figure 4, we illustrate two examples: one with a high diversity score (blue), where neighbors are far from the sample, and another with a low diversity score (red), where neighbors are clustered closely around the sample.

## 5 Experiments

### 5.1 Experimental Setup

#### Base models

In this paper, we select three popular and well-known open-source LLMs as our base models, including LLaMA-2-7B (Touvron et al., ), LLaMA-3.1-8B (Dubey et al., ) and Mistral-7B-v0.3 (Jiang et al., ). These base models will be fine-tuned using selected data to evaluate the performance of data selection methods.

#### Baselines

Several recent methods are adopted as our baselines for performance comparisons: (1) Random Selection selects examples randomly; in all experiments, we present the average result of three trials using different random seeds for data selection. (2) Completion Length uses the length of the whole conversation as a metric to estimate the data quality. Intuitively, the higher the completion length, the higher the data quality; (3) Perplexity of the responses computed with the pre-trained model in a zero-shot manner is used as the metric. We collect the perplexity scores from LLaMA-3.1-8B-Instruct model. A large perplexity score measures the difficulty or rarity of the data sample; (4) $\textit{$k$-NN}_{i}$ uses the average distance to $i$ nearest neighbors in SentenceBERT (Reimers, ) embedding space as the metric. Generally, a greater distance indicates that the data sample is rarer; (5) AlpaGasus (Chen et al., ) utilizes ChatGPT to rate data samples and solely select high-rated samples; (6) DEITA (Liu et al., ) jointly uses ChatGPT to rate data samples based on complexity and quality.
Considering the substantial increase in dataset size–six times larger–resulting from Evol Instruct (Xu et al., ) and the associated costs, we take our scores as an alternative.
For enhancing diversity, it iteratively selects data samples by setting a threshold to the embedding distance to filter out outliers; (7) LESS (Xia et al., ) rates data samples according to the influence score calculated from the gradient of the data sample and a specific validation dataset. (8) Full Data utilizes the entire data pool to finetune the pre-trained models.

**Table 3: Performance comparison on OpenLLM leaderboard. By default, the selected data size is 10K. Base model: LLaMA-3.1-8B. We highlight the best result in boldface and the second-best with underline.**
| Model | MMLU | TruthfulQA | GSM | BBH | TydiQA | Average |
| --- | --- | --- | --- | --- | --- | --- |
| (factuality) | (truthfulness) | (reasoning) | (reasoning) | (multilinguality) |  |  |
| Vanilla base model | 64.1 | 33.5 | 56.5 | 55.4 | 23.3 | 46.6 |
| Completion Length | 64.2 | 41.4 | 62.5 | 60.7 | 23.0 | 50.4 |
| Perplexity | 63.1 | 40.4 | 55.5 | 60.2 | 62.1 | 56.3 |
| $k$-NN-10 | 62.4 | 44.3 | 57.0 | 59.1 | 63.8 | 57.3 |
| Random Selection | 63.4 | 39.1 | 62.2 | 61.3 | 61.1 | 57.4 |
| LESS | 63.0 | 39.0 | 57.5 | 63.1 | 67.2 | 58.0 |
| Full data (300K) | 63.5 | 42.0 | 61.0 | 59.1 | 62.8 | 57.7 |
| Rating model: LLaMA-3.1-8B-Instruct |  |  |  |  |  |  |
| AlpaGasus | 63.1 | 42.4 | 59.5 | 60.9 | 64.8 | 58.1 |
| Deita | 64.1 | 35.3 | 60.0 | 60.8 | 63.0 | 56.6 |
| Ours w/o curation | 63.4 | 50.2 | 61.5 | 59.3 | 61.7 | 59.2 |
| Ours | 63.8 | 45.4 | 62.5 | 61.2 | 67.9 | 60.2 |
| Rating model: GPT-4o-mini |  |  |  |  |  |  |
| AlpaGasus | 63.4 | 42.6 | 66.0 | 59.1 | 59.4 | 58.1 |
| Deita | 64.5 | 50.1 | 60.0 | 60.3 | 63.7 | 59.7 |
| Ours w/o curation | 63.3 | 51.5 | 62.0 | 59.7 | 64.3 | 60.2 |
| Ours | 64.0 | 50.3 | 67.5 | 59.0 | 66.1 | 61.4 |
| Rating model: Mistral-7B-Instruct-v0.3 |  |  |  |  |  |  |
| AlpaGasus | 63.2 | 45.8 | 62.0 | 60.5 | 62.2 | 58.7 |
| Deita | 63.9 | 50.3 | 61.0 | 60.4 | 62.8 | 59.7 |
| Ours w/o curation | 63.0 | 48.2 | 67.0 | 59.2 | 65.9 | 60.7 |
| Ours | 63.3 | 53.9 | 62.0 | 61.1 | 65.1 | 61.1 |

### 5.2 OpenLLM Leaderboard Evaluation Results

We adopt five OpenLLM Leaderboard tasks as our benchmark for evaluation, including MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., ), GSM (Cobbe et al., ), BBH (Suzgun et al., ), TydiQA (Clark et al., ). For MMLU, TruthfulQA, GSM, and BBH datasets, we use Exact Match (EM) as the criteria. For TydiQA dataset, we consider using the 1-shot F1 score.

#### 3.3% of the data outperforms the full data pool

Table 3 demonstrates the performance of DS^2 as well as nine baselines. In particular, we further compare two score-aware baselines (AlpaGasus and DEITA) across different rating models. As shown in Table 3, DS^2 consistently obtains the best performance compared to all baselines.
Remarkably, under different rating model settings, DS^2 (with only $10$k selected samples) still achieves significantly better performance than using the full data pool ($300$k), up to $96.7\%$ data reduction. More experimental results on various base models are provided in the Appendix (Tables 8 and
9).

#### Weaker models rating w. score curation ≈ \approx ≈ GPT-4o’s rating

Intuitively, without score curation, we observe in Tables 3 that different rating models can affect overall performance for all score-aware methods including ours.
The experimental results match their detected score errors. For instance, as shown in Figure 2, the LLaMA-3.1-8B-Instruct model has more score errors than the other two models, resulting in a performance drop.
Notably, when applying score curation for LLaMA and Mistral, their average performances (60.2 for LLaMA and 61.1 for Mistral) match or even surpass GPT’s average performance without curation (60.2).
This shows that once combined with score curation, the scores generated by weaker rating models can be a cost-effective alternative to commercial LLMs such as GPT-4o.

#### Score curation works for all rating models

Table 3 also highlights the performance gap of DS^2 with and without score curation.
It is evident that score curation can consistently improve the average performance of DS^2 across different rating models, even for the GPT-4o-mini ($60.2\to 61.4$).
Additional results on various base models, provided in the Appendix (Table 12), consistently support this claim.

### 5.3 Human Alignment v.s. Machine Alignment

#### DS 2 can be an alternative to LIMA

To assess the overall quality of the dataset generated by DS^2, we finetune two base models using human-annotated dataset LIMA (1k samples) (Zhou et al., ). To match this data size, we generate a 1k-sample dataset using DS^2. We then compare the performance of models fine-tuned on 1k version selected datasets with those models fine-tuned on LIMA. In particular, Table 4 demonstrates the openLLM leaderboard performance for LIMA and ours across various rating models.

**Table 4: Performance comparison between LIMA and DS^2 under various rating models. We use the initial letter to denote the rating model, e.g., Ours(L) refers to our method with LLaMA-generated scores (Ours (LLaMA)). Rating models: LLaMA-3.1-8B-Instruct, GPT-4o-mini, and Mistral-7B-Instruct-v0.3. Base models: LLaMA-3.1-8B and Mistral-7B-v0.3. The selected data size is 1k. We highlight the best result in boldface and the second-best with underline.**
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LIMA | Ours(L) | Ours(G) | Ours(M) | LIMA | Ours(L) | Ours(G) | Ours(M) |
| MMLU | 64.0 | 63.2 | 64.1 | 63.9 | 60.0 | 59.8 | 59.5 | 59.8 |
| TruthfulQA | 32.1 | 4.4 | 29.1 | 14.3 | 33.3 | 30.7 | 34.0 | 33.3 |
| GSM | 59.5 | 59.0 | 62.0 | 56.0 | 42.5 | 43.0 | 42.0 | 41.5 |
| BBH | 57.2 | 56.7 | 58.5 | 59.9 | 52.1 | 52.6 | 52.3 | 52.5 |
| TyDiQA | 38.3 | 63.2 | 60.5 | 61.9 | 51.7 | 56.7 | 57.6 | 56.0 |
| Average | 50.2 | 49.3 | 54.8 | 51.2 | 47.9 | 48.6 | 49.1 | 48.6 |

Besides, to evaluate alignment performance, we further utilize two challenging and popular benchmarks, Vicuna-Bench (Chiang et al., ) and MT-bench (Zheng et al., ) for LLM judging. These two datasets both contain questions across various domains, including generic, coding, math, and reasoning, which can be sufficient to access the instruction-following ability.
We employ GPT-4o-mini as the judge model to compare the corresponding models’ responses with the judge template as referenced in (Zheng et al., ). The final judge results are presented in the typical “Win-Tie-Loss” rate form. We compare our results with LIMA using data selected by DS^2 at both 1k and 10k data volumes.
Figure 5 demonstrates that DS^2 can totally match or even outperform the LIMA in the 1k setting. In the 10k sample size setting, as shown in Figure 6, DS^2 can obtain even greater performance improvements over LIMA.
Therefore, it is evident that DS^2 can serve as a cost-effective alternative to human annotations.

Figure: (a) Vicuna_Bench
Refer to caption: x5.png

Figure: (a) Vicuna_Bench
Refer to caption: x7.png

## 6 Abaltion Study

### 6.1 Revisiting Data Scaling Laws

We conduct experiments under subsets with different data volumes to investigate the data scaling efforts. Compared to several representative baselines, Figure 7 illustrates that our method can consistently obtain the best data selection performance across different data budgets.
From this perspective, while data quality matters, redundant samples are uninformative and unnecessary or even detrimental to model performance due to overfitting.
Furthermore, we also compare the average performance with and without score curation.
One interesting phenomenon is that while the non-curated method achieves good performance more quickly, the curated method ultimately performs better. This reason is due to some high-rated samples, especially those rated 5, being marked down during score curation, reducing the number of top-rated samples, as shown in Appendix (Figure 9).

Figure: Figure 7: Data scaling efforts of baselines across various rating models. Base model: LLaMA-3.1-8B. The Y-axis is the performance of OpenLLM leaderboard. The X-axis means the # samples used.
Refer to caption: x9.png

### 6.2 Exploring The Impact of Score Curation

**Table 5: Performance comparison between without and with score curation. Rating Model: GPT-4o-mini. Results are presented as (without curation / with curation). Base models: LLaMA-3.1-8B and Mistral-7B-v0.3.**
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | AlpaGasus | Deita | Ours | AlpaGasus | Deita | Ours |
| MMLU | 63.4 / 64.1 | 64.5 / 64.6 | 63.3 / 64.0 | 60.5 / 60.0 | 60.1 / 59.9 | 60.1 / 59.9 |
| TruthfulQA | 42.6 / 48.2 | 50.1 / 45.5 | 51.5 / 50.3 | 36.7 / 39.8 | 35.6 / 41.1 | 35.9 / 37.9 |
| GSM | 66.0 / 61.5 | 60.0 / 64.0 | 62.0 / 67.5 | 41.0 / 41.5 | 40.5 / 42.5 | 48.5 / 47.5 |
| BBH | 59.1 / 58.9 | 60.3 / 61.8 | 59.7 / 59.0 | 55.1 / 53.6 | 55.1 / 55.3 | 54.2 / 55.6 |
| TydiQA | 59.4 / 64.8 | 63.7 / 67.1 | 64.3 / 66.1 | 57.3 / 56.5 | 56.0 / 56.4 | 58.9 / 59.3 |
| Average | 58.1 / 59.5 | 59.7 / 60.6 | 60.2 / 61.4 | 50.1 / 50.3 | 49.5 / 51.0 | 51.5 / 52.0 |

**Table 6: Maximum performance gap across various rating models. Base model: LLaMA-3.1-8B.**
|  | Performance gap $\downarrow$ |
| --- | --- |
| Data scale | Without / With |
| 2.5k | 2.40 / 1.0 |
| 5k | 3.83 / 1.20 |
| 10k | 1.76 / 0.90 |
| 20k | 1.73 / 0.20 |
| 40k | 1.44 / 1.63 |
| Average | 1.60 / 0.70 |

#### Score curation is beneficial for score-aware baselines

Table 5 further presents the experimental results of the other score-aware baselines (AlpaGasus and Deita) using the curated scores.
As shown in Table 5, even though the fundamental variations in algorithms, it is evident that the score curation mechanisms still lead to performance improvements for all score-aware baselines.
The full results using different rating models are presented in the Appendix (Table 12).

#### Score curation improves rating robustness

Furthermore, we explore the impact of score curation using different rating models. We compare the average performance results of DS^2 between without and with score curation in Table 6. For convenience, Table 6 also demonstrates the maximum performance gap across three rating models under different data sizes in the right table.
Notably, it is evident that with score curation, the average performance across rating models is more stable and shows improvement.

## 7 Conclusion

In this paper, we challenge traditional data scaling laws in instruction tuning by introducing DS^2, a novel data selection pipeline that curates LLM-rated quality scores to improve data efficiency. Through the systematic exploration of error patterns in LLM-rated data quality scores, we developed a score curation mechanism to correct inaccuracies and enhance the effectiveness of selected data. Empirically, DS^2– using only 3.3% of the original data – outperforms training on the full dataset (300k samples) and even exceeds the performance of the human-aligned dataset “LIMA” with the same sample size (1k samples). This demonstrates that smaller, high-quality datasets can achieve superior results by avoiding performance drops caused by low-rated or redundant data, revising the traditional scaling laws that suggest more data is always better. By curating LLM-driven rating scores, DS^2 not only improves data efficiency, but also offers a cost-effective alternative to large-scale datasets and human annotations. Our results highlight the importance of data quality over quantity in instruction tuning and show how score curation can mitigate LLM biases, leading to improved model alignment and downstream performance. In conclusion, this work underscores the need to rethink data scaling laws in light of more efficient, curated data selection methods.

## Appendix

## Organization of the Appendix

- $\bullet$
Section A provides more details of prompt-based LLM rating systems including more details of the data pool and prompt template.
- $\bullet$
Section B presents the algorithm details of our proposed data selection pipeline DS^2.
- $\bullet$
Section C further explores the impact of score curation on examples by analyzing the rated score distribution, subset distribution as well as the score transition matrix.
- $\bullet$
Section D: demonstrates training details as well as evaluation details.
- $\bullet$
Section E provides more experimental results, including more OpenLLM leaderboard evaluation, LLM judge evaluation, exploring the curation impact on score-aware methods, comparison with LIMA, new combined baseline which concatenating high-rated examples across rating models.
- $\bullet$
Section F presents several wrongly-rated examples.

## Appendix A Prompt-based LLM Rating Systems

### A.1 Data Pool

The data pool used in this work consists of four proceed datasets, which originate either from human annotations or generated by powerful LLMs. More details about these datasets are provided in Table 7. In particular, these datasets vary in the format, quality, prompt length, and target tasks, demonstrating the diversity of our basic data pool. For convenience, we standardize the format of these datasets by using the “Tulu” template format introduced by Wang et al. [Wang et al., ]. The “Tulu” template consists of two main tags 
 and 
, reflecting the respective role of the user and the assistant.

**Table 7: Details of training datasets used in this work. WizardLM and Flan_v2 are sampled to 100K to match the dataset size. We report the average number of conservation turns ($\bar{N}_{\text{rounds}}$), average length of prompts ($\bar{L}_{\text{prompt}}$), average length of response ($\bar{L}_{\text{response}}$).**
| Datasets | Sourced from | # Data size | Data quality | $\bar{N}_{\text{rounds}}$ | $\bar{L}_{\text{prompt}}$ | $\bar{L}_{\text{response}}$ |
| --- | --- | --- | --- | --- | --- | --- |
| Flan V2 | Human-generated instruction | 100K | Normal | 1.0 | 304.1 | 27.7 |
| Open-Assistant 1 | human-generated instruction | 33K | Both | 1.6 | 32.3 | 189.1 |
| WizardLM | ChatGPT-generated instruction | 100K | High | 1.0 | 122.3 | 352.5 |
| Dolly | Human-generated instruction | 15K | Normal | 1.0 | 99.5 | 79.3 |
| Stanford Alpaca | Generated w/ Davinci-003 | 52K | Normal | 1.0 | 23.5 | 56.4 |

### A.2 Quality-based Prompt Template

The prompt template used in this work across various rating models is presented as follows. Our prompt template mainly accesses the data quality based on three criteria including rarity, complexity, and informativeness. For clarity and convenience, we adopt a JSON format to better capture the evaluation scores, following the LLaMA-3.1 template(^2^22[https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/)), as shown in Table A.2,.

#### Rated score rescaling

Initially, to capture the subtle differences between data samples, we first prompt the LLMs to rate them on a continuous integer scale $\{1,2,\cdots,10\}$. Intuitively, a lower score indicates that the data sample is of lower quality.
To simplify the score distribution, we first merge the lower scores in $\{1,2,3,4\}$ and the higher scores in $\{9,10\}$, resulting in a new scale of $\{4,5,\cdots,9\}$. For ease of convenience, we then shift this scale down to $\{0,1,\cdots,5\}$. Note that we focus primarily on high-rated samples in LLM ratings, so merging low-rated examples would not affect the overall performance and is more convenient for analyzing score errors in Section 3.2.
Directly rating samples on a small scale of $\{0,1,\cdots,5\}$ seems more convenient but fails to capture the subtle difference between samples, especially among higher-rated samples. Meanwhile, this commonly leads to the issue where most of the samples are rated as 3. Starting with a larger scale and then narrowing it down allows LLMs to distinguish subtle quality differences in mid-rated samples better, improving performance.

## Appendix B Data Selection Pipeline DS 2

### B.1 Algorithm Details

We provide the algorithm details of our proposed data selection pipeline in Algorithm 1.

Figure: Algorithm 1 Proposed Data Selection Pipeline DS^2

### B.2 Exploring the Impact of Embedding Models

By default, we use the newly released open-source model BGE as the embedding model throughout this paper. To explore the impact of embedding models, we adopt a popular alternative SetenceBERT [Reimers, ] to encode data samples.
The score transition matrix across various rating models in the SetenceBERT embedding space are provided in Figure 8.
In comparison to Figure 2 in the BGE embedding space, we can observe that the impact of embedding space is limited, the choice of embedding model does not significantly affect the error patterns produced by LLMs.

Figure: Figure 8: Score transition matrices across different rating models in the SentenceBERT embedding space.
Refer to caption: x10.png

## Appendix C Exploring The Impact of Score Curation on Examples

### C.1 Impact of Score Curation on Distribution

#### Rated score distribution between without and with curation

Here, we compare the rated score distribution between without and with score curation, as shown in Figure 9. We observe a decrease in the number of high-rated examples, while the number of samples with a rating of 3 has increased significantly. The rationale behind this is that our score curation mechanism is based on $k$-NN statistical information. As a result, given the imbalanced distribution of rated scores, samples with a rating of 5 are rare and are inevitably drawn toward the majority rating of 3. Therefore, the results in Figure 9 also highlight the importance of confidence probability proposed in Section 4.

Figure: Figure 9: Comparison of rated score distribution between without and with score curation.
Refer to caption: x11.png

#### Subset distribution of selected examples

Recall that the data pool is constructed by five subsets. Here, we summarize the statistical infomation of 10K samples generated by DS^2, focusing on the proportion of subsets. We can observe that 60%-70% of selected examples are from Wizardlm. The observation corresponds to the differences in data quality across five subsets summarized in Table 7.

Figure: Figure 10: Subset distribution proportion within 10K samples generated by DS^2.
Refer to caption: x12.png

### C.2 Impact of Score Curation on Score Errors

Instead of the impact of score curation on final performance, we are also interested in the impact of score curation on the detected score transition matrix. Figure 11 illustrates the error pattern of different rating models after applying score curation. In comparison to the results without applying score curation illustrated in Figure 2, the improvements are remarkable. Our score curation mechanism can significantly reduce the probability of incorrect score transition in the matrices.

Figure: Figure 11: Comparison of score transition matrices across different rating models after applying score curation.
Refer to caption: x13.png

## Appendix D Setup Details

#### Training details

In our experiments, we fine-tune 7B, 8B, and 70B models using four or eight NVIDIA Tesla A100 GPUs. Following the experimental setup [Wang et al., ], for all experiments based on 7B/8B models, we consistently apply Lora [Hu et al., ] with a rank size of 64 and a scaling factor of 16. Then, we set the overall batch size to 128, learning rate at 1e-4, the training epochs to 5, the dropout rate to 0.1, and a warm ratio of 0.03. The default maximum input length is 2048 tokens for all models.
In terms of the 70B base models, due to the larger scale, we apply Quantized Lora (QLora) [Dettmers et al., ], using 4-bit quantization to allow for efficient training. Besides, the hyperparameters are set to a batch size of 128, learning rate of 1e-4, training epoch 5, as well as a warm ratio of 0.1.

#### Evaluation details

In this paper, we select five tasks to conduct experiments for evaluation, consisting of MMLU, BBH, GSM, TydiQA, and TruthfulQA. The hyperparameter settings mainly follow recent work [Wang et al., ]’s. For ease of reproduction, we present some brief details here.

- $\bullet$
MMLU [Hendrycks et al., ]: Following the setup of MMLU, we conduct all evaluation in the 0-shot setting. In particular, to allow for faster evaluation, we quantize 70B models using 8 bits.
- $\bullet$
GSM [Cobbe et al., ]: We evaluate fine-tuned models on a randomly selected subset with 200 samples from the original test set (1319 samples). In particular, we apply 8-shot in-context examples to simulate the chain-of-thoughts (CoT) setting for reasoning.
- $\bullet$
BBH [Suzgun et al., ]: Given the official prompts provided in [Suzgun et al., ], we also apply 3-shot settings without CoT to make generations. Besides, we select 40 examples from each BBH sub-task.
- $\bullet$
TruthfulQA [Lin et al., ]: We prompt the fine-tuned models to generate answers for 818 TruthfulQA questions using the default QA prompt template with 6 in-context examples. Following the setting of [Wang et al., ], We apply two LLaMA-2-7B-based models for judging the generated responses’ truthfulness(^3^33https://huggingface.co/allenai/truthfulqa-truth-judge-llama2-7B) and informativeness(^4^44https://huggingface.co/allenai/truthfulqa-info-judge-llama2-7B). Judge models will help to evaluate the truthful and informative rate of responses, respectively. Following [Lin et al., ], we finally take the Informative-Truthful Rate as our metric, which is calculated by the numerical product of the Informative and the Truthful Rate.
- $\bullet$
TydiQA [Clark et al., ]: This dataset is used to evaluate the model performance in answering multilingual questions across nine different languages. For each language, we select 100 examples. To help the models become familiar with the answer format, one in-context example is provided during testing.
We report the average F1 score across various languages in this paper.

## Appendix E More Experiment Results

### E.1 OpenLLM Leaderboard Evaluation Results

We conduct additional experiments to evaluate the performance of the OpenLLM leaderboard across different baselines, utilizing various base models such as Mistral-7B-v0.3 and LLaMA-2-7B-hf. Tables 8 and 9 present the results of the OpenLLM leaderboard using Mistral-7B-v0.3 and LLaMA-2-7B-hf as the base model, respectively.
Both tables consistently demonstrate the effectiveness and superiority of our proposed pipeline DS^2, following the previous claims provided in Secion 5.

**Table 8: Performance comparison on OpenLLM leaderboard. By default, the selected data size is 10K. Base model: Mistral-7B-v0.3. We highlight the best result in boldface and the second-best with underline.**
| Models | MMLU | TruthfulQA | GSM | BBH | TydiQA | Average |
| --- | --- | --- | --- | --- | --- | --- |
| (factuality) | (truthfulness) | (reasoning) | (reasoning) | (multilinguality) |  |  |
| Vanilla base model | 59.7 | 30.2 | 38.0 | 49.6 | 54.9 | 46.5 |
| Completion length | 58.9 | 34.4 | 42.5 | 53.1 | 59.6 | 49.7 |
| Perplexity | 59.8 | 40.3 | 36.0 | 48.9 | 57.4 | 48.5 |
| $k$-NN-10 | 58.3 | 41.7 | 43.5 | 54.1 | 53.4 | 50.2 |
| Random Selection | 59.4 | 36.7 | 41.8 | 54.2 | 54.0 | 49.3 |
| LESS | 59.5 | 34.8 | 42.0 | 54.5 | 57.5 | 49.7 |
| Full data (300K) | 60.0 | 43.5 | 43.5 | 52.5 | 53.4 | 50.6 |
| Rating model: LLaMA-3.1-8B-Instruct |  |  |  |  |  |  |
| AlpaGasus | 59.9 | 36.4 | 39.0 | 52.6 | 56.3 | 48.8 |
| Deita | 60.0 | 37.1 | 43.5 | 54.0 | 57.7 | 50.5 |
| Ours w/o curation | 60.0 | 37.2 | 45.0 | 53.5 | 54.5 | 50.0 |
| Ours | 59.7 | 37.8 | 48.5 | 54.4 | 55.2 | 51.1 |
| Rating model: GPT-4o-mini |  |  |  |  |  |  |
| AlpaGasus | 60.5 | 36.7 | 41.0 | 55.1 | 57.3 | 50.1 |
| Deita | 60.1 | 35.6 | 40.5 | 55.1 | 56.0 | 49.5 |
| Ours w/o curation | 60.1 | 35.9 | 48.5 | 54.2 | 58.9 | 51.5 |
| Ours | 59.9 | 37.9 | 47.5 | 55.6 | 59.3 | 52.0 |
| Rating model: Mistral-7B-Instruct-v0.3 |  |  |  |  |  |  |
| AlpaGasus | 59.5 | 35.6 | 46.0 | 55.7 | 52.1 | 49.8 |
| Deita | 59.9 | 40.0 | 43.5 | 56.9 | 53.1 | 50.7 |
| Ours w/o curation | 59.5 | 37.9 | 46.5 | 55.8 | 57.2 | 51.4 |
| Ours | 59.5 | 40.3 | 48.5 | 53.0 | 55.9 | 51.4 |

**Table 9: Performance comparison on OpenLLM leaderboard. By default, the selected data size is 10K. Base model: LLaMA-2-7B-hf. We highlight the best result in boldface and the second-best with underline.**
| Model | MMLU | TruthfulQA | GSM | BBH | TydiQA | Average |
| --- | --- | --- | --- | --- | --- | --- |
| (factuality) | (truthfulness) | (reasoning) | (reasoning) | (multilinguality) |  |  |
| Vanilla LLaMa-2-7B | 41.9 | 28.4 | 6.0 | 38.3 | 35.7 | 30.1 |
| Completion Length | 42.4 | 36.4 | 1.5 | 36.8 | 33.9 | 30.2 |
| Perplexity | 45.0 | 41.5 | 12.0 | 31.7 | 39.5 | 33.9 |
| $k$-NN-10 | 38.2 | 40.8 | 15.0 | 36.0 | 43.8 | 34.8 |
| Random Selection | 44.7 | 41.8 | 14.0 | 37.9 | 40.8 | 35.8 |
| LESS | 44.3 | 38.2 | 18.0 | 35.2 | 46.3 | 36.4 |
| Full data (300K) | 50.1 | 36.2 | 16.5 | 40.5 | 46.7 | 38.0 |
| Rating model: llama-3.1-8B-Instruct |  |  |  |  |  |  |
| AlpaGasus | 45.1 | 41.2 | 18.0 | 35.6 | 39.8 | 35.9 |
| Deita | 43.6 | 36.4 | 14.5 | 33.9 | 39.7 | 33.6 |
| Ours w/o curation | 45.4 | 39.7 | 15.0 | 35.5 | 42.1 | 35.5 |
| Ours | 44.9 | 44.9 | 14.0 | 38.3 | 44.8 | 37.4 |
| Rating model: GPT-4o-mini |  |  |  |  |  |  |
| AlpaGasus | 45.3 | 41.0 | 14.5 | 37.0 | 45.3 | 36.6 |
| Deita | 45.2 | 44.7 | 13.5 | 35.6 | 43.4 | 36.5 |
| Ours w/o curation | 42.0 | 39.5 | 15.0 | 38.1 | 46.1 | 36.1 |
| Ours | 40.2 | 43.8 | 13.5 | 38.9 | 46.5 | 36.6 |
| Rating model: Mistral-7B-Instruct-v0.3 |  |  |  |  |  |  |
| AlpaGasus | 42.3 | 41.9 | 16.0 | 34.1 | 41.6 | 35.2 |
| Deita | 43.6 | 41.1 | 19.0 | 35.7 | 42.9 | 36.5 |
| Ours w/o curation | 46.0 | 48.6 | 15.0 | 35.2 | 43.7 | 37.7 |
| Ours | 40.8 | 50.9 | 15.0 | 37.9 | 45.5 | 38.0 |

### E.2 LLM Judge Evaluation

To evaluate alignment performance across baselines, we utilize Vicuna-Bench to access the instruction-following ability [Chiang et al., ]. Vicuna-Bench contains questions across nine domains, including generic, coding, math, and counterfactual. The judge model is GPT-4o-mini. Similarly, we present the final judge result in the typical "Win-Tie-Loss" rate form.
For convenience, the judge prompt template as referenced in [Zheng et al., ] can be found in Table 10.

**Table 10: The prompt template used for GPT-4o judge evaluation from [Zheng et al., ]**
| LLM Judge Prompt Template |
| --- |
| System Prompt: |
| You are a helpful and precise assistant for checking the quality of the answer. |
| User Prompt:<br>[Question]<br>[Assistant 1]: Assistant 1’s Answer<br>[Assistant 2]: Assistant 2’s Answer<br>We would like to request your feedback on the performance of two AI assistants in response to the user question<br>displayed above. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant<br>receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please<br>first output a single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The<br>two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your<br>evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not<br>affect your judgment. |

We compare all baselines, including our method against the full data baseline on Vicuna_Bench, as shown in Table 11. In particular, we conduct evaluations on two base models LLaMA-3.1-8B and Mistral-7B-v0.3. For score-aware baselines (AlpaGasus and Deita), we also compare them under three rating model settings. Notably, our method with curation outperforms almost all other baselines. What’s more, in most cases, we can observe that the score curation step improves model performance by reducing the loss rate without compromising the original win rate.

**Table 11: Performance comparison with full data baseline on Vicuna_Bench. Base models: LLaMA-3.1-8B and Mistral-7B-v0.3. LLM judge model: GPT-4o-mini. $\widetilde{\textbf{Win}}$ represents the adjusted win rate, which equals the win rate plus half of the tie rate. We highlight the best result in boldface and the second-best with underline.**
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Win(%) | Loss(%) | Tie(%) | $\widetilde{\textbf{Win}}$(%) | Win(%) | Loss(%) | Tie(%) | $\widetilde{\textbf{Win}}$(%) |
| Completion Length | 55.5 | 32.5 | 12.0 | 61.5 | 61.3 | 25.0 | 13.8 | 68.1 |
| Perplexity | 35.6 | 51.3 | 13.1 | 42.2 | 45.0 | 38.8 | 16.3 | 53.1 |
| $k$-NN-10 | 51.3 | 29.4 | 19.4 | 60.9 | 51.3 | 32.5 | 16.3 | 59.4 |
| Random Selection | 33.1 | 45.0 | 21.9 | 44.1 | 46.3 | 35.0 | 18.8 | 55.6 |
| LESS | 35.0 | 51.3 | 13.8 | 41.9 | 36.3 | 48.8 | 15.0 | 43.8 |
| Rating model: LLaMA-3.1-8B-Instruct |  |  |  |  |  |  |  |  |
| AlpaGasus | 50.6 | 28.8 | 20.6 | 60.9 | 57.5 | 27.5 | 15.0 | 65.0 |
| Deita | 40.6 | 45.0 | 14.4 | 47.8 | 46.3 | 36.3 | 17.5 | 55.0 |
| Ours w/o curation | 56.3 | 30.0 | 13.8 | 63.1 | 55.0 | 30.0 | 15.0 | 62.5 |
| Ours | 53.8 | 27.5 | 18.8 | 63.1 | 63.8 | 22.5 | 13.8 | 70.6 |
| Rating model: GPT-4o-mini |  |  |  |  |  |  |  |  |
| AlpaGasus | 67.5 | 18.8 | 13.8 | 74.4 | 73.8 | 10.3 | 15.9 | 81.7 |
| Deita | 54.6 | 32.1 | 13.3 | 61.3 | 63.1 | 26.3 | 10.6 | 68.4 |
| Ours w/o curation | 70.4 | 19.6 | 10.0 | 75.4 | 67.5 | 22.5 | 10.0 | 72.5 |
| Ours | 63.8 | 20.0 | 16.3 | 71.9 | 65.0 | 20.0 | 15.0 | 72.5 |
| Rating model: Mistral-7B-Instruct-v0.3 |  |  |  |  |  |  |  |  |
| AlpaGasus | 48.8 | 22.5 | 28.8 | 63.1 | 55.0 | 28.8 | 16.3 | 63.1 |
| Deita | 46.3 | 36.3 | 17.5 | 55.0 | 45.0 | 41.9 | 13.1 | 51.6 |
| Ours w/o curation | 51.7 | 33.8 | 14.6 | 58.9 | 61.9 | 25.0 | 13.1 | 68.4 |
| Ours | 51.3 | 31.3 | 17.5 | 60.0 | 62.5 | 20.0 | 17.5 | 71.3 |

### E.3 Exploring The Curation Impact on Other Score-aware Methods

Here, we present the curation impact on other score-aware methods, especially for Alpagasus and Deita under different rating model settings. The full experimental results can be found in Table 12.

**Table 12: Performance comparison between without and with score curation across all score-aware methods. Results are presented as (without curation / with curation). The selected base models are LLaMA-3.1-8B and Mistral-7B-v0.3.**
|  | Rating Model: LLaMA-3.1-8B-Instruct |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |
|  | AlpaGasus | Deita | Ours | AlpaGasus | Deita | Ours |
| MMLU | 63.1 / 63.8 | 64.1 / 64.6 | 63.4 / 63.8 | 59.9 / 59.4 | 60.0 / 59.8 | 60.0 / 59.7 |
| TruthfulQA | 42.4 / 36.1 | 35.3 / 46.3 | 50.2 / 45.4 | 36.4 / 41.7 | 37.1 / 39.8 | 37.2 / 37.8 |
| GSM | 59.5 / 65.5 | 60.0 / 64.0 | 61.5 / 62.5 | 39.0 / 40.0 | 43.5 / 43.0 | 45.0 / 48.5 |
| BBH | 60.9 / 63.1 | 60.8 / 58.3 | 59.3 / 61.2 | 52.6 / 53.5 | 54.0 / 52.4 | 53.5 / 54.4 |
| TydiQA | 64.8 / 62.7 | 63.0 / 61.3 | 61.7 / 67.9 | 56.3 / 52.3 | 57.7 / 58.0 | 54.5 / 55.2 |
| Average | 58.1 / 58.2 | 56.6 / 58.9 | 59.2 / 60.2 | 48.8 / 49.4 | 50.5 / 50.6 | 50.0 / 51.1 |
|  | Rating Model: GPT-4o-mini |  |  |  |  |  |
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |
|  | AlpaGasus | Deita | Ours | AlpaGasus | Deita | Ours |
| MMLU | 63.4 / 64.1 | 64.5 / 64.6 | 63.3 / 64.0 | 60.5 / 60.0 | 60.1 / 59.9 | 60.1 / 59.9 |
| TruthfulQA | 42.6 / 48.2 | 50.1 / 45.5 | 51.5 / 50.3 | 36.7 / 39.8 | 35.6 / 41.1 | 35.9 / 37.9 |
| GSM | 66.0 / 61.5 | 60.0 / 64.0 | 62.0 / 67.5 | 41.0 / 41.5 | 40.5 / 42.5 | 48.5 / 47.5 |
| BBH | 59.1 / 58.9 | 60.3 / 61.8 | 59.7 / 59.0 | 55.1 / 53.6 | 55.1 / 55.3 | 54.2 / 55.6 |
| TydiQA | 59.4 / 64.8 | 63.7 / 67.1 | 64.3 / 66.1 | 57.3 / 56.5 | 56.0 / 56.4 | 58.9 / 59.3 |
| Average | 58.1 / 59.5 | 59.7 / 60.6 | 60.2 / 61.4 | 50.1 / 50.3 | 49.5 / 51.0 | 51.5 / 52.0 |
|  | Rating Model: Mistral-7B-Instruct-v0.3 |  |  |  |  |  |
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |  |  |  |  |
|  | AlpaGasus | Deita | Ours | AlpaGasus | Deita | Ours |
| MMLU | 63.2 / 64.2 | 63.9 / 63.5 | 63.0 / 63.3 | 59.5 / 59.6 | 59.9 / 59.5 | 59.5 / 59.5 |
| TruthfulQA | 45.8 / 40.0 | 50.3 / 51.3 | 48.2 / 53.9 | 35.6 / 38.9 | 40.0 / 38.7 | 37.9 / 40.3 |
| GSM | 62.0 / 60.5 | 61.0 / 61.0 | 67.0 / 62.0 | 46.0 / 46.5 | 43.5 / 44.0 | 46.5 / 48.5 |
| BBH | 60.5 / 63.5 | 60.4 / 59.5 | 59.2 / 61.1 | 55.7 / 55.6 | 56.9 / 54.1 | 55.8 / 53.0 |
| TydiQA | 62.2 / 63.5 | 62.8 / 64.6 | 65.9 / 65.1 | 52.1 / 56.6 | 53.1 / 55.1 | 57.2 / 55.9 |
| Average | 58.7 / 58.3 | 59.7 / 60.0 | 60.7 / 61.1 | 49.8 / 51.4 | 50.7 / 50.3 | 51.4 / 51.4 |

### E.4 Comparison with High-quality Human-annotated Examples: LIMA

In this section, we also utilize the original LIMA test set (300 samples) to compare the performance between LIMA (human annotation) and DS^2 (machine annotations).
Similarly, we finetune two base models (LLaMA-3.1-8B and Mistral-7B-v0.3) on 1k LIMA samples.
The finetuned models are then directly compared with finetuned models using DS^2 selected examples at both 1k and 10k sample sizes.
The experimental results for 1k and 10k settings are shown in Figure 12 and 13, respectively.
While DS^2 performs worse than LIMA in the 1k sample setting, it totally surpasses LIMA in the 10k setting, consistently demonstrating the superiority of DS^2. This lower performance at the 1k setting is expected, as LIMA has a natural advantage in a limited sample size scenario due to the IID nature of its training and test sets.

Figure: Figure 12: Performance of models fintuned on DS^2 (1k samples, machine-curated) v.s. LIMA (1k samples, human-curated). Evaluation set: LIMA (300 samples). We use the initial letter to denote the rating model, e.g., Ours (L) refers to our method with LLaMA-generated scores (Ours (LLaMA)).
Refer to caption: x14.png

Figure: Figure 13: Performance of models fintuned on DS^2 (10k samples, machine-curated) v.s. LIMA (1k samples, human-curated). Evaluation set: LIMA (300 samples). We use the initial letter to denote the rating model, e.g., Ours (L) refers to our method with LLaMA-generated scores (Ours (LLaMA)).
Refer to caption: x15.png

### E.5 Exploring the Impact of Concatenating High-Rated Examples Across Rating Models

#### Combined Baseline

In this section, we are also interested in the performance of concatenating samples from three rating models. We combined all high-rated samples with a score of 5, resulting in a subset of 8K samples. To reach a total of 10K samples, we added 2K samples from the data pool that were both rated 4 by all rating models.
Compared to the results shown in Table 3 and Table 8, one can observe that the combined baseline still fails to achieve strong performance.

**Table 13: Performance of combined baseline on OpenLLM Leaderboard.**
|  | Combined baseline |  |
| --- | --- | --- |
|  | LLaMA-3.1-8B | Mistral-7B-v0.3 |
| MMLU | 64.2 | 59.6 |
| TruthfulQA | 41.7 | 37.1 |
| GSM | 62.5 | 43.5 |
| BBH | 61.9 | 51.0 |
| TydiQA | 60.8 | 53.1 |
| Average | 58.2 | 48.9 |

## Appendix F Mis-rated Examples

Here, we provide more wrongly-rated examples detected by our algorithms. For convenience, we highlight the raw (original) rated score and the curated score of each example.

### F.1 Original High-rated Examples vs. Original Low-rated Examples

Here are several examples to illustrate the data quality between high-rated and low-rated examples from different datasets, including Flan_v2, Open-Assistant1, Stanford Alpaca, Dolly and WizardLM.

**Table 14: Comparison of low-rated and high-rated samples from Flan_v2 dataset**
| Low-rated Flan_v2 sample | High-rated Flan_v2 sample |
| --- | --- |
| 
: Triple: Alimentum eatType restaurant; Alimentum food Indian; Alimentum priceRange moderate; Alimentum area riverside; Alimentum familyFriendly no. What is a sentence that describes this triple?<br>
: "Alimentum is an Indian restaurant located the the riverside area. It is moderately priced, but not kid friendly." | 
: I walked in here during an all day rain-a-thon in Pittsburgh (so rare, right?) on a Saturday. It was pretty busy, but all of the hustle and bustle was contained to the upstairs, which looked like a cool loft area. It wasn’t loud either. The first thing I noticed is how high the ceiling was and I loved that. The art on the walls was a delight to look at too. I ordered something vegan with tofu (I’m not vegan by any means, but wanted to try it), and I got it delivered to me on a plate with two small toasted pieces of bread, lettuce, slices or bell pepper, and warmed tofu. There was something else on the bread that I don’t remember, but I was really surprised by how delicious and satisfying this was. I am a fan. For my drink I got an iced vanilla latte and I have to say it’s the best vanilla latte I’ve had around town. I ended up having two more throughout my time there…I just inhaled them. Seriously..so good. I came back on Sunday and had an iced mocha, which was just as good. This is my new favorite place for real!<br>Choose: was this review given positively or negatively?<br>
: positive |

**Table 15: Comparison of low-rated and high-rated samples from Open-Assistant 1 dataset**
| Low-rated Open-Assistant 1 sample | High-rated Open-Assistant 1 sample |
| --- | --- |
| 
: que actividades se supone que son comunes en una persona de 20 años?<br>
: Las actividades más comunes entre personas de 20 años usualmente son estudiar, debido a la alta demanda de tareas en la universidad, así como, salir a divertirse, tomar alcohol eventualmente, o salir de fiesta, así mismo, se puede incluir hacer deporte, y jugar videojuegos.<br>
: Las actividades que se supone que son comunes en una persona de 20 años varían dependiendo de muchos factores, como la cultura, el entorno, la educación y los intereses personales. Sin embargo, aquí hay algunas actividades que pueden ser comunes para muchas personas de 20 años:<br>•<br>Estudiar o trabajar: Muchas personas de 20 años se encuentran en una etapa de transición entre la educación y el mundo laboral, por lo que pueden estar estudiando o trabajando en sus primeras carreras o empleos.<br>•<br>Establecer relaciones: Las personas de 20 años a menudo están explorando su vida amorosa y estableciendo relaciones íntimas.<br>•<br>Viajar: Muchas personas de 20 años aprovechan esta etapa para viajar y explorar el mundo.<br>•<br>Desarrollar habilidades y pasatiempos: Las personas de 20 años pueden estar explorando nuevas habilidades y pasatiempos, y tratando de encontrar aquellos que son significativos para ellos.<br>•<br>Participar en actividades comunitarias: Las personas de 20 años a menudo se involucran en actividades comunitarias, como voluntariado, grupos de interés o política.<br>
: Entiendo que me estabas poniendo a prueba. Tomaré en cuenta esas actividades para mis próximas respuestas. | 
: Hazme una clave dicotómica de los animales invertebrados.<br>
: ¡Por supuesto! Aquí tienes una clave dicotómica para clasificar los animales invertebrados:<br>•<br>¿Tiene un exoesqueleto duro y articulado? -> Ir al paso 2<br>•<br>¿No tiene exoesqueleto duro y articulado? -> Ir al paso 3<br>•<br>¿Tiene patas articuladas y segmentadas? -> Es un artrópodo<br>•<br>¿No tiene patas articuladas y segmentadas? -> Ir al paso 3<br>•<br>¿Tiene tentáculos y una estructura de cuerpo en forma de campana? -> Es un cnidario<br>•<br>¿No tiene tentáculos y una estructura de cuerpo en forma de campana? -> Ir al paso 4<br>•<br>¿Tiene un cuerpo blando y una concha univalva o bivalva? -> Es un molusco<br>•<br>¿No tiene un cuerpo blando y una concha univalva o bivalva? -> Ir al paso 5<br>•<br>¿Tiene una simetría radial y estructuras como brazos o patas para moverse y capturar alimentos? -> Es un equinodermo<br>•<br>¿No tiene una simetría radial y estructuras como brazos o patas para moverse y capturar alimentos? -> Ir al paso 6<br>•<br>¿Tiene una estructura de cuerpo segmentada y sin patas articuladas? -> Es un anélido<br>•<br>¿No tiene una estructura de cuerpo segmentada y sin patas articuladas? -> Ir al paso 7<br>•<br>¿Tiene un cuerpo alargado y segmentado con tentáculos alrededor de la cabeza? -> Es un gusano platelminto<br>•<br>¿No tiene un cuerpo alargado y segmentado con tentáculos alrededor de la cabeza? -> Es un invertebrado no clasificado en las categorías anteriores. |

**Table 16: Comparison of low-rated and high-rated samples from WizardLM dataset**
| Low-rated WizardLM sample | High-rated WizardLM sample |
| --- | --- |
| 
: Instruction: As an intermediate English learner at B1 level, I have been trying to familiarize myself with certain words like "captivity", "chew out" and "cuddly". Would you kindly ask me a question using any of these three terms, to help me better comprehend the usage and context of these words?<br>
:<br>Response: Certainly! Here are three questions using each of the terms you mentioned:<br>1.<br>Have you ever visited a zoo where animals are kept in captivity?<br>2.<br>Has your boss ever chewed you out for making a mistake at work?<br>3.<br>Do you prefer to have a pet that is cuddly and affectionate, or one that is independent and aloof? | 
: Instruction: As an archaeologist exploring the depths of an uncharted jungle, you come across a temple that appears to be a blend of Mayan and Roman architecture. The engravings on the walls and artifacts found within the temple suggest that the civilization that built it was a fusion of both cultures. Your task is to piece together the history and culture of this ancient civilization and unravel the mystery of how and why it came to be. Were these two cultures united in peace or war? Did they form a new civilization or remain separate? Let’s excavate the truth behind this enigma.<br>
:<br>Response: As you begin your exploration of the temple, you come across a set of glyphs on one of the walls that seem to represent both Mayan and Roman symbols. You take note of them and continue your journey deeper into the temple. Inside, you find a variety of artifacts that suggest a fusion of both cultures, including pottery, weapons, and clothing….<br>….Further exploration reveals that this civilization was not a separate entity, but rather a part of a larger network of civilizations that spanned the continent. The Mayan and Roman cultures in this region worked alongside other cultures, such as the Aztecs and the Incas, to form a vast and complex system of trade and cultural exchange.<br>In the end, you realize that this ancient civilization was a testament to the power of cooperation and cultural exchange. The Mayan and Roman peoples came together in peace, combining their knowledge and skills to create something new and beautiful. Their legacy lives on in the unique artifacts and engravings that remain in the temple, waiting to be discovered and appreciated by future generations. |

**Table 17: Comparison of low-rated and high-rated samples in Dolly dataset**
| Low-rated Dolly sample | High-rated Dolly sample |
| --- | --- |
| 
: Instruction: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.<br>Instruction: When was the 8088 processor released?<br>Input: The 8086 (also called iAPX 86) is a 16-bit microprocessor chip designed by Intel between early 1976 and June 8, 1978, when it was released. The Intel 8088, released July 1, 1979, is a slightly modified chip with an external 8-bit data bus (allowing the use of cheaper and fewer supporting ICs),[note 1] and is notable as the processor used in the original IBM PC design.<br>
:<br>Response: The Intel 8088 processor was released July 1, 1979. | 
: Instruction: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.<br>Instruction: Who is the best surfer in the world?<br>Input: Robert Kelly Slater (born February 11, 1972) is an American professional surfer, best known for being crowned World Surf League champion a record 11 times. Slater is widely regarded as the greatest professional surfer of all time, and holds 56 Championship Tour victories. Slater is also the oldest surfer still active on the World Surf League, winning his 8th Billabong Pipeline Masters title at age 49.<br>
:<br>Response: Kelly Slater |
| Low-rated Alpaca sample | High-rated Alpaca sample |
| 
: Instruction: Given a set of numbers, find the maximum value.<br>Input: Set: {10, 3, 25, 6, 16}<br>
:<br>Response: The maximum value is 25. | 
: Instruction: Create a list of best practices that should be followed in this field.<br>Field: Healthcare information technology<br>
:<br>Response: The following best practices should be followed in the healthcare information technology field:<br>1.<br>Adopt a risk-based approach to cybersecurity, utilizing measures such as multi-factor authentication, encryption, and vulnerability assessments.<br>2.<br>Develop clear guidelines and policies governing the use of technology.<br>3.<br>Develop protocols to ensure the secure sharing of patient data.<br>4.<br>Implement robust data processing and storage systems that adhere to industry regulations.<br>5.<br>Train healthcare staff in both the technical and non-technical aspects of IT use.<br>6.<br>Utilize professional IT staff for any maintenance or development tasks.<br>7.<br>Regularly test and monitor IT systems for potential vulnerabilities. |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: For the k-NN agreement score, assuming the k-NN clusterability characteristic, isn't the similarity score just always unity?  I.e., due to clusterability, the frequency will only occur in the same index as the one-hot encoding vector, and thus the numerator and denominator turn out to be the same.  Also, have the authors empirically explored whether using the embedding feature-vector or the one-hot encoding rated score vector work better in practice?

On the initial read through the paper, there was some initial confusion between: the avoidance of relying on the ground truth scores for DS^2, yet the appearance of the ground truth per-class probability in Section 4.  However, after careful rereading, we have the following: assuming k-NN score clusterability, we are thus able to leverage the concensus vectors and solve the LP for \mathbf{T} and \mathbf{p} (I understand this is stated, but it is easy to miss this simple statement buried in lines 231-241).  I would recommend either reiterating that solving the LP formed from Equation 1 provides both the transition matrix and ground truth probability vector, or breaking up the paragraph on lines 231-241 and emphasizing this.

"the average likelihood of identifying the sample n as misrated over multiple epochs." <- How is this calculated in practice?

For Completion Length, please cite the following:
Zhao, Hao, et al. "Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning." arXiv preprint arXiv:2402.04833 (2024).

For Table 3, please specify the data pool is listed in Table 2, e.g., " Performance comparison on OpenLLM leaderboard, using the data pool listed in Table 2."

Could the authors please bold the winners of each column (per grouping) in Table 3?
soundness: 3
strengths: The procedure to estimate the transition matrix from LLM scores is intuitive.  The beginning of the paper is also well written, and the examples showing the deficiencies in LLM-based scores is compelling.  The subsequent probability quantities leveraged throughout the DS^2 algorithm also largely make sense.  The large number of experiments demonstrating the effectiveness of the approach is also compelling.
summary: The authors introduce a data curation algorithm, DS^2, to correct inaccuracies in LLM-based data quality evaluation strategies.  The authors first show how, leveraging recent work in noisy label estimation given k-NN clusterability, both the Transition matrix and true probability distribution of LLM scores may be estimated by solving a linear program.  Under this k-NN clusterability assumption, the authors demonstrate that widely used LLM scorers (GPT-4o-mini, Llama-3.1 8B, and Mistral 7B) provide incorrect scores.  Using both Transition matrix and (true score) probability distribution, the layout the steps of their DS^2 algorithm and calculation of critical quantities: the error threshold for classifying samples as misrated, the confidence probability for mitigating imbalances in LLM scores, and the long-tail diversity score.

The authors then compare several data selection experiments leveraging the DS^2 algorithm over a large data pool (300k) containing several widely used datasets (i.e., Alpaca, Dolly, Flan V2, etc.).  Compared to several data selection heuristics and competitor LLM-based data selection algorithms (i.e., AlpaGasus and Deita), the authors show that their methods outperforms competitors per-LLM-judge and averaged over common natural language benchmarks (MMLU, TruthfulQA, GSM, BBH, and TydiQA) at a data selection size of 10k samples.  Representative experiments than compare the performance of DS^2 selecting 1k samples (over the 300k data pool) compared to the LIMA dataset (which was heavily vetted and selected using human feedback), once again outperforms finetuning LLama-3.1-8B and Mistral 7B models using the LIMA dataset.  Scaling law experiments as well as hybrid data selection schemes (e.g., AlpaGasus leveraging DS^2's data curation scores) further show the efficacy of the presented methods.

# Post-rebuttal update
The authors have posted additional experiments to address my major concerns.  Particularly, an apples to apples comparison to AlpaGasus demonstrating DS^2's superiority and a discussion/experiments tackling the practical aspects of k-NN clusterability for this use case.  I am raising my score from 5 to 6.
weaknesses: Firstly, I would like to congratulate the authors on the DS^2 algorithm and the idea of leveraging k-NN clusterability to correct LLM-based scores.  Such a line of work seems promising.  With that said, I have several concerns that I am hoping can be addressed (I am greatly looking forward to the authors feedback).

While the beginning of the paper is well written, there seemed to be either missing details or a lack clarity for important concepts.  In particular, Algorithm 1 is difficult to follow; "CuratedScores" is not defined in Algorithm 1. Are "ConfidenceProbs" the curated scores?  Furthermore:
> the average likelihood of identifying the sample n as misrated over multiple epochs.

How is this calculated in practice?  Given a dataset with an arbitrary ordering of samples, how can a statistic dealing with the arbitrary index $n$ be meaningful?  Put another way, this calculates the statistics at the sample index, but the data pool samples (line 172) are not stated as being ordered in any specific way.  Thus, how can a statistic calculated over the sample index be anything other than arbitrary?

Furthermore, while the notion of k-NN score clusterability (as defined in Definition 3.2) is assumed, this notion is never actually demonstrated for the LLM judges and datasets considered.  Do the score distributions depicted in FIgure 2 abide by k-NN score clusterability?  If not, could the authors speak to how this affects the Transition matrix and (true score) probability distribution calculations from the concensus vectors (Equation 1)?  If k-NN score clusterability is violated in practice, it does not seem possible to guarantee the correctness of estimating the Transition matrix and (true score) probability distribution in this case.

There were some concerns regarding the evaluations of AlpaGesus:
> AlpaGasus (Chen et al., 2023) utilizes ChatGPT to rate
data samples and solely select high-rated samples"

One of the major innovations of AlpaGasus was the prompt template.  Was this used for the AlpaGasus results in Table 3?  Note that they also showed the training data size is a very important hyperparameter for Alpaca; thus, a fairer/more apples-to-apples comparison would have been DS^2 applied to Alpaca with a budget of 9k samples and compared to fine-tuning performance (Table 3) of the AlpaGasus dataset.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: * It looks like that there is an advantage of using GPT-40 mini. How to obtain embeddings from proprietary LLMs?
* Figure 1 reads like ratings of multiple LLMs are combined. The experimental results say otherwise.
* The proposed approach sometimes perform much worse (e.g., in Table 2). Is it possible to decide the appropriate conditions for using the proposed approach (and which LLM)?
soundness: 3
strengths: * The idea of detecting error pattern is novel. Its implementation using score transition matrix was validated experimentally.
summary: This study tackles the problem of rating samples using LLM, more specifically, how to improve the accuracy of LLM-based rating methods. To this end, DS2 (Diversity-Aware Score Curation for Data Selection) is proposed to model rating error pattens using a score transition matrix. Experiments on machine-alignment benchmarks show that strong results can be achieved by using just 3.3% of the original dataset.
weaknesses: * The description of "deriving the score transition matrix" can be improved. Not sure I can reproduce it based on the given explanation. Some examples can help, which can be included in appendix.
* It is assumed that "the transition matrix is independent of sample-level features". This can be a limitation as each LLM have its own strengths and weakness (i.e., better/worse at some samples).
* Sometimes, the improvement is small, within the variance of different runs.
* Some discussions about limitations would be appreciated.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: I find the discussion around line 200, saying that similar embeddings should belong to the same category, not very convincing. Because here each class is the rating score, in my opinion, two samples could have similar textual embeddings but completely different correctness or quality, which lead to distinct rating scores. I think it is crucial to better explain this intuition behind the proposed method.
soundness: 3
strengths: Originality: The paper proposed a novel method to utilize the score transition matrix to detect scoring errors. Although the score transition matrix is proposed by previous works, the application of that into LLM data selection is novel.

Quality: The paper provides detailed explanations of their methods and validates the results using thorough experiments, in which they include three models (GPT-4o-mini, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3) and various benchmarking tasks (MMLU, TruthfulQA, etc). Moreover, they also evaluated the alignment performance. All these greatly improve the quality of the paper.

Clarity: The paper demonstrated great clarity when exhibiting the error patterns of LLM scores, as well as when explaining the experiment setup and results.

Significance: The significance is validated with the experiment results and the ablation studies.
summary: The paper proposed the data selection pipeline named “DS^2”, which utilizes the kNN statistical information based on the rating scores to curate ratings, and then achieve better data selection. The authors validated their methods by conducting experiments with various rating models and benchmark tasks.
weaknesses: The paper does not include discussions about the efficiency of the proposed method and the efficiency comparison with other baseline methods. Part of the computation relies on estimating the score transition matrix, but the details about this estimation are lacking. Although the score transition matrix is proposed and discussed in previous works, it is still necessary to analyze their cost within the current framework.  Moreover, the paper does not include any baselines that also improve data quality by modifying the score ratings of the data. The current framework essentially proposed a method to modify the rating scores and then conduct data selection based on new scores. It would be helpful to further validate the method by comparing other methods that also study LLM ratings.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the authors provide more examples of typical errors in LLM-rated scores and explain how DS2 specifically addresses these? This would clarify the practical benefits of the score transition matrix.
2. Could the authors discuss how the diversity score metric performs in datasets with different distributions or in tasks with varying levels of data complexity? Providing case studies or examples would help demonstrate the versatility of this metric.
3. How sensitive is DS2's performance to the choice of embedding model used for the k-NN agreement score? Exploring different embeddings beyond the BGE model may provide insights into the robustness of the approach.
4. In an open-ended question, the responses may not be similar at all (resulting in very different embedding vectors), but they could all be equally excellent answers. In this method, I believe the two responses may be curated into completely different scores, leading to errors. ”Intuitively, a lower cosine similarity score indicates a higher likelihood of a rating error”, This assumption does not adequately explain the examples above.
5. "please tell me the answer of below math question '1+1=?' answer:2"
"please tell me the answer of below math question '1+1=?' answer:3"
The two questions have similar embedding vectors, but they should have completely different scores. This seems to contradict the notion that samples with high embedding vector similarity are more likely to have the same labels.
soundness: 3
strengths: 1. The paper argue that a smaller, high-quality dataset can outperform larger datasets. The introduction of a score transition matrix to model LLM-based errors adds a novel approach to data curation, combining insights from label noise reduction and LLM evaluation.
2. The experimental section is substantial and persuasive, featuring a rich number of comparative experiments that cover various scenarios and conditions, effectively demonstrating the robustness and efficacy of the proposed algorithm. Furthermore, the experimental results indicate significant performance improvements, further validating the superiority of the proposed approach.
summary: The paper presents DS2, a method for improving data efficiency in instruction tuning for LLMs. DS2 corrects inaccuracies and biases in LLM-generated data quality scores using a score transition matrix and promotes diversity in the data selection process. The authors demonstrate that using a curated subset  can outperform larger datasets, challenging traditional data scaling laws and achieving better results than human-curated datasets like LIMA. Key contributions include modeling score errors across various LLMs, developing a novel data curation pipeline, and conducting extensive experiments to validate DS2's effectiveness against multiple baselines.
weaknesses: 1. While the score transition matrix is central to DS2, the paper lacks an in-depth analysis of its limitations or conditions under which it may not be effective. For instance, the independence assumption (that the transition matrix is not sample-specific) might limit its ability to model certain data-specific errors.
2. The paper heavily relies on the capabilities of pre-trained LLMs to generate initial quality ratings and the transition matrix to correct the final quality score.
3. If there is a strong LLM evaluator (eg, GPT-4o or o1), the score error correction process seems unnecessary.

Meta-review: The paper introduces a curation method for finetuning data that selects data samples and demonstrates that a model finetuned on a small dataset curated with the method can outperform a model trained on the larger full-scale dataset. 

All reviewers find the proposed approach interesting and clever (Reviewer ZhKe find leveraging the k-NN clusterability interesting, Reviewers 81Qt and gzew find the idea to detect error patterns novel, and Reviewer XKFq the method original). The paper also provides relatively comprehensive experiments and demonstrates that the method performs well. 

The reviewers raised a variety of concerns (such as a comparison to AlpaGauss required, and many clarity issues), and the authors addressed them sufficiently.

Like the reviewers, I find the idea introduced clever and the result that a relatively small finetuning dataset curated with the proposed method can outperform the full dataset very interesting. Therefore I recommend acceptance.

Decision: Accept (Poster)