Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

## Abstract

Abstract Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset.
To tackle this problem, we propose a new framework that boosts the alignment of LLMs through Sel f-generated Pre fe r e nce data ( Selfee ) using only a very small amount of human-annotated preference data.
Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data.
To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model’s inherent preference.
Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective.
In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs.
For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.

## 1 Introduction

Recently, large language models (LLMs) have made huge progress in various NLP tasks, leading to real-world applications that are used by millions of users, such as coding assistants and chatbots [, , ].
Aligning LLMs with human feedback, particularly through learning from human preferences, is widely considered a crucial technique for their success [, , ].
To enhance this alignment, various preference learning algorithms have been extensively explored [, ].
Despite these advancements, one of the remaining challenges is the reliance on large-scale human-annotated preference data.
As the quality and quantity of preference data are critical for the successful alignment of LLMs [, ], the huge cost to acquire such data inevitably presents significant obstacles.

To mitigate this challenge, engaging LLMs in constructing preference data and improving their alignment using this data has recently gained attention.
For example, a representative way on this line is generating multiple responses for the input prompts, and then approximating human preference between them through LLM’s predictions, often referred to as LLM-as-judge [, ].
However, these approaches are only effective when the given LLM is sufficiently large and well-aligned to mimic human preference via in-context learning.
On the other hand, using an external reward model is considerable to substitute human preference annotation efficiently [, ], but it is built on the availability of large human preference data and also could be ineffective if the distribution mismatch exists.
Lastly, these approaches have a risk of potential labeling noise from LLMs, but this aspect has not been explored yet.
Therefore, in this work, we aim to develop a method to effectively improve the alignment of LLM by overcoming these limitations but only relying on the small human annotation.

Figure: Figure 1: Illustration of the proposed Selfee framework. Selfee progressively improves the alignment of LLMs by iterating (1) the generation of new preference data and (2) the preference learning on the constructed data with self-refinement. Technical details are presented in Section 4.
Refer to caption: x1.png

Contribution. We introduce a simple yet effective framework, coined Selfee, to improve the alignment of LLMs via Self-generated Preference data, with only a small amount of human-labeled preference data.
Our key idea is to progressively expand the knowledge of human preference within the small (seed) data, by iteratively generating the responses and learning from them through the self-annotated preference labels.
Specifically, our technical contributions are three-fold as described in what follows.
First, we derive the preference labels using the logits of LLM to explicitly extract the model’s inherent preference.
This approach is more effective compared to previous methods that rely on external reward models or implicit in-context learning.
Second, we introduce a confidence-based refinement of preference labels to reduce the risk of noise in preference learning with generated data.
Third, to further enhance the effectiveness of this refinement, we propose using a linearly extrapolated prediction between the current and reference models; this approach approximates the predictions of a more strongly aligned model, leading to better noise identification.

Figure: Figure 2: Summary of main result. Evaluation results on AlpacaEval2.0 benchmark []. Our framework significantly improves the alignment of LLMs, without additional human preference data. See detailed results in Section 5.
Refer to caption: x2.png

We demonstrate the effectiveness of the proposed Selfee by aligning recent LLMs with small human-annotated preference data and evaluating their alignment on the commonly used benchmarks.
For example, using only 3.3% of ground-truth preference in Ultrafeedback data [] with the mistral-7b-0.1v SFT model [], our framework achieves over 16.4% increase in AlpacaEval2.0 [] win rate compared to the initial SFT model (see Figure 2).
Additionally, the AlpacaEval2.0 length-controlled win rate is improved from 7.58% to 15.39%, and MT-bench score [] increased from 6.38 to 6.94.
Compared to preference judgment methods like LLM-as-judge [], and even strong reward models such as PairRM [], which have recently shown state-of-art performance in AlpacaEval2.0 benchmark, our approach consistently outperforms them across all metrics.
More interestingly, the proposed Selfee successfully improves the alignment of various LLMs, even without the initial human preference data.
These results demonstrate that our framework is highly competitive and practical for real-world applications.

## 2 Related Work

Alignment of LLMs with human preference.
Learning from human preferences now serves as a core component for the state-of-the-art LLMs [, , , ] for aligning their responses with users’ intent and values [, ].
Arguably, one of the most popular frameworks is reinforcement learning with human preference (RLHF) [, ], which first trains the reward model, and then fine-tunes LLM to maximize that reward with KL divergence regularization to prevent the reward over-optimization of LLM.
On the other hand, various training algorithms have recently been proposed to fine-tune LLMs with human preference more efficiently [, , , , , ].
For example, Rafailov et al. [] proposes Direct Preference Optimization (DPO) which allows one to fine-tune LLMs without a separate reward modeling stage, by deriving the training objective mathematically equivalent to RLHF.
Ethayarajh et al. [] further removes the reliance on pair-wise preference labels by formulating the objective based on a human utility model.
However, these methods assume that large human-annotated preference data is available, which requires a huge data acquisition cost.

Engagement of LLMs for constructing preference data.
For an efficient and scalable alignment procedure, engaging LLMs for preference dataset construction has recently received attention.
One common approach involves generating multiple responses to input prompts from LLM, and using an LLM’s predictions to approximate human preferences between them, a technique often referred to as LLM-as-judge [, ].
However, this method is effective only when the LLM is sufficiently large and well-aligned to mimic human preferences through in-context learning.
Alternatively, employing an external reward model can efficiently replace human preference judgment [, ], but this approach relies on the availability of extensive human preference data to pre-train reward model and may be ineffective if there is a distribution mismatch.
Some concurrent works [, , , ] have proposed the alignment procedure with iterative data expansion and preference learning.
However, they use the external reward model or stronger LLM for the preference judgment.
In contrast, we only utilize the intrinsic knowledge of training LLM with new data expansion and preference learning methods.

## 3 Preliminaries

Let denote LLM as $\pi_{\theta}$, which generates an output sequence (e.g., response) $y$ for a given input sequence (e.g., prompt) $x$,
i.e., $y\sim\pi_{\theta}(\cdot|x)$.
Then, our goal is to make $\pi_{\theta}$ provide human-aligned responses for the various input prompts.
To this end, we consider the popular framework of preference learning, which optimizes $\pi_{\theta}$ to learn the human preferences between two different responses [, , ].
Specifically, we assume that the preference dataset $\mathcal{D}=\{(x,y_{l},y_{w})\}$ is available which consists of the triplets of input prompt $x$, preferred response $y_{w}$, and dispreferred response $y_{l}$.
Here, the preference labels were annotated by a ground truth annotator, that is usually a human expert.

Reward modeling and RL fine-tuning.
Since a pairwise preference between $y_{w}$ and $y_{l}$ is hard to model directly, one of the common practices is introducing reward function $r(x,y)$ and modeling the preference based on this using the Bradley-Terry model []:

$$ $p(y_{w}\succ y_{l}\mid x)=\frac{\exp\left(r(x,y_{w})\right)}{\exp\left(r(x,y_{ w})\right)+\exp\left(r(x,y_{l})\right)}.$ (1) $$

From this formulation, one can introduce a parametrized reward model $r_{\phi}(x,y)$ by estimating its parameters with the maximum-likelihood objective:

$$ $\mathcal{L}_{R}(r_{\phi})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[ \log\sigma\left(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})\right)\right].$ (2) $$

where $\sigma$ is a sigmoid function.
After this reward modeling procedure, one could improve the alignment of LLM $\pi_{\theta}$ by optimizing it to maximize the reward captured by $r_{\phi}$.
Here, KL-distance from the reference model $\pi_{\text{ref}}$ is usually incorporated as a regularization to prevent the reward over-optimization of $\pi_{\theta}$, with a hyper-parameter $\beta>0$ [, ]:(^1^11$\pi_{\text{ref}}$ is usually initialized with supervised fine-tuned (SFT) LLM [, ]. Also, $\pi_{\theta}$ is initialized with $\pi_{\text{ref}}$.)

$$ $\mathcal{L}_{\text{RLHF}}(\pi_{\theta})=-\mathbb{E}_{y\sim\pi_{\theta},x\sim \rho}\left[r_{\phi}(x,y)\right]+\beta\mathrm{D}_{\mathrm{KL}}\left(\pi_{\theta }(y|x)\parallel\pi_{\text{ref}}(y|x)\right).$ (3) $$

Direct preference modeling and optimization.
Rafailov et al. [] propose an alternative approach to align LLM $\pi_{\theta}$ with preference dataset $\mathcal{D}$, which is called Direct Preference Optimization (DPO).
DPO integrates a two-step alignment procedure with reward modeling and RL fine-tuning into a single unified fine-tuning procedure.
Specifically, the optimal reward function is derived from RLHF objective (Eq. 3), with the target LLM $\pi_{\theta}$ and reference model $\pi_{\text{ref}}$ [, , ].

$$ $r(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta \log Z(x),~{}\text{where}~{}Z(x)=\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left( \frac{1}{\beta}r(x,y)\right).$ (4) $$

Then, the preference between two responses could be measured using this reward derivation, and $\pi_{\theta}$ is optimized to maximize this preference of $y_{w}$ over $y_{l}$ using the preference dataset $\mathcal{D}$.

$$ $p_{\theta}(y_{w}\succ y_{l}|x)=\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x )}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text {ref}}(y_{l}|x)}\right).$ (5) $$

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta})=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal {D}}\left[-\log p_{\theta}(y_{w}\succ y_{l}|x)\right].$ (6) $$

## 4 Selfee : Boosting Alignment of LLMs with Self-generated Preference

Overview.
In this section, we present Selfee: Self-generated Preference data to align LLMs while mitigating the huge cost for preference dataset construction.
Our main idea is to fully exploit the human prior knowledge within the small (seed) data, and progressively update LLM to improve the alignment.
To be specific, Selfee iterates two steps: (1) data expansion with self-generated preference (Section 4.1) and (2) fine-tuning LLM with self-refined preference learning (Section 4.2).
We present full procedure of Selfee in Algorithm 1
(see Figure 1 for the overview).

Initial stage.
We assume that a small (seed) preference dataset $D_{0}$ and an initial LLM $\pi_{\text{init}}$ are given.
Here, following the common practice [, , ], we use $\pi_{\text{init}}$ which has been supervised fine-tuned (SFT) LLM on the instruction dataset [, ], but not aligned with human preference yet.
Then, we first obtain weakly aligned LLM $\pi_{0}$ by fine-tuning $\pi_{\text{init}}$ on $D_{0}$ using DPO [] (Eq. 6).
We adopt DPO among various preference learning methods due to its simplicity and effectiveness.

Figure: Algorithm 1 Selfee algorithm

### 4.1 Data expansion with self-generated preference data

For the $i$-th iteration ($i=1,\dots)$, we assume that the new prompt set $X_{i}=\{x\}$ is available, i.e., $X_{i}\cap X_{j}=\emptyset$ for all $j=0,\dots,i-1$.(^2^22$X_{0}=\{x|(x,y_{l},y_{w})\in\mathcal{D}_{0}$})
From $X_{i}$, we construct $i$-th artificial preference dataset $\mathcal{D}_{i}=\{(x,y_{l},y_{w})|x\in X_{i}\}$, by using LLM’s intrinsic generation and reward modeling capabilities.
Specifically, for each input prompt $x\in X_{i}$, we sample two responses $y_{1}$ and $y_{2}$ from $\pi_{i-1}$, i.e., $y_{1},y_{2}\sim\pi_{i-1}(x)$ where $\pi_{i-1}$ is the resulting model from the previous iteration.
Then, using the reward captured with $\pi_{i-1}$ and $\pi_{\text{init}}$ (Eq. 4), we measure the preference of $\pi_{i-1}$ between $y_{1}$ and $y_{2}$:

$$ $p_{i-1}(y_{1}\succ y_{2}|x)=\sigma\left(\beta\log\frac{\pi_{i-1}(y_{1}|x)}{\pi _{\text{init}}(y_{1}|x)}-\beta\log\frac{\pi_{i-1}(y_{2}|x)}{\pi_{\text{init}}( y_{2}|x)}\right).$ (7) $$

Then, we directly judge the preference label as below and construct $\mathcal{D}_{i}$ through this:

$$ $(y_{w},y_{l})=(y_{1},y_{2})~{}\text{ if }~{}p_{i-1}(y_{1}\succ y_{2}|x)>0.5~{} \text{ else }~{}(y_{w},y_{l})=(y_{2},y_{1}).$ (8) $$

### 4.2 Self-refinement of generated preference data for effective learning

After the construction of $\mathcal{D}_{i}$, we conduct $i$-th preference learning by fine-tuning $\pi_{\theta}$, which is initialized by $\pi_{i-1}$, using DPO (here, we also use $\pi_{i-1}$ as $\pi_{\text{ref}}$ in Eq. 6).
Learning the self-generated preference data $\mathcal{D}_{i}$ could improve the alignment by effectively spreading the human preference prior from $\mathcal{D}_{0}$ using the power of LLM.
However, it also has a risk of the potential labeling noise which could occur from the distribution shift with $X_{i}$ or insufficient reward modeling with $\pi_{i-1}$.
Therefore, we further propose an improved preference learning method by introducing a novel denoising technique: self-refinement of preference labels with de-coupled noise detection.

Self-refinement of preference label:
Our key intuition is that one can view the derived preference (Eq. 5) can be viewed as the confidence of the currently training LLM $\pi_{\theta}$ for the labels assigned by $\pi_{i-1}$.
Then, $\pi_{\theta}$ would exhibit lower confidence if the given pair of responses is uncertain to answer, indicating a higher probability of labeling noise.
Notably, we also remark that confidence is one of the most popular metrics in the noisy label learning literature [, , ].
Under this intuition, we first identify the $K$% least confident samples:

$$ $z_{\theta}=1~{}\text{ if }~{}p_{\theta}(y_{w}\succ y_{l}|x)<\tau~{}\text{ else }~{}z_{\theta}=0,$ (9) $$

where $\tau$ is the confidence of $K$ percentile sample of $\mathcal{D}_{i}$. Then, with this (potentially) noise identification label $z_{\theta}$, we refine the assigned preference label using label smoothing [], to train $\pi_{\theta}$ less confidently when the risk of label noise is high (i.e., $z_{\theta}=1$):

$$ $\mathcal{L}_{\text{rf}}(\pi_{\theta})=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{ D}_{i}}\left[-\big{(}(1-\alpha\ast z_{\theta})\log p_{\theta}(y_{w}\succ y_{l} |x)+\alpha\ast z_{\theta}\log p_{\theta}(y_{l}\succ y_{w}|x)\big{)}\right],$ (10) $$

where $\alpha$ is a hyper-parameter.
Then, we train $\pi_{\theta}$ using $\mathcal{L}_{\text{rf}}(\pi_{\theta})$ instead of naive DPO (Eq. 6).

De-coupled noise preference detection:
While learning with the refined preference label reduces the risk of learning $\pi_{\theta}$ the noisy preference, its effectiveness could be limited as the model $\pi_{\theta}$ for noise detection originated from the label generation model $\pi_{i-1}$.
Therefore, to further improve the effectiveness of our denoising framework with preference label refinement, we introduce the de-coupled noise detection [, ] technique for LLM alignment.
Specifically, we identify the preference noise by mimicking the preference prediction of a more strongly aligned LLM $\pi_{\widetilde{\theta}}$:
(^3^33With $\lambda$ in Eq. 12, $\pi_{\widetilde{\theta}}$ is equivalent to model trained with (1 + $\lambda$) times smaller KL term than $\pi_{\theta}$ via Eq. 3.)

$$ $z_{\widetilde{\theta}}=1~{}\text{ if }~{}p_{\widetilde{\theta}}(y_{w}\succ y_{ l}|x)<\tau~{}\text{ else }~{}z_{\widetilde{\theta}}=0.$ (11) $$

With this de-coupled identification, $\pi_{\theta}$ is trained with refined preference labels via Eq. 10.
Here, we obtain the prediction of $\pi_{\widetilde{\theta}}$ by approximating its logit $h_{\widetilde{\theta}}$ through the linear combination of the logits of $\pi_{\theta}$ and $\pi_{\text{ref}}$.
(^4^44$\pi_{\theta}(\cdot|x,y_{1:t-1})=\text{Softmax}\big{(}h_{\theta}(x,y_{1:t-1}) \big{)}$)
It is motivated by the recent work [] that shows the aligned models via RLHF with varying $\beta$ are geometric mixtures of a reference model and a single aligned model:

$$ $h_{\widetilde{\theta}}(x,y_{1:t-1})=(1+\lambda)\ast h_{\theta}(x,y_{1:t-1})- \lambda\ast h_{\text{ref}}(x,y_{1:t-1}),$ (12) $$

where $\lambda>0$ is a hyper-parameter and $y_{1:t-1}$ indicates the output sequence before $t$-th output.
We remark that this de-coupled noise identification by approximating $p_{\widetilde{\theta}}(y_{w}\succ y_{l}|x)$ does not require additional computations compared to DPO, since the required measurements $h_{\theta}$ and $h_{\text{ref}}$ are obtained during the calculation of the original DPO objective (Eq. 6).

## 5 Experiments

In this section, we present our experimental results to answer the following question:

- $\circ$
Does Selfee improve the alignment of LLMs only using a small amount of human-labeled preference data? (Table 1, Figure 4)
- $\circ$
Does the proposed method outperform other preference labeling methods? (Table 2, Figure 3)
- $\circ$
Is Selfee generalizable across various LLMs? (Table 4)
- $\circ$
Is Selfee also effective to improve overall LLM’s capability? (Table 6)

### 5.1 Experimental Setups

Models.
When there are no specific mentions, our experiments were conducted using the supervised fine-tuned Mistral-7b-0.1 model [], as the initial model $\pi_{\text{init}}$ in Section 4.
Specifically, we use the open-sourced model(^5^55[alignment-handbook/zephyr-7b-sft-full](alignment-handbook/zephyr-7b-sft-full)) that follows the recipe of zephyr [] and fine-tuned on the instructions of Ultrachat [].
More details are presented in Appendix B.

Baselines.
To evaluate the effectiveness of the proposed preference judgment method (Eq. 7), we compare it with other preference judgment methods.
Specifically, we consider the baselines that train the model via iterative DPO [, ], which iteratively generate preference data and update the model, using LLM-as-judge [, ] (i.e., in-context learning) or an external powerful reward model (PairRM []) for the preference judgment.
Notably, these approaches are the same in the case of changing the judgment method and removing self-refinement in Selfee.
Details are presented in Appendix B.

Datasets.
For the preference learning, we utilized UltraFeedback dataset [], following the previous works [, ].(^6^66["argilla/ultrafeedback-binarized-preferences-cleaned"](%22argilla/ultrafeedback-binarized-preferences-cleaned%22))
To be specific, from this dataset, we first construct the seed data, consisting of 2K samples with prompts, responses, and ground truth preference labels.
Then, the remaining samples are divided into subsets of 8K, 20K, and 30K samples, leaving only the prompts.
These subsets were used as the prompt sets for the iteration stages 1, 2, and 3, respectively.

Evaluations.
Following the common practice, we mainly evaluate each model our evaluations using AlpacaEval2.0 [, , ].
AlpacaEval2.0 approximately evaluates human preference for instruction following.
Using 805 instructions from various datasets, the evaluation is conducted by comparing the response of GPT-4 [] and the testing model to measure win rates.
To mitigate the length bias of LLM’s preference [, ], both vanilla and length-controlled win rates are simultaneously measured.

Implementation details.
After the initialization stage, we conduct three rounds of data expansion with self-generated preference data.
For data expansion, we sampled 2 responses independently per each prompt with a temperature of 0.7.
Then, using the SFT model as the reference model, we assign the preference label (Eq. 7).
The initial DPO training to obtain $\pi_{0}$ was conducted for 3 epochs on the seed dataset.
Training on each subsequent iteration was carried out for 1 epoch.
For the hyper-parameter $\beta$ of DPO, we used a fixed value of $\beta=0.1$.
The batch size was set to 32, and the learning rate was $5\times 10^{-7}$.
We employed adamw_torch optimizer and a cosine learning rate scheduler with a warm-up phase corresponding to 10% of the total training steps.
For the hyper-parameters $\alpha$ and $K\%$ for Selfee, we used fixed values of $\alpha=0.1$ and $K=10$.
Additionally, a warm-up phase was included in the denoising stage, with denoising activated after 20% of the total training steps had been completed.
Regarding the hyper-parameters $\lambda$ for de-coupled noise detection, we utilized the progressively reduced values of 1/2, 1/4, and 1/8 for iterations 1, 2, and 3, respectively.

Table: Table 1: Main results. Evaluation results on AlpacaEval2.0 with different variants of Mistral-7B-v0.1. Here, the human label (%) indicates the proportion of used human preference data.

Table: Table 2: Comparison between judgment methods. Evaluation results on AlpacaEval2.0 with iteratively trained models (from SFT model) under different preference judgment methods.

### 5.2 Main results

After completing 3 iterations of data expansion and fine-tuning, our model achieved a 21.13% win rate against GPT-4 on the AlpacaEval2.0 benchmark, as presented in Table 1.
This represents a significant improvement compared to the 7.68% (7.68% $\rightarrow$ 21.13%) win rate achieved when using only 3.3% of labeled data with the standard DPO training, while the length-control win rate is also improved. (9.03% $\rightarrow$ 15.39%).
Comparing our model to the Zephyr-7b series, which uses the same base model (Mistral-7B-0.1v) and is trained on the same SFT dataset and the entire UltraFeedback preference dataset, our framework achieved superior performance in both win rate (11% vs 21.13%) and length-control win rate (13.2% vs 15.39%).
However, at the same time, it is noteworthy that the average length of model outputs increased from 1802 to 2884 characters through the iterations.
While such length increases are commonly observed in DPO-based algorithms and likely contributed to the higher win rate, the significant improvements in length-control win rate affirm the overall enhancement in performance from Selfee.

Figure: Figure 3: Improvements during iterations. Length control (LC.) win rate (%) measured by AlpacaEval2.0 is consistently improved by Selfee and it outperforms other baselines.
Refer to caption: x3.png

Next, in Table 2, we present additional experimental results to validate the proposed preference judgment method.
For a fairer comparison, we provide the result of the variant of Selfee (Selfee (no refine)), which removes the self-refinement procedure (in Section 4.2) and makes it as Iterative DPO with preference judgment via Eq. 7.
Consequently, three experiments in Table 2 become iterative DPO variants with different preference judgment methods.
Here, one can observe that our approach showed significantly better performance compared to other methods.
Specifically, Selfee achieved a win rate of 19.91% against GPT-4 on AlpacaEval2.0, compared to 9.46% for the baseline with an external reward model, PairRM.
In terms of length control win rate, Selfee achieved 14.41%, surpassing the reward model’s 11.84%.
On the other hand, the in-context learning approach (LLM-as-judge) shows a similar win rate compared to PairRM, but falls short in length control win rate (11.87% vs 9.28%), showing the limitations of the LLM-as-judge approach.
Overall, the results reveal the superiority of our direct preference judgment over other judgment methods.
This superiority is consistently observed through the iterations, as shown in Figure 3.

Table: Table 3: Ablation study. Evaluation results on AlpacaEval2.0 with iteratively trained models (from SFT) under different methodological configurations of Selfee. DE, SR, DND are abbreviations of data expansion, self-refinement, and de-coupled noise detection, respectively.

### 5.3 More analyses

Ablation study.
To evaluate the impact of the self-refinement components, we conducted ablation experiments by excluding both self-refinement (SR) and decoupled noise detection (DND) from the existing framework.
The results are presented in Table 3.
With self-refinement without decoupled noise detection (Eq. 10), we observed a slight performance improvement, with the win rate against GPT-4 marginally increasing from 19.91% to 19.94%, and the length control win rate rising from 14.41% to 14.7%.
But, when the decoupled noise detection is incorporated into the self-refinement (Eq. 11), we observed significant improvements, with the win rate increasing from 19.91% to 21.13% and the length control win rate improving from 14.41% to 15.39%.
Furthermore, the inclusion of the self-refinement components led to a reduction in the average response length, decreasing from 3204 to 3043 and finally to 2884 characters (i.e., improvement in length bias).
Also, these results confirm that the self-refinement component is a crucial factor in enhancing performance, contributing to both higher win rates and better length control.

Compatibility with different models.
Next, to verify the compatibility of our framework across different models, we conducted experiments using the Phi-2 model [].(^7^77[https://huggingface.co/microsoft/phi-2](https://huggingface.co/microsoft/phi-2))
Specifically, we conducted experiments based on the supervised fine-tuned Phi-2(^8^88[lole25/phi-2-sft-ultrachat-full](lole25/phi-2-sft-ultrachat-full)), which, like Mistral, has been supervised fine-tuned on the UltraChat dataset.
While maintaining the core structure of our framework, we made slight adjustments to the learning rate to accommodate the different characteristics of the Phi-2 ($5\times 10^{-6}$).
In addition, due to the smaller size of the Phi-2, we observed that performance improvements were not evident beyond iteration 2. Therefore, we present the results of iteration 1.
As shown in Table 4, the experimental results showed that applying Selfee to Phi-2 yields a significant increase in performance.
Specifically, the win rate improved from 5.67% to 9.43%, and the length control win rate increased from 7.02% to 9.1%, compared to Phi-2 trained with DPO.
These results demonstrate that Selfee is effective across different models, although its impact may be limited when applied to smaller models.

Figure: Figure 4: Improvements without seed data. Evaluation results on AlpacaEval2.0 with Mistral-7B-instruct-v0.1 and Selfee without seed preference dataset.
Refer to caption: x4.png

Dependency on seed data.
In addition, we evaluated the feasibility of using our method even without seed preference data.
Namely, we want to answer whether LLM can derive explicit human preference between responses, by leveraging their intrinsic knowledge learned about humans, during the previous training, such as pre-training or supervised instruction tuning (SFT).
For this experiment, we used the Mistral-7b-instruct-0.1v [] as the initial model (i.e., $\pi_{0}$) and the Mistral-7b-0.1v-base as the reference model (i.e., $\pi_{\text{init}}$) (see the initial setup in Section 4).
This setup allows us to demonstrate that our framework can function effectively even in the absence of seed preference data, when the model is sufficiently fine-tuned with iterative data expansion and learning through self-refinement.
As shown in Figure 4, the win rate increased from 6.31% to 9.79%, and the length-control win rate improved from 10.14% to 11.59%, after 3 iterations.
Notably, unlike experiments with seed preference data, there was no significant increase in the average length of model outputs (1077 to 1461 characters).
This indicates that Selfee can effectively leverage internal information to be aligned with human preference even without seed data.

Table: Table 4: Compatibility across various LLMs. Evaluation results on AlpacaEval2.0 with different variants fine-tuned from Phi-2.

Table: Table 5: Additional analyses with Selfee. Evaluation results on AlpacaEval2.0 with models that fine-tuned with different judgment methods, from the resulting model of 1st iteration of Selfee.

Additional analysis with judgment methods.
In Table 5, we further analyzed the impact of the reference model in the preference judgment process in Eq. 7.
This analysis was conducted during the transition from iteration 1 to iteration 2, where the most significant performance changes were observed (i.e., we fine-tune from the resulting model of iteration 1).
To isolate and compare the effect of judgment methods, we followed the setup in Table 2 and so excluded the influence of the self-refinement component.
Then, we experimented with three setups by varying the judgment method using (1) the current policy from the previous iteration as the reference model, (2) performing judgment without any reference model, and (3) using the PairRM for judgment.

The results are presented in Table 5.
Here, the experimental results demonstrated that the method used in Selfee, where the SFT model was utilized as the reference model for preference judgment, achieved the highest performance increase.
Specifically, using the model from the previous iteration as the reference model showed lower performance, with a relatively larger decrease in the length control win rate (15.08% vs 13.73%) compared to the win rate (19.56% vs 17.66%).
Despite these decreases, it still outperforms using PairRM.
These results may imply the importance of judging the preference through the training LLM rather than the external model, as it is less suffering from the distribution mismatch.
However, without reference model (i.e., only using the likelihood of the current model), the performance increase was the lowest compared to all other cases.
These findings underscore the substantial impact of the choice of proper judgment method and reference model.

Evaluation on MT-Bench.
We further evaluate trained models using MT-Bench [] to assess different aspects of LLMs.
Namely, MT-Bench evaluates a chatbot’s overall abilities across multiple categories related to key LLM capabilities such as math, coding, roleplay, writing, etc.
The evaluation is conducted by scoring responses to multi-turn questions using GPT-4.
As shown in Table 6, Selfee achieved a score of 6.94 on the MT-Bench, outperforming other baselines and the model trained with DPO on 3.3% gold labeling data. In addition, when we refine the preference using relatively less aligned predictions (i.e., $\lambda=0$), denoted by Selfee^∗, it even outperforms the strong baseline with the external reward model.
Moreover, we observe that Selfee consistently improves the MT-bench performance without seed preference data or with smaller LLM, as presented in Table 7 in Appendix C.
These results indicate that Selfee not only excels in specific tasks but also contributes to the overall performance enhancement of LLMs.

Table: Table 6: MT-Bench. Evaluation results on MT-bench with different variants of Mistral-7B-v0.1.

## 6 Conclusion

In this paper, we proposed Selfee, a method that can efficiently improve the alignment of LLMs using minimal human-labeled preference data.
Our main contributions include the development of an effective data expansion method with the direct preference judgment method and preference learning algorithm with the self-refinement of (potentially) noise preference.
We demonstrate the effectiveness of Selfee by fine-tuning the recent LLMs with the various setups, and observing the significant improvements when evaluating them on the commonly used benchmarks.

Limitation and future work.
In the experiments, Selfee has shown the tendency to increase the responses’ length.
We demonstrated that the improvement by Selfee is not a simple result of such length increase, by observing the increase of win rate under a length length-controlled setup or MT-bench.
However, depending on the user, this behavior could be dispreferred.
In this sense, focusing on mitigating this bias during the self-improving alignment will be an interesting future direction, and can enhance the robustness and generalizability of Selfee across more diverse scenarios.

## Appendix A Societal Impact

Selfee enables efficient human preference learning, allowing for cost-effective training of models in data-scarce or domain-specific areas.
Our framework supports alignment learning in various fields, including multilingual language learning and preferences beyond human helpfulness.
Consequently, it could contribute to facilitating the widespread adoption of LLM technology across diverse sectors.
By lowering the barriers to alignment learning, Selfee makes it more accessible to a broader audience.
However, the widespread availability of this technology also brings potential risks.
The reduced cost of training models could enable malicious actors to misuse the technology, leading to societal issues.
Therefore, it is crucial to implement ethical considerations and safety measures when deploying Selfee technology to mitigate these risks.

## Appendix B More Details of Experimental Setups

### B.1 SFT model setup

Mistral. For supervised fine-tuning, Ultrachat dataset [] is used(^9^99[https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)), batch size was set 128, total epoch was 1, and the learning rate was $2\times 10^{-5}$.
It employed Adam optimizer [] and a cosine learning rate scheduler with a warm-up phase corresponding to 10% of the total training steps.

Phi-2. For supervised fine-tuning, Ultrachat dataset is used, batch size was set 64, total epoch was 3, and the learning rate was $2\times 10^{-5}$.
It employed Adam optimizer and a cosine learning rate scheduler with a warm-up phase corresponding to 10% of the total training steps.

### B.2 Baselines experiment setup

Zephr-7b-$\beta$. Zephr-7b-$\beta$ [], which is compared in Table 1, was trained using the same pre-trained model (mistral-7b-0.1v []) and the same SFT data (Ultrachat []), but there are marginal differences compared to our setup.
Specifically, Zephr-7b-$\beta$ used the batch size of 512, but 128 was used for the model in our experiments.
As a result, the SFT model of Zephr-7b-$\beta$(^10^1010[https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta](https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta)), according to AlpacaEval2.0, showed a length control win rate of 8.14% and a win rate of 5.26%, approximately 0.5% better than the SFT model used in our experiments (8.14% vs 7.58%, 5.26% vs 4.72%).
In addition, regarding the preference dataset, Zephr-7b-$\beta$ was trained using 100% of the original Ultrafeedback [] (^11^1111[https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)) but we use 3.3% of its cleaned version(^12^1212[https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)).
Consequently, the results under the two datasets might be slightly different, and the impact of these method differences has been specifically denoted in []; here, Ultrafeedback clean shows approximately a 0.9% improvement in performance based on the AlpacaEval2.0 win rate (from 11.33% to 12.2%).
Consequently, although Zephr-7b-$\beta$ is not a perfectly strict comparison, we believe the minor variations are relatively small compared to the differences between Selfee and Zephr-7b-$\beta$, and hence the comparison results are meaningful.

LLM-as-Judgement.
For LLM-as-judge, we used an SFT model to employ Consitual AI’s pairwise comparison prompt for judging preferences [].
Preference is measured by comparing the logprob value of the token output as input to the following prompt (Listing 7).
To ensure fair comparison and prevent low judgment performance, evaluation instructions were created using seed preference data which is the same form as Consitual AI’s pairwise comparison. (Listing 7)
Using these, additional SFT learning is performed to obtain an independent LLM-as-judge model.
For this supervised fine-tuning, we set the batch size 32, total epoch is 3, and the learning rate was $2\times 10^{-5}$.
We employed Adam optimizer and a cosine learning rate scheduler with a warm-up phase corresponding to 10% of the total training steps.

Reward model judgment.
For the reward model baseline, we selected PairRM [] due to its high performance on AlpacaEval2.0 [, ]. Unlike SELFEE, which was trained on only 2K gold label data, PairRM was trained on a large-scale dataset. The training data for PairRM includes the following:

- •
openai/summarize_from_feedback []
- •
openai/webgpt_comparisons []
- •
Dahoas/synthetic-instruct-gptj-pairwise(^13^1313[https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise](https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise))
- •
Anthropic/hh-rlhf []
- •
lmsys/chatbot_arena_conversations []
- •
openbmb/UltraFeedback []

The total number of pairwise samples in this training data is approximately 500K, compared to 2K for SELFEE. Specifically, the summarize_from_feedback dataset contributes 179K samples, and the hh-rlhf dataset contributes 161K samples, making up a significant portion of the total.

### B.3 Implementation details

Resources and computation cost.
For all experiments, we utilized 4 A6000 GPUs.
Under this computational resource, generating responses for 10K prompts takes approximately 1 to 2 hour, and preference judging for generated responses also takes about 1 to 2 hour.
For training of model with Eq. 10, it takes about 1 to 2 hours per epoch.
Therefore, the total time required to complete response generation, preference judgment, and one epoch of training was between 5 to 6 hours per 10K prompt.

Response generation.
To mitigate the length bias from training with Direct Policy Optimization (DPO), we restricted the maximum token length for self-generated responses to 300 tokens.
This limit corresponds to approximately 900 characters.

Refinement. Using the simple moving average (SMA) method, we measured the confidence over the most recent 50 samples.
Based on this, we calculated the boundary corresponding to the lower K% of the confidence distribution (Eq. 11).

Table: Table 7: MT-Bench. Evaluation results on MT-bench with different models. Selfeeinst and Selfeephi denotes the fine-tuned Mistral-7B-Instruct-v0.1 and Phi-2 SFT with Selfee, respectively.

[!ht]
{minted}[fontsize=, frame=single, breaklines]python
f”’
Consider the following conversation between a human and an assistant:
[prompt]
Please choose the response that is the most helpful, honest, and harmless.
Options:
(A) [response 1]
(B) [response 2]
The answer is:
”’
Prompt used for judging preferences.
{listing}[!ht]
{minted}[fontsize=, frame=single, breaklines]python
f”’
Consider the following conversation between a human and an assistant:
[Horses are hybrids of which two animals?]
Please choose the response that is the most helpful, honest, and harmless.
Options:
(A) [Horses, scientifically known as Equus caballus, are not the result of hybridization between two other animals. They are considered a species of their own within the Equidae family, which also includes other equines such as donkeys, mules, and zebras. Horses are believed to have evolved from a small, multi-toed forest-dwelling animal known as Eohippus, which lived around 60 million years ago. Over millions of years, these horses evolved into the large, single-toed animals we know today.]
(B) [Horses are not hybrids of any other animals. They are a unique species and are not a product of cross-breeding between different animals. Horses are domesticated animals and have been bred and developed over thousands of years by humans for various purposes such as transportation, riding, and agriculture.]
The answer is: (A)

”’
Evaluation instruction.

## Appendix C More Evaluation Results in MT-bench

In this section, we further provide the additional evaluation results on MT-bench [], which couldn’t be presented in the main text, due to the space issue.

Figure: ((a)) MT-bench task-wise evaluation results
Refer to caption: x5.png

We first present (a) task-wise evaluation results and (b) iteration-wise average improvement in Figure 5.
As shown in Figure 5(a), Selfee consistently improves the performance in various tasks.
Notably, there is almost no gain in Coding and degradation in Math.
We remark that this phenomenon is commonly observed in the relevant literature [], which indicates that different training [] or inference [] schemes might be necessary to improve the performance in these tasks.

Next, in Figure 5(b), one can observe that the average performance on the MT-bench is increased with more iterations.
Specifically, while the iterative DPO using PairRM shows the best performance until iteration 2, Selfee^∗ outperforms it in iteration 3.
It demonstrates the effectiveness of our framework for iteratively improving the alignment of LLM.

Lastly, we present additional evaluation results on MT-bench, not presented in Table 6.
Specifically, we measure the performances of Phi-2 variants and Mistral-7B-Instruct-v0.1 variants, which presented in Table 4 and Figure 4.
The results are presented in Table 7.
As one can see, Selfee consistently yields the improvement across different backbones of Mistral-7B-Instruct-v0.1 and Phi-2.

## Appendix D Qualitative Examples

In this section, we present examples of the responses from the models trained under different methods (SFT, DPO, and Selfee), for a qualitative comparison.
The results are presented in Figures
6, 7, and 8.
As one can see in the examples, the model trained under Selfee provides more detailed and informative responses to various questions.

Figure: Figure 6: Response in AlpacaEval2.0

Figure: Figure 7: Response in AlpacaEval2.0

Figure: Figure 8: Response in AlpacaEval2.0

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 10
presentation: 4
questions: Minor Typos
1. "Lengh-control" in Figure 2
2. "additional codes" , L267
soundness: 4
strengths: 1. The paper is well-written and very easy to follow throughout. The paper contextualizes itself within the alignment literature well, covering the fundamentals of pairwise preference learning (Bradley-Terry modeling) to direct alignment algorithms like DPO (Section 3)
2. SPA is able to use only a small seed preference dataset to then directly score preference labels using the implicit reward model learned by DPO (Section 4.1). Since these predictions can be noisy, the authors introduce a novel self-refinement denoising technique using a confidence prediction (eq 9) to smooth the preference label (eq 10)
3. Reproducibility: the authors provide implementation details and hyperparameters in Section 5 (*L311-321*). The authors will open-source the code and models after acceptance, which is appreciated. Lastly, because the modification to the DPO objective is minimal, the authors mention only a few lines of change to the DPO codebase, which is another advantage for practical utility of SPA
4. The authors compare to popular categories of baselines: iterative DPO methods, LLM-as-judge methods, and explicit reward-modeling + RLHF methods (*L288 - L291*) and achieve strong results on AlpacaEval 2.0 and MT-Bench
5. The authors show SPA extends beyond Mistral to other popular LLMs like Phi and Llama (Table 5) and is robust, in the win rate variance sense, to the seed of the initial preference data (Table 4)
summary: This work proposes SPA, a framework to lower the high costs of collecting large preference datasets for alignment. SPA uses an LLM's logits to generate pairwise preference data, without reward model learning or in-context learning, which is then used for preference learning for LLMs. The authors show the practical usefulness of SPA by aligning mistral 7B on 3.3% of Ultrafeedback preference data to achieve strong performance compared to state-of-the-art methods on AlpacaEval 2.0 and MT-Bench
weaknesses: No major weaknesses, mainly minor clarifications:
1. Can the authors provide a little more description about the "length control" aspect of AlpacaEval 2.0 in the main paper? This setting is used in nearly all results, but is not explained clearly where first introduced (Section 5.2)
2. What is "gold label" (Table 1, 5)? Is this the Ultrafeedback preference data? Please make this explicit in the writeup

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the model used for the 'LLM-as-judge' method in Table 2 ? Have you tried using ${\pi}_{i-1}$ in this baseline?
2. There are lines out of the border in references and appendix.
3. (corresponding to weakness 2) Why this method can work with only 3.3% of total data? Does this method elicit the latent human preference knowledge from pre-trained model, or the 3.3% subset is already enough to define the human preference in UltraFeedback dataset? (Note that this is not a fatal question, feel free to provide any discussion or hypothesis).
soundness: 3
strengths: 1. This work inherits the idea of self-rewarding, but leverages the inherent preference of current aligned model in an intuitive way, which sounds novel to me.
2. The reduction on data usage seems promising, and the performance is robust.
summary: This paper introduces an efficient LLM alignment method, namely Spread Preference Annotation, aiming at reducing the demand for human-labeled preference data. By leveraging the inherent preference of current aligned model, this work generates new preference data points and conducts alignment iteratively. To reduce the potential noise caused by distribution shift, this work incorporates a self-refinement mechanism on preference labels, where this approximate a more strongly aligned model to better identify noise through a linearly extrapolated prediction method. Through experiments this paper proves that SPA can achieve better alignment performance with much less data.
weaknesses: 1. 'De-coupled noise preference detection' is not stated clearly enough in section 4.2. Based on my understanding, $z_{\tilde{\theta}}$ is used to substitute for $z_{\theta}$ in the 'Self-refinement' part, which is also supported in Algorithm 1. If I am correct, I think it would be easier for readers to understand if the final usage of the approximated logits and labels are stated in the main text.
2. Lacks some explanatory discussion on why this method can work on such a small subsets and even perform better than DPO with the full dataset (Details in questions part).

If the author can address my concern in weakness/questions and provide some insightful discussion, I am willing to raise my score.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. How do you choose the initial seed data? Are the data randomly chosen as seeds? Are there differences in human preferences across data? How do you handle such differences?
2. In section 4.1, can you elaborate on the sampling process used to generate the responses y1 and y2? In particular, does the model use specific diversity techniques to ensure diversity of reactions, or does it rely purely on randomness in the generation process?
3. How does it ensure the reliability of the expanded data?
soundness: 4
strengths: 1. The overall idea is relatively simple, but the method achieves good performance and has great potential with limited human labeling.
2. The paper is well-structured and presents a rigorous methodology, with comprehensive experimental validation that supports the claims made about SPA’s effectiveness.
summary: This paper proposes SPA, an approach to enhance the alignment performance of large language models by using minimal human-annotated preference data.
Introduce a confidence-based refinement of preference labels to reduce the risk of noise in preference learning with generated data.
It is experimentally verified that a tiny percentage of preference data (3.3%) achieves results comparable to or exceeding those obtained using the entire data and the existing optimal baseline method in the AlpacaEval 2.0 evaluation.
weaknesses: This is a very solid piece of work. The proposed method is simple yet effective. I don't have any particular concerns or issues with it.

Meta-review: This work presents a novel framework, called Spread Preference Annotation with direct preference judgment (SPA), aimed at reducing the high costs associated with collecting large preference datasets for alignment. Overall, all reviewers agreed that this work is novel and important, and they gave very high scores to it. I believe this approach is both simple and effective and makes clearly contribution to LLM Alignment. The achieved results are promising, for example, with only 3.3% of the ground-truth labels this method still achieved superior alignment performance. Based on these, I would like to suggest accept this work.

Decision: Accept (Oral)