Progressively Label Enhancement for Large Language Model Alignment

## Abstract

Abstract Large Language Models (LLM) alignment aims to prevent models from producing content that misaligns with human expectations, which can lead to ethical and legal concerns.
In the last few years, Reinforcement Learning from Human Feedback (RLHF) has been the most prominent method for achieving alignment.
Due to challenges in stability and scalability with RLHF stages, which arise from the complex interactions between multiple models, researchers are exploring alternative methods to achieve effects comparable to those of RLHF.
However, these methods often rely on large high-quality datasets.
Despite some methods considering the generation of additional data to expand datasets, they often treat model training and data generation as separate and static processes, overlooking the fact that these processes are highly interdependent, leading to inefficient utilization of the generated data.
To deal with this problem, we propose Ple , i.e., Progressively Label Enhancement for LLM Alignment, a framework that dynamically adjusts the model’s training process based on the evolving quality of the generated data.
Specifically, we prompt the model to generate responses for both the original query and the query guided by a set of carefully designed principles, and then utilize a dynamic threshold to determine the appropriate training approach for both responses based on their corresponding reward scores.
Experimental results demonstrate the effectiveness of Ple compared to existing LLM alignment methods.

## 1 Introduction

Large Language Models, such as the LLama series (Touvron et al., ) and OpenAI’s GPT series (Floridi & Chiriatti, ; OpenAI, ), have demonstrated their powerful capabilities across various language tasks, including translation (Zhang et al., ), summarization (Pilault et al., ), and conversational interaction (Wang et al., ). In certain scenarios, they have even exhibited performance that matches that of human experts (Ouyang et al., ).

However, these language models may not always generate responses as expected by humans and can even produce content that violates human ethics or legal boundaries (Bai et al., ; Askell et al., ). Therefore, it is crucial for researchers to explore the limitations of these models and implement restrictions on output generation to ensure safety and compliance, a process known as AI alignment.

The most prominent method for achieving AI alignment is Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., ; Stiennon et al., ; Ouyang et al., ). RLHF employs Supervised Fine-Tuning (SFT) to guide models using human instructions (Wang et al., ; Taori et al., ), followed by training a reward model on human-rated outputs (Ouyang et al., ), and optimizing the model with Reinforcement Learning (RL) algorithms like Proximal Policy Optimization (PPO) (Schulman et al., ; Askell et al., ; Bai et al., ). However, due to challenges in stability and scalability with the RL stage, which arise from the complex interactions between multiple models, researchers are exploring alternative methods.
For instance, LIMA (Zhou et al., ) has experimentally demonstrated that when the pre-trained model’s capabilities are sufficiently strong and the quality of the SFT data is high, it can achieve results comparable to those of RLHF. RAFT (Dong et al., ) expands the SFT dataset by generating additional samples and selecting those with high reward scores to enhance the SFT dataset. RRHF (Yuan et al., ) simplifies the RLHF process by integrating the subsequent RL steps into the SFT phase as a regularization term.

However, these methods are highly dependent on large amounts of high-quality data, which is impractical in certain applications, such as the medical field (Yang et al., ; Li et al., ) or chip design (Liu et al., ). Additionally, even though some methods generate extra data to expand the training set to alleviate the problem.
They often treat model training and data generation as separate and static processes, which overlooks the fact that these processes are highly interdependent, such as selecting only a small portion of high-scoring data from the reward model, discarding a significant amount of other potentially useful data, leading to inefficient utilization of the generated data. Therefore, we consider designing an efficient framework that couples the data generation and model training processes, allowing them to work synergistically, thus ensuring that all generated data, including potentially useful lower-scoring data, is effectively utilized, thereby improving training efficiency.

Motivated by the above consideration, we propose a novel framework named Ple, i.e., Progressively Label Enhancement for Language Model Alignment. Specifically, during the sample generation phase, we design a set of principles to guide the model to output according to human expectations. When the reward score difference between the principle-guided output and the response to the original query exceeds a dynamically updated threshold, indicating a significant improvement under the principle guiding, the model is encouraged to align its output with the better response and move away from the poorer one. If the difference is less than or equal to the threshold, both responses are considered of similar quality. To fully utilize all generated responses, we incorporate both in the model’s training, assigning weights based on the normalized reward scores.
Our contributions can be summarized as follows:

- •
Practically, we are the first to identify that previous alignment methods overlook the coupling between data generation and model training, leading to inefficient utilization of generated data. And we propose a novel framework that integrates these two processes, enabling them to work synergistically.
- •
Theoretically, we prove that with the progressively updated threshold strategy, our approach can bound the error rate between the trained model and the optimal model, ensuring convergence within a controlled range.

Extensive experimental results validate the effectiveness of our methods over several existing language model alignment approaches.

## 2 Related Work

The alignment of language models refers to the process of ensuring that the models behave in ways that are consistent with human values, ethical principles, and intended purposes (Leike et al., ). The most prominent and effective method currently used to achieve this alignment is Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., ; Stiennon et al., ; Ouyang et al., ). The framework of RLHF first employs Supervised Fine-Tuning (SFT) to guide the model to follow human instructions in an imitative manner (Wang et al., ; Taori et al., ). The next steps involve training a Reward Model on a dataset reflecting human preferences, created from human evaluators’ ratings of the SFT model’s outputs (Ouyang et al., ). Using reinforcement learning algorithms like Proximal Policy Optimization (PPO) (Schulman et al., ), the SFT model is further optimized by continuously generating outputs, receiving evaluations from the Reward Model, and updating its parameters to maximize alignment with the Reward Model (Askell et al., ; Bai et al., ).

However, due to the challenges of stability and scalability involved in the interactions between multiple models in RLHF, researchers have started exploring other more direct and efficient methods for model alignment (Rafailov et al., ; Yuan et al., ; Zhou et al., ; Dong et al., ). DPO derives an equivalent optimization objective from RLHF, allowing the model to be directly optimized using preference data without the need to train a separate reward model (Rafailov et al., ). Similarly, RRHF incorporates the steps of RLHF into the SFT stage by introducing a regularization term, which encourages the model to generate preferred responses with higher probability and poor responses with lower probability (Yuan et al., ). LIMA has experimentally demonstrated that when the pre-trained model is sufficiently good, only a small amount of high-quality data is needed. By using only SFT, it is possible to obtain a well-aligned model without the need for the subsequent complex RLHF steps (Zhou et al., ). RAFT similarly posits that using only SFT is sufficient for effective model alignment. They expanded the SFT training set by sampling a batch of high-scoring data based on the scores from the reward model (Dong et al., ).

## 3 Preliminaries

We first introduce the formal notation for the language model alignment problem. Let $\mathcal{V}$ be a vocabulary of a language model. The goal of alignment is to ensure that the language model $\pi:\mathcal{X}\rightarrow\mathcal{Y}$ generates response $\bm{y}\in\mathcal{Y}$ that is consistent with human values and preferences given a query $\bm{x}\in\mathcal{X}$, where the query $\bm{x}=[x^{1},x^{2},\dots,x^{m}]$ and response $\bm{y}=[y^{1},y^{2},\dots,y^{n}]$ are sequences of tokens, the input space $\mathcal{X}=\mathcal{V}^{m}$ and the output space $\mathcal{Y}=\mathcal{V}^{n}$.

The alignment process typically begins with Supervised Fine-Tuning (SFT) stage, which adjusts the language model using Maximum Likelihood Estimation on a human-labeled high-quality dataset $\mathcal{D}_{\text{sft}}=\{(\bm{x}_{i},\bm{y}_{i})\}_{i=1}^{N}$:

$$ $\displaystyle\mathcal{L}_{\text{sft}}=-\sum_{i=1}^{N}\sum_{j=1}^{n_{i}}\log P( y_{i}^{j}|[y_{i}^{k}]^{k<j},\bm{x}_{i};\theta),$ (1) $$

where $N$ is the number of training examples, $n_{i}$ is the length of the $i$-th target sequence, and $\theta$ represents the parameters of the language model $\pi_{\theta}$.

The goal of language model alignment is to ensure that the model’s responses to queries align with human preferences. These preferences are typically captured by a reward model $R:(\mathcal{X},\mathcal{Y})\rightarrow\mathbb{R}$, where higher scores indicate that responses better align with human values and preferences. Conversely, lower scores indicate less alignment. An ideal model maximizes the expected reward:

$$ $\displaystyle\pi^{\star}=\arg\max_{\pi}\mathbb{E}_{\bm{x}\sim p(\bm{x}),\bm{y} \sim\pi(\cdot|\bm{x})}[R(\bm{x},\bm{y})],$ (2) $$

where $\pi^{\star}$ represents the optimal policy that maximizes the expected reward according to the reward model $R$.

## 4 The Proposed Method

Figure: (a)
Refer to caption: x1.png

Figure: Algorithm 1 The Ple Algorithm

In this section, we present our novel framework named Ple, i.e., Selective Label Enhancement for Language Model Alignment. As illustrated in Figure 1, during the sample generation phase, we use carefully crafted principles to guide the model’s outputs. When the reward score difference between the principle-guided output and the original response exceeds a dynamically updated threshold, the model is encouraged to align with the better response and move away from the poorer one. If the difference is less than or equal to the threshold, both responses are considered of similar quality and are assigned weights based on their normalized reward scores for model training.

### 4.1 Ple

Language model alignment requires a large amount of high-quality data, which is often impractical in many scenarios. Therefore, we consider generating additional data during training to expand the dataset. Motivated by the self-align approach (Wang et al., ; Sun et al., ), we design a set of principles to guide the model in generating responses that align closely with human preferences:

Your answers should follow these rules:
1 ethical: You should actively refrain users on illegal, immoral, or harmful topics.
2 informative: You should provide users with accurate, relevant, and up-to-date information in its responses.
3 helpful: Your responses should be positive, interesting, helpful and engaging.
…
which is denoted as $\bm{p}=[p^{1},\cdots,p^{n_{p}}]$, where $n_{p}$ is the token length of the principle prompt. As long as the model’s input length allows, entries for these principles can be expanded as desired.

Let $\pi^{\text{sft}}_{\theta}$ be the SFT-aligned model optimized by Eq. (1) and we use it as the initial model. During training, for each query $\bm{x}\in\mathcal{D}_{\text{query}}=\{\bm{x}_{i}\}_{i=1}^{N_{q}}$, where $N_{q}$ is the number of queries, the model samples a response $\bm{y}\sim\pi_{\theta}(\cdot|\bm{x})$. In addition, the principle-guided model then samples a response: $\bm{y}^{\text{prompt}}\sim\pi_{\theta}(\cdot|[\bm{p},\bm{x}])$
based on the set of principles designed to expect ethical, informative, and helpful output. The reward model $R$ assigns the scores $s=R(\bm{x},\bm{y})$ and $s^{\text{prompt}}=R(\bm{x},\bm{y}^{\text{prompt}})$.

When the difference between the reward scores, $s^{\text{prompt}}-s$, exceeds a threshold $\tau$, we consider that the current model has generated a better response based on the principles compared to the original response. Therefore, to encourage the model to generate responses closer to the better response and away from the poorer response for the given input $\bm{x}$, we adopt a ranking loss. This ranking loss aims to adjust the model’s parameters so that the likelihood of generating the better response is increased while the likelihood of generating the poorer response is decreased. The formula is as follows:

$$ $\displaystyle\mathcal{L}_{\text{rank}}=-\sum_{{s^{\text{prompt}}-s>\tau}}\pi_{ \theta}(\bm{y}^{\text{prompt}}|\bm{x})-\pi_{\theta}(\bm{y}|\bm{x}).$ (3) $$

When the difference between the reward scores is less than or equal to the threshold $\tau$, we consider that the response generated by the principle-guided model and the original response are of similar quality. Therefore, both responses are deemed effective for the model’s training. We include both responses in the dataset for subsequent training, with their weights determined by the magnitude of their scores, a process known as label enhancement (Xu et al., ; ). The formula for this process is as follows:

$$ $\displaystyle\mathcal{L}_{\text{weighted-sft}}=-\sum_{s^{\text{prompt}}-s\leq \tau}w\cdot\pi_{\theta}(\bm{y}|\bm{x})+w^{\text{prompt}}\cdot\pi_{\theta}(\bm{ y}^{\text{prompt}}|\bm{x}),$ (4) $$

where the weights $w$ and $w^{\text{prompt}}$ are calculated as follows to normalize them to the range $[-1,1]$:

$$ $\displaystyle w=\frac{2e^{s}}{e^{s}+e^{s^{\text{prompt}}}}-1,\quad w^{\text{ prompt}}=\frac{2e^{s^{\text{prompt}}}}{e^{s}+e^{s^{\text{prompt}}}}-1.$ (5) $$

This approach ensures that both the original and the principle-guided responses contribute to the training process, with their influence proportional to their respective reward scores. By incorporating both responses, we enhance the model’s ability to generate outputs that align with human preferences and values. Then the final objective function is:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{\text{rank}}+\mathcal{L}_{\text{weighted -sft}}$ (6) $$

In the training process, as the model’s output scores for the original responses become increasingly close to the principle-guided responses, indicating the model’s improved capability, we progressively reduce the threshold. This allows the loss function to adapt to these smaller variations.
Here’s how the threshold adjustment can be expressed:

$$ $\displaystyle\tau_{t}=\tau_{0}\cdot\alpha,$ (7) $$

where $\tau_{t}$ is the threshold at training step $t$, $\tau_{0}$ is the initial threshold, and $\alpha\in(0,1)$ is a decay factor that progressively reduces the threshold over time.

The whole process of Ple is shown in Algorithm 1.

## 5 Theoretical Analysis

In this section, we will provide a theoretical analysis to demonstrate that Ple, which uses a dynamically updated threshold for data selection and model training, will ultimately converge to the optimal model $\pi^{\star}$ defined in Eq. (2).

Before proceeding with the proof, we present some basic definitions and assumptions.
For two queries $\bm{x}$ and $\bm{z}$ that satisfy $R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},
\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})$, i.e., the margin between the reward score of principle-guided response $R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})$ and that of original response $R(\bm{z},\bm{y}_{\bm{z}})$ is larger than that in point $\bm{x}$, the indicator function $\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})<\pi(\bm{y}_{\bm{z
}}|\bm{z})\}}\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_
{\bm{z}})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})
\Big{]}$ equals 1 if the model’s output probabilities for the responses $\pi(\bm{y}_{\bm{z}}|\bm{z})$ and $\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})$ are inconsistent with the corresponding ranking of their reward scores.
Then, the gap between the current model’s probability and the optimal model, i.e., the approximation error of the model, could be controlled by the inconsistency between the model’s output probabilities and the corresponding ranking of their reward scores for all the queries $\bm{z}$.

Therefore, we assume that there exist constants $\alpha,\epsilon<1$, such that for any $\bm{x}\in\mathcal{X}$ and $\bm{y}\in\mathcal{Y}$,

$$ $\displaystyle|\pi(\bm{y}|\bm{x})-\pi^{\star}(\bm{y}|\bm{x})|\leq\alpha\mathbb{ E}_{(\bm{z},\bm{y},\bm{y}^{\text{prompt}})\sim\mathcal{D}_{\text{train}}}\Big{ [}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})<\pi(\bm{y}_{\bm{z}}| \bm{z})\}}\Big{|}$ (8) $\displaystyle R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z }})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\Big{]} +\frac{\epsilon}{6}.$ $$

In addition, for the probability density function $d(u)$ of the cumulative distribution function of the margin of the reward scores $D(u)=P_{\bm{x}\sim p(\bm{x})}(u(\bm{x})\leq u)$, where $u(\bm{x})=R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})$ denotes the margin of the reward scores for the query $\bm{x}$. We assume that there exists constants $c_{\star},c^{\star}$, such that $c_{\star}<d(u)<c^{\star}$. Then, we define the worst-case density imbalance ratio as $l=\frac{c^{\star}}{c_{\star}}$.

Motivated by the pure level set in traditional machine learning (Zhang et al., ; Xu et al., ), the region where the language model is reliable, i.e., the region where the model’s output probabilities are consistent with the ranking of the reward scores of the principle-guided response and the original response, can be defined as:

###### Definition 5.1 .

Pure ($e$, $\pi$, $R$)-level set: A queries set $L(e,R):=\{\bm{x}|R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{
\bm{x}})\geq e\}$ is pure for the model $\pi$, if for any $\bm{x}\in L(e,R)$, $\pi(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})>\pi(\bm{y}_{\bm{x}}|\bm{x})$.

We now present a lemma to demonstrate that, given an initial non-empty level set, this level set will expand progressively with each iteration of the algorithm. In other words, as the algorithm iterates, the model becomes increasingly reliable. Specifically, there will be an increasing number of queries where the probability distribution of the response pairs aligns with the reward model.

###### Lemma 5.2 .

For a given language model $\pi$, there exists a pure $L(e,\pi,R)$-level set. For query $\bm{x}\in\mathcal{D}_{\text{query}}$, if $\pi(\bm{y}^{\text{prompt}}|\bm{x})-\pi(\bm{y}|\bm{x})>e$, we add the instance-responses pair into the preference dataset $\mathcal{D}_{\text{train}}$ for calculating ranking loss. And assume the updated model $\pi_{\text{new}}=\arg\min\mathbb{E}_{(\bm{x},\bm{y}^{\text{prompt},\bm{y}})
\sim\mathcal{D}_{\text{train}}}\mathcal{L}_{\text{rank}}(\bm{x},\bm{y}^{\text{
prompt}},\bm{y})$. Let $e_{\text{new}}=\min\{e|e>0,L(e,R)\text{ is pure for }\pi_{\text{new}}\}$ and assume that $e_{\text{new}}>\epsilon$ . Then,

$$ $R(\bm{y}^{\text{prompt}}|\bm{x})-e_{\text{new}}\geq(1+\frac{\epsilon}{6\alpha l })(R(\bm{y}^{\text{prompt}}|\bm{x})-e).$ $$

The detail of the proof is provided in Appendix A.1. Lemma 5.2 shows that the updated model will have a larger pure level set as the threshold $e$ decreasing, which indicates that the model’s output probabilities are more consistent with the ranking of the reward scores.

Finally, we present the main theorem to demonstrate that the Ple algorithm will bound the difference between the learned model and the optimal model $\pi^{\star}$, provided there exists a pure level set for the initialized model.

###### Theorem 5.3 .

Suppose there exists a pure $L(e_{0},\pi_{0},R)$-level set for the initialized model $\pi_{0}$, if one runs purification in the Ple algorithm with enough iterations and the initialization: (1) $e_{0}\geq\frac{\alpha+\frac{\epsilon}{6}}{1+\alpha}$, (2) $e_{\text{end}}>\epsilon$ (3) The iteration steps $I\geq\frac{6l}{\epsilon}\log(\frac{1-\epsilon}{\frac{1}{|\mathcal{Y}|}-e_{0}})$ , then we have:

$$ $\displaystyle\mathbb{P}_{\bm{x}\sim p(\bm{x}),\bm{y}\sim p(\bm{y})}\big{(}|\pi (\bm{y}|\bm{x})-\pi^{\star}(\bm{y}|\bm{x})|>\frac{\epsilon}{2}\big{)}\leq 1-c_ {\star}\epsilon.$ (9) $$

The proof of Theorem 5.3 is provided in Appendix A.2. This result provides theoretical support for the effectiveness of our method in aligning language models with generated preference data.

###### Definition 5.1 .

###### Lemma 5.2 .

###### Theorem 5.3 .

## 6 Experiments

### 6.1 Experimental Configurations

Datasets. We use Anthropic’s Helpful and Harmless (HH) dataset as the experimental dataset (Bai et al., ). This dataset is designed to evaluate the alignment of language models with human preferences, ensuring that the models produce responses that are both helpful and harmless. For each query in the HH dataset, there are two responses: a chosen response and a rejected response. The chosen response is preferred based on human evaluators’ ratings, while the rejected response is considered less appropriate or effective. See Table 1 for an example of the HH dataset. The dataset consists of 161K training data points and 8.55K test data points.

Baselines. We compare our method with several existing language model alignment approaches, including:

- •
SFT (Ouyang et al., ): Supervised Fine-Tuning (SFT) trains the model by predicting the next token in a sequence based on a dataset of human-labeled examples to guide it toward desired outputs.
- •
PPO (Ziegler et al., ): Proximal Policy Optimization (PPO) is a reinforcement learning algorithm commonly used in the RLHF process. It encourages the model to produce outputs that receive higher reward scores from the reward model while also maintaining stability by ensuring the model’s outputs remain consistent with those of the initial model.
- •
DPO (Rafailov et al., ): Direct Policy Optimization (DPO) simplifies the RLHF process by deriving an equivalent optimization objective of PPO. This approach allows the model to be directly optimized using human preference data, eliminating the need to train a separate reward model and the subsequent reinforcement learning step.
- •
RAFT (Dong et al., ): Reward-based Fine FineTuning (RAFT) expands the SFT dataset by generating additional samples and selecting those with high reward scores to enhance the SFT dataset. This approach aims to improve the quality of the training data for SFT by including only high-scoring samples from the reward model.

Implementation Details. In our experiments, we use the LLama3 8B base model (Touvron et al., ). This model has been pre-trained on a diverse corpus of text data, enabling it to generate coherent and contextually relevant responses across a wide range of topics.
For implementing SFT, PPO, and DPO, we utilized the Transformer Reinforcement Learning (TRL) library (^1^11[https://github.com/huggingface/trl](https://github.com/huggingface/trl)). For RAFT, we employed the official LMflow library (^2^22[https://github.com/OptimalScale/LMFlow](https://github.com/OptimalScale/LMFlow)). In RAFT, the hyperparameter for the number of sample generations was set to 4. To save memory, we used the Parameter-Efficient Fine-Tuning (PEFT) technique, specifically, Low-Rank Adaptation (LoRA) (Hu et al., ) with rank $r=8$, scaling factor $\alpha=16$, and targeted all linear modules for all experiments. For all baselines, we used the default parameters from their codebases, as we tried other parameters and found no significant difference in the results. For Ple, we set the initial threshold $\tau_{0}=0.2$ and the decay factor $\alpha=0.9$. All experiments were conducted on 8$\times$Huawei Ascend 910B (64GB) hardware with RAM 1000GB.

Evaluation Metrics. We evaluate the performance of our method and the baselines using two metrics: Perplexity (PPL) and Reward Margin (RM). PPL measures the model’s ability to predict the next token in a sequence, with lower values indicating better performance. Additionally, we sampled 1024 queries from the HH dataset. Each model generated responses to these queries, and RM was used to score these responses. The average RM score was calculated to assess the quality of the generated responses, with higher scores indicating better model performance.

Generation Configurations. For each query in the $\mathcal{D}_{\text{query}}$, we discard the queries with more than 256 tokens to reduce NPU memory costs. For algorithms involving online sampling, i.e., PPO, RAFT and Ple, the model is set to generate up to 1024 new tokens given a query. For a fair comparison, we keep the test configuration for
all methods and report the metrics on the test set of HH dataset. For a fair comparison, we maintain the same test configuration across all methods and report the RM metric on a query test set of size 2048, sampled from the HH test set. The perplexity metric is calculated on the entire test set.

### 6.2 Main Results

**Table 1: Example of the HH dataset.**
| Query | Rejected Response | Chosen Response |
| --- | --- | --- |
| If you were going to steal from a convenience store, do you think it would be better in the morning or at night? | Night time would probably be better for avoiding security cameras, but you would be more visible to the store employees at night. | I really couldn’t say, I’m not familiar with stealing convenience store items. |

The main results of our method and the baselines on the HH dataset are summarized in Table 2. For the PLL metric, since the training objective of SFT is aligned with the PPL metric, SFT achieves the best results on this metric. However, our method obtains comparable results to SFT. For the RM metric, our method surpasses all baselines. This highlights the effectiveness of our approach in aligning the model’s outputs with human preferences, resulting in responses that are more favorably evaluated by the reward model.

In addition, to further evaluate the performance of our model, we randomly selected 50 queries from the test set of the HH dataset and generated responses from the models for evaluation. The quality of these responses was assessed by both the Claude API and human annotators, as shown in Figure 4. The results demonstrate that our method consistently outperforms baseline models. Specifically, our approach shows a clear advantage in aligning with human preferences, as reflected in the higher win rates in both API and human evaluations. These findings underscore the effectiveness of our model in generating more desirable responses compared to other baselines.

**Table 2: Main results of our method and the baselines on the HH dataset using LLama3 8B base model.**
| Methods | PPL | RM |
| --- | --- | --- |
| BASE | 14.36 | -1.80 |
| SFT (Ouyang et al., ) | 8.42 | -1.34 |
| PPO (Ziegler et al., ) | 16.35 | -0.87 |
| DPO (Rafailov et al., ) | 15.59 | -0.96 |
| RAFT (Dong et al., ) | 8.52 | -0.82 |
| OURS | 8.45 | -0.75 |

Figure: (a) Win rate evaluated by Claude
Refer to caption: x3.png

Figure: Figure 3: Reward curve of principle-guided responses and original responses on the HH dataset.
Refer to caption: x5.png

### 6.3 Learning Curves

The blue and yellow lines in Figure 3 respectively show the average scores of the model’s responses to each batch of questions at each step, with and without the principle prompt. It can be observed that their scores continuously improve throughout the training process, indicating that the model’s performance is consistently enhancing. We can observe that the scores of both lines are consistently improving over time, which validates the effectiveness of our training method in steadily enhancing the model’s overall performance. Additionally, we notice that the score gap between the original response and the principle-prompt-guided response gradually decreases. This also confirms the rationality of our dynamic threshold reduction approach, as the gap closing indicates that the model is increasingly capable of generating high-quality responses without requiring strict guidance from the principle prompt.

### 6.4 Case Study

To further evaluate the alignment capabilities of our model, we designed several provocative queries aimed at testing the model’s response to potentially illegal or harmful questions. The results show that our model effectively refused to provide answers to these problematic queries, emphasizing the importance of legality and compliance in its responses. Additionally, we included a standard everyday query to assess whether the model could still provide helpful advice without being overly restrictive due to alignment training. The results demonstrate that the model not only successfully rejected unethical requests but also offered practical and constructive suggestions for everyday queries.

Figure: Figure 4: Model’s responses to ethical and productivity-related queries. The first two responses demonstrate the model’s ability to avoid providing assistance on unethical actions, while the third response shows the model’s capability to offer helpful advice on time management.

## 7 Conclusion

In this work, we addressed the challenges of aligning Large Language Models with human expectations by proposing Ple (Progressively Label Enhancement for LLM Alignment). Unlike existing methods that depend on large high-quality datasets and inefficiently utilize generated data, Ple fully leverages all generated responses. By using a dynamically updated threshold and weighting responses based on reward scores, our approach ensures efficient data utilization and alignment with human preferences. Experimental results on HH dataset validate the effectiveness of Ple, demonstrating its superiority over existing language model alignment methods.

## Appendix A Appendix

### A.1 Proof of Lemma 5.2

Assume that there exists a queries set $L(e,R):=\{\bm{x}|R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{
\bm{x}})\geq e\}$ is pure for the model $\pi$, i.e., for any $\bm{x}\in L(e,R)$, $\pi(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})>\pi(\bm{y}_{\bm{x}}|\bm{x})$. we have for any $\bm{x}\in L(e,R)$

$$ $\displaystyle\mathbb{E}_{(\bm{z},\bm{y},\bm{y}^{\text{prompt}})\sim\mathcal{D} _{\text{train}}}\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})< \pi(\bm{y}_{\bm{z}}|\bm{z})\}}\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}}) -R(\bm{z},\bm{y}_{\bm{z}})>$ (10) $\displaystyle R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x }})\Big{]}=0.$ $$

Let $e_{\text{new}}$ be the new threshold and $\frac{\epsilon}{6l\alpha}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-e)\leq e-e
_{\text{new}}\leq\frac{\epsilon}{3l\alpha}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{
prompt}})-e)$. Since the probability density function $d(u)$ is bounded by $c_{\star}$ and $c^{\star}$, we have following inequality for $\bm{x}$ that satisfy $R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\geq e_{
\text{new}}$

$$ $\displaystyle\mathbb{E}_{(\bm{z},\bm{y},\bm{y}^{\text{prompt}})\sim\mathcal{D} _{\text{train}}}\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})< \pi(\bm{y}_{\bm{z}}|\bm{z})\}}\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}}) -R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x}, \bm{y}_{\bm{x}})\Big{]}$ (11) $\displaystyle=$ $\displaystyle\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{ z})<\pi(\bm{y}_{\bm{z}}|\bm{z})\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}} )-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x} ,\bm{y}_{\bm{x}})\right]$ $\displaystyle=$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}} )-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x} ,\bm{y}_{\bm{x}})\right]}{\mathbb{P}_{\bm{z}}\left[R(\bm{z},\bm{y}_{\bm{z}}^{ \text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{ prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle\leq$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}} )-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}{\mathbb{P}_{\bm{z}}\left[R(\bm{z}, \bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{ \bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle+$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),e_{\text{new}}\leq R(\bm{z},\bm{y}_{\bm{ z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})<e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle=$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}} )-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}{\mathbb{P}_{\bm{z}}\left[R(\bm{z}, \bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R(\bm{x},\bm{y}_{ \bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle+$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),e_{\text{new}}\leq R(\bm{z},\bm{y}_{\bm{ z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})<e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle=$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}} )-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}{\mathbb{P}_{\bm{z}}\left[R(\bm{z}, \bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}$ $\displaystyle\quad\frac{\mathbb{P}_{\bm{z}}\left[R(\bm{z},\bm{y}_{\bm{z}}^{ \text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle+$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),e_{\text{new}}\leq R(\bm{z},\bm{y}_{\bm{ z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})<e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle=$ $\displaystyle\underbrace{\mathbb{E}_{\bm{z}}\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z }}^{\text{prompt}}|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z})\}}\Big{|}R(\bm{z},\bm{y }_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>e\Big{]}}_{=0\text{ according to Eq. \ref{eq:proof_init}}}$ $\displaystyle\quad\frac{\mathbb{P}_{\bm{z}}\left[R(\bm{z},\bm{y}_{\bm{z}}^{ \text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})\geq e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle+$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z}),e_{\text{new}}\leq R(\bm{z},\bm{y}_{\bm{ z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})<e\right]}{\mathbb{P}_{\bm{z}} \left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})>R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle\leq$ $\displaystyle\frac{\mathbb{P}_{\bm{z}}\left[e_{\text{new}}\leq R(\bm{z},\bm{y} _{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})<e\right]}{\mathbb{P}_{\bm {z}}\left[R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})> R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\right]}$ $\displaystyle\leq$ $\displaystyle\frac{c^{\star}(e-e_{\text{new}})}{c_{\star}(R(\bm{x},\bm{y}_{\bm {x}}^{\text{prompt}})-e)}$ $$

Then, we can further relax the inequality by using the boundary of $e_{\text{new}}$, we have:

$$ $\displaystyle\mathbb{E}_{(\bm{z},\bm{y},\bm{y}^{\text{prompt}})\sim\mathcal{D} _{\text{train}}}\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})< \pi(\bm{y}_{\bm{z}}|\bm{z})\}}\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}}) -R(\bm{z},\bm{y}_{\bm{z}})>$ (12) $\displaystyle\qquad\qquad\qquad\qquad R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}} )-R(\bm{x},\bm{y}_{\bm{x}})\Big{]}$ $\displaystyle\leq$ $\displaystyle\frac{c^{\star}(e-e_{\text{new}})}{c_{\star}(R(\bm{x},\bm{y}_{\bm {x}}^{\text{prompt}})-e)}$ $\displaystyle\leq$ $\displaystyle\frac{c^{\star}}{c_{\star}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt }})-e)}\frac{\epsilon}{3l\alpha}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-e)$ $\displaystyle=$ $\displaystyle\frac{\epsilon}{3\alpha}$ $$

Then, the gap between $\pi$ and the optimal model $\pi^{\star}$ should be controlled by:

$$ $\displaystyle|\pi(\bm{y}|\bm{x})-\pi^{\star}(\bm{y}|\bm{x})|$ (13) $\displaystyle\leq$ $\displaystyle\alpha\mathbb{E}_{(\bm{z},\bm{y},\bm{y}^{\text{prompt}})\sim \mathcal{D}_{\text{train}}}\Big{[}\bm{1}_{\{\pi(\bm{y}_{\bm{z}}^{\text{prompt} }|\bm{z})<\pi(\bm{y}_{\bm{z}}|\bm{z})\}}\Big{|}$ $\displaystyle R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z }})>R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\Big{]} +\frac{\epsilon}{6}$ $\displaystyle\leq$ $\displaystyle\alpha\frac{\epsilon}{3\alpha}+\frac{\epsilon}{6}$ $\displaystyle=$ $\displaystyle\frac{\epsilon}{2}$ $$

Then, for $\bm{x}$ that satisfy $R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\geq e_{
\text{new}}$, we have:

$$ $\displaystyle\pi(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-\pi(\bm{y}_{\bm{x}}| \bm{x})$ (14) $\displaystyle\geq$ $\displaystyle(\pi^{\star}(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-\frac{ \epsilon}{2})-(\pi^{\star}(\bm{y}_{\bm{x}}|\bm{x})+\frac{\epsilon}{2})$ $\displaystyle=$ $\displaystyle\pi^{\star}(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-\pi^{\star}( \bm{y}_{\bm{x}}|\bm{x})-\epsilon$ $\displaystyle\geq$ $\displaystyle e_{\text{new}}-\epsilon\geq 0,$ $$

which means that $L(e_{\text{new}},R)$ is pure for $\pi$. Here, we assume that the range of the reward function is between 0 and 1. As a result, the output probability distribution of $\pi^{\star}$ is directly equal to the reward scores. Meanwhile, we have:

$$ $\displaystyle R(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-e_{\text{new}}$ (15) $\displaystyle\geq$ $\displaystyle R(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-\big{(}e-\frac{ \epsilon}{l\alpha}(R(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-e)\big{)}$ $\displaystyle=$ $\displaystyle R(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-e+\frac{\epsilon}{l \alpha}(R(\bm{y}_{\bm{x}}^{\text{prompt}}|\bm{x})-e)$ $\displaystyle\geq$ $\displaystyle(1+\frac{\epsilon}{l\alpha})(R(\bm{y}_{\bm{x}}^{\text{prompt}}| \bm{x})-e)$ $$

### A.2 Proof of Theorem 5.3

Firstly, we prove that there exists a pure level set for the initialized model $\pi_{0}$. Considering $\bm{x}$ that satisfy $R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\geq e_{0}$, we have $\mathbb{P}_{\bm{z}}\Big{[}\pi(\bm{y}_{\bm{z}}^{\text{prompt}}|\bm{z})<\pi(\bm{
y}_{\bm{z}}|\bm{z})\Big{|}R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},
\bm{y}_{\bm{z}})\geq e_{0}\Big{]}\leq R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}}
)-e_{0}$. Since the assumption in Eq. (8) holds, we have $\alpha(R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-e_{0})+\frac{\epsilon}{6}\leq
e
_{0}$ to ensure that $\pi$ have the similar output with $\pi^{\star}$. Then, we can choose $e_{0}\geq\frac{\alpha+\frac{\epsilon}{6}}{1+\alpha}$.

Then, in the rest of the iterations we assume that the level set $R(\bm{z},\bm{y}_{\bm{z}}^{\text{prompt}})-R(\bm{z},\bm{y}_{\bm{z}})\geq e$ is pure. We decrease $e$ by a factor, i.e., $\frac{\epsilon}{6l\alpha}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-e)\leq e-e
_{\text{new}}\leq\frac{\epsilon}{3l\alpha}(R(\bm{x},\bm{y}_{\bm{x}}^{\text{
prompt}})-e)$, such that in the level set $R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})\geq e_{
\text{new}}$, we have $|\pi(\bm{y}|\bm{x})-\pi^{\star}(\bm{y}|\bm{x})|\leq\frac{\epsilon}{2}$. This condition ensures that the correctness of the chosen of the samples for the ranking loss when $e\geq\epsilon$. To get the largest pure level set, we can choose $e_{\text{end}}=\epsilon$. Since the probability density function $d(u)$ is bounded by $c_{\star}$ and $c^{\star}$, we have:

$$ $\displaystyle\mathbb{P}_{\bm{x}\sim p(\bm{x}),\bm{y}\sim p(\bm{y})}\big{(}|\pi (\bm{y}|\bm{x})-\pi^{\star}(\bm{y}|\bm{x})|\leq\frac{\epsilon}{2}\big{)}$ (16) $\displaystyle=$ $\displaystyle\mathbb{P}_{\bm{x}\sim p(\bm{x}),\bm{y}\sim p(\bm{y})}\big{(}R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})<e_{\text{end }}\big{)}$ $\displaystyle\geq$ $\displaystyle\mathbb{P}_{\bm{x}\sim p(\bm{x}),\bm{y}\sim p(\bm{y})}\big{(}R( \bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-R(\bm{x},\bm{y}_{\bm{x}})<\epsilon\big {)}$ $\displaystyle\geq$ $\displaystyle c_{\star}\epsilon$ $$

Then $\mathbb{P}_{\bm{x}\sim p(\bm{x}),\bm{y}\sim p(\bm{y})}\big{(}|\pi(\bm{y}|\bm{x
})-\pi^{\star}(\bm{y}|\bm{x})|>\frac{\epsilon}{2}\big{)}\leq 1-c_{\star}\epsilon$.

The rest of the proof is to show that the iteration step $I\geq\frac{6l}{\epsilon}\log(\frac{1-\epsilon}{\frac{1}{|\mathcal{Y}|}-e_{0}})$:

$$ $\displaystyle\left(1+\frac{\epsilon}{6l\alpha}\right)^{I}\left(R(\bm{x},\bm{y} _{\bm{x}}^{\text{prompt}})-e_{0}\right)\geq R(\bm{x},\bm{y}_{\bm{x}}^{\text{ prompt}})-\epsilon$ (17) $\displaystyle\Rightarrow\left(1+\frac{\epsilon}{6l\alpha}\right)^{I}\geq\frac{ R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-\epsilon}{R(\bm{x},\bm{y}_{\bm{x}}^{ \text{prompt}})-e_{0}}$ $\displaystyle\Rightarrow I\log\left(1+\frac{\epsilon}{6l\alpha}\right)\geq\log \left(\frac{R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-\epsilon}{R(\bm{x},\bm{y }_{\bm{x}}^{\text{prompt}})-e_{0}}\right)$ $\displaystyle\Rightarrow I\frac{\epsilon}{6l\alpha}\geq I\log\left(1+\frac{ \epsilon}{6l\alpha}\right)\geq\log\left(\frac{R(\bm{x},\bm{y}_{\bm{x}}^{\text{ prompt}})-\epsilon}{R(\bm{x},\bm{y}_{\bm{x}}^{\text{prompt}})-e_{0}}\right)$ $\displaystyle\Rightarrow I\geq\frac{6l\alpha}{\epsilon}\log\left(\frac{R(\bm{x },\bm{y}_{\bm{x}}^{\text{prompt}})-\epsilon}{R(\bm{x},\bm{y}_{\bm{x}}^{\text{ prompt}})-e_{0}}\right)\geq\frac{6l\alpha}{\epsilon}\log\left(\frac{1-\epsilon }{\frac{1}{|\mathcal{Y}|}-e_{0}}\right)$ $$

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Why GPT-4 is not used for annotation when calculating the win rates?
soundness: 2
strengths: Guiding the language model with a set of principles is a straightforward yet effective approach for generating high-quality responses. This simplicity suggests promising potential for broader application of this method in the future.
summary: The paper proposes a new method for aligning large language models. 

The method proposed in the paper is PLE, i.e. Progressively Label Enhancement for LLM Alignment. This framework can dynamically adjust the model's training process based on the evolving quality of the generated data. 

Specifically, the method designs a set of principles and uses these principles to guide the model to generate good responses, and then designs a ranking loss and a re-weight loss to train the language models. 

On the dataset of HH, the proposed is demonstrated to be better than PPO, and DPO, these commonly used methods.
weaknesses: 1. **Lack of Ablation Study**: The training loss comprises a ranking loss and a re-weighting loss; however, the paper does not include an ablation study to analyze the individual contributions of these losses to the model’s performance.

2. **Limited Scope of Principle-Based Evaluation**: While using principles proves effective in generating quality responses, the evaluation is limited to alignment tasks. This narrow focus leaves the generalizability of the approach uncertain. Testing across a wider range of tasks, such as mathematical reasoning, summarization, and controlled text generation, would provide stronger evidence of the method’s broader applicability.

3. **Lack of Systematic Study on Principle Design**: The paper does not systematically examine the design of principles or how different principles impact the model’s overall performance.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See my comments in weaknesses. 
note: I did not check into the details the theoretical analysis.
Minor comment:
DPO is not short for direct policy optimization, it is for direct preference optimization.
soundness: 2
strengths: 1. prompting with principles is interesting and important; 
2. the idea of changing training data progressively using reward function is interesting;
3. there are positive performances in the experiment.
summary: This paper proposed progressively label enhancement for LLM alignment, where the authors prompt the model to generate responses for both the original query and the query guided by a set of carefully designed principles, and then utilize a dynamic threshold to determine the appropriate training approach for both responses based on their corresponding reward scores.
The central idea lies in that at the earlier stage of training, the method tries to use pairs with larger reward margin, as the training proceeds, the reward margin can be decreased.
weaknesses: 1. the method does not seem sound to me, there are a few questions left unclear:
1) Essentially, this is a PPO method? so why does EQ 2 have no regularization terms (the KL term between \pi_{sft} and \pi_{\theta};
2) in Eq 7 and the line 15 of the algorithm, the tau seems to be a constant all the time, where you should have \tau_{t+n} = \tau_{t}*\alpha?
3) in the loss functions, normally we are optimizing with mini-batch, which sums all the log probability together in a mini batch (because samples are IID), so a better formulation should be like log\pi_{\theta}? Instead, the authors directly used the probability distribution, which is weird for me. 
2. About evaluation:
1) the authors only tested the method with a base model of LLama3 8B, where I believe more base models should be verified, for instance , chatglm-x, qwen-x, and particularly with larger sizes to see if the claims still hold on larger models;
2) the authors only experimented with one dataset, of course more datasets should be validated with;
3) the authors measured the winrate and reward scores over baselines, but it is more interesting to see the results on other benchmarks, for instance, IFEval, MATH, BBQ, and other typical benchmarks.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Sec 4.1.: `Language model alignment requires a large amount of high-quality data` Consider LIMA, for example, only contains 1k examples, although its base model (65B) is larger than the model used in this work. 

* How are the principles integrated into the response generation? How many of such generations are required?
soundness: 2
strengths: * Proof of the convergence is provided in the work, showing evidence of the effectiveness of the proposed method on the theoretical aspect.

* This paper considers the training process along the data quality together, giving a novel perspective compared to prior alignment works that consider training and data generation as two separate processes.

* The paper is clear and easy to understand.
summary: This work proposes a novel language model alignment method by adaptively adjusting the training process based on the generated data quality, which is computed by determining whether a reward score of principle-guided response is higher than the generated response. The authors provide a theoretical analysis to show that the proposed method would lead to a convergence to the optimal model. The experiments are conducted on a single dataset, and the method is shown to achieve a higher performance than the baseline methods (vanilla SFT, DPO, PPO, RAFT).
weaknesses: * The experiment is only conducted on a single dataset and a single model variant. The paper could be strengthened by a series of post-training analyses.

* Although the method is shown to have better performance on the Helpful and Harmless (HH) dataset, it would be useful if the authors could present a complexity comparison between the baselines, which may seem to be more efficient than the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. It would be better to make the font size of Figure 1 larger.
soundness: 2
strengths: 1. The proposed approach is well-motivated.  
2. The proposed approach is novel.  
3. The approach show good performance.
summary: This paper proposes the Progressively Label Enhancement (PLE) framework to improve LLM alignment with human expectations. PLE dynamically adjusts the training process by weighting both principle-guided and original responses based on their reward scores, enhancing data utilization efficiency. Experiment results show that PLE outperforms existing studies.
weaknesses: 1. Although the approach is new, the high-level idea of using prompt to instruct LLM to generated human preferred answer and using reward to optimize the LLM is not new.  
2. The paper evaluates model performance using a reward model. It would be better more details of the reward model can be provided. Also, it is possible that the reward model has bias. It would be better that the authors can discuss the effect of the reward model. Otherwise, trying multiple reward models may address the issue.  
3. Some details of the experiments are missing. For the results of SFT, PPO, DPO in Table 2, are they the results obtained by the LLM trained by the authors? or they are the results of the instruct version of LLaMA-3 released by Meta? If they are the results obtained from the author, I'm curious about the results of directly evaluating the instruct version of LLaMA-3.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. When training baselines like SFT and DPO, it appears that the authors used preference responses from the HH dataset. I’m curious how the baselines would perform if principled guided responses were used as preference responses for training instead.

2. A few potential typos:

    a. For PLE, is the full name “Progressively Label Enhancement for LLM Alignment” as stated on line 21, or “Selective Label Enhancement for Language Model Alignment” as on line 156?

    b. In line 333, “Reward-based Fine FineTuning (RAFT)”—it seems this is not the full name of RAFT.

3. Could the authors expand the experiments on the effects of different components of PLE (e.g., varying loss functions) or hyper-parameters? This would provide deeper insights into which parts contribute most effectively.

4. For Figure 3, how many iterations were trained in total? Are there only 8 iterations in the entire training process?
soundness: 2
strengths: The authors identify the decoupling issue between data generation and model training in current methods, and propose PLE to improve data utilization. Through both automatic and human evaluation on the HH dataset, they demonstrate that PLE outperforms baseline approaches. The authors theoretically prove that with a progressively updated threshold strategy, PLE can bound the error rate between the trained model and the optimal model.
summary: The paper addresses the decoupling of data generation and model training in current RLHF methods by proposing PLE (Progressively Label Enhancement for LLM Alignment). PLE dynamically adjusts the training process based on the quality of the generated data to maximize its utility. Specifically, the contributions of the paper are as follows:

1. The authors are the first to identify that previous alignment methods overlooked the coupling of data generation and training. They introduce PLE to combine these two stages, demonstrating reasonable performance compared to baselines in experiments.

2. The authors theoretically prove that, through a progressively updated threshold strategy, PLE can bound the error rate between the trained model and the optimal model.
weaknesses: 1. The authors provide comparative experiments with 4 baselines on the HH dataset and include training curves comparing principle-guided responses with original responses, but I believe the experiments remain insufficient despite the theoretical proof.

- Additional evaluations on datasets like IMDb (as seen in DPO), the Reddit TL;DR summarization dataset, and UltraFeedback would offer valuable insights into model performance on diverse data using PLE. 
- It would also be beneficial to test generalization capabilities on out-of-distribution data—for instance, training on UltraFeedback and evaluating on HH.
- Including additional evaluation metrics, such as MSSTR and Distinct-n(as seen in RAFT), could enhance the credibility and robustness of the results.

2. The paper lacks sufficient motivation for introducing progressive approaches and label enhancement methods. Additionally, the experimental results show somewhat minor improvements over the baselines and lack the comprehensiveness needed to justify these components(see weakness 1 for suggestions).

Meta-review: This paper proposes Progressively Label Enhancement (PLE) for LLM Alignment, which dynamically adjusts the model's training process based on the quality of the generated data. The reviewers raised significant concerns including the novelty of the method and the soundness of the evaluation. No reviewer was willing to support the paper for acceptance.

Decision: Reject