Progressively Label Enhancement for Large Language Model Alignment
Authors: Biao Liu
Ning Xu
Xin Geng
Venue: ICML 2025 poster (2025)
Keywords: Language Model Alignment
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLM) alignment aims to prevent models from producing content that misaligns with human expectations, which can lead to ethical and legal concerns. 
   In the last few years, Reinforcement Learning from Human Feedback (RLHF) has been the most prominent method for achieving alignment.
   Due to challenges in stability and scalability with RLHF stages, which arise from the complex interactions between multiple models, researchers are exploring alternative methods to achieve effects comparable to those of RLHF.
   However, these methods often rely on large high-quality datasets.
   Despite some methods considering the generation of additional data to expand datasets, they often treat model training and data generation as separate and static processes, overlooking the fact that these processes are highly interdependent, leading to inefficient utilization of the generated data.
   To deal with this problem, we propose PLE, i.e., Progressively Label Enhancement for LLM Alignment, a framework that dynamically adjusts the model’s training process based on the evolving quality of the generated data.
   Specifically, we prompt the model to generate responses for both the original query and a set of carefully designed principle guided query, and then utilize a dynamic threshold to determine the appropriate training approach for both responses based on their corresponding reward scores. 
   Experimental results demonstrate the effectiveness of PLE compared to existing LLM alignment methods.

Progressively Label Enhancement for Large Language Model Alignment
Biao Liu 12 Ning Xu 12 Xin Geng
Abstract
1. Introduction
Large Language Models, such as the LLama series (Touvron et al., 2023) and OpenAI's GPT series (Floridi & Chiriatti, 2020; OpenAl, 2023), have demonstrated their powerful capabilities across various language tasks, including trans- lation (Zhang et al., 2023), summarization (Pilault et al., 2020). and conversational interaction (Wang et al., 2023a), In certain scenarios, they have even exhibited performance that matches that of human experts (Ouyang et al., 2022).
Large Language Models (LLM) alignment aims to prevent models from producing content that mis- aligns with human expectations, which can lead to ethical and legal concerns. In the last few years, Reinforcement Learning from Human Feedback (RLHF) has been the most prominent method for achieving alignment. Due challenges in sta- bility and scalability with RLHF stages, which arise from the complex interactions between mul- tiple models, researchers are exploring alternative methods to achieve effects comparable to those of RLHF. However, these methods often rely on large high-quality datasets. Despite some meth- ods considering the generation of additional data to expand datasets, they often treat model training and data generation as separate and static pro- cesses, overlooking the fact that these processes are highly interdependent, leading to inefficient utilization of the generated data. To deal with this problem, we propose PLE, i.e., Progressively Label Enhancement for LLM Aligmment, frame- work that dynamically adjusts the model's train- ing process based on the evolving quality of the generated data. Specifically, we prompt the model to generate responses for both the original query and a set of carefully designed canncipleggiided query, and then utilize a dynamic threshold to determine the appropriate training approach for both responses based their corresponding re- ward scores. Experimental results demonstrate the effectiveness of PI.F compared to existing LLM alignment methods.
However, these language models may not always generate produce
responses as expected by humans and can content that violates human ethics or legal boundaries (Bai et al., 2022a; Askell et al., 2021). Therefore, it is crucial for researchers to explore the limitations of these models and implement restrictions on output generation to ensure safety and compliance, process known as Al alignment.
The most prominent method for achieving Al alignment Reinforcement Learning from Human Feedback (RLHF) (Ziegler al., 2019; Stiennon et al., 2020; Ouyang et al., 2022). RLHF employs Supervised Fine-Tuning (SFT) to guide models using human instructions (Wang et al., 2023b; Taori et al., 2023), followed by training a Reward Model on human-rated outputs (Ouyang et al., 2022), and optimizing the model with Reinforcement Learning (RL) algorithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017; Askell al., 2021; Bai et al., 2022b). However, due to challenges in stability and scalability with the RL stage, which arise from the complex interactions between multi- ple models, researchers are exploring alternative methods. For instance, LIMA (Zhou et al., 2023) has experimentally demonstrated that when the pre-trained model's capabilities are sufficiently strong and the quality of the SFT data is high, it achieve results comparable those of RLHE. RAFT (Dong et al., 2023) expands the SFT dataset by gen- erating additional samples and selecting those with high reward scores to enhance the SFT dataset. RRHF (Yuan et al., 2023) simplifies the RLHF process by integrating the subsequent RI. steps into the SFT phase as a regularization term.
'School of Computer Science and Engineering, South- cast University, Namjing, China. E-mail: {liubia001, xning, xgeng) @seu.edu.cn. *Key Laboratory of New Generation Artifi cial Intelligence Technology and Its Interdisciplinary Applications (Southcast University), Ministry of Education, China. Correspon- dence to: Ning Xu <xning @ seu.edu.en>.
However, these methods highly dependent on large amounts of high-quality data, which is impractical in certain applications, such as the medical field (Yang et al., 2024b; Li et al., 2023) or chip design (Liu et al., 2023b; 2024a).
Proceedings of the n2nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Progressively Label Enhancement for Large Language Model Alignment
Additionally, even though some methods generate extra data to expand the training set to alleviate the problem. They often treat model training and data generation as separate and static processes, which overlooks the fact that these processes are highly interdependent, such as selecting only a small portion of high-scoring data from the reward model, discarding a significant amount of other potentially use- ful data, leading to inefficient utilization of the generated data. Therefore, we consider designing an efficient frame- work that couples the data generation and model training processes, allowing them to work synergistically, thus en- sures that all generated data, including potentially useful lower-scoring data, is effectively utilized, thereby improving training efficiency.
Reinforcement Learning from Human Feedback (RLHF) (Ziegler al., 2019; Stiennon et al., 2020; Ouyang et al., 2022). The framework of RLHF first employs Supervised Fine-Tuning (SFT) to guide the model to follow human instructions in an imitative manner (Wang et al., 2023b; Taori et al., 2023). The next steps involve training a Reward Model on a dataset reflecting human preferences, created from human evaluators' ratings of the SFT model's outputs (Ouyang al., 2022). Using reinforcement learning algo- rithms like Proximal Policy Optimization (PPO) (Schulman et al., 2017), the SFT model is further optimized by contin- uously generating outputs, receiving evaluations from the Reward Model, and updating its parameters to maximize alignment with the Reward Model (Askell et al., 2021; Bai et al., 2022b).
Motivated by the above consideration, we propose novel framework named PLE, i.e., Progressively Label Enhance- ment for Language Model Alignment. Specifically, during the sample generation phase, we design set of principles to guide the model to output according to human expectations. When the reward score difference between the principle- guided output and the response to the original query exceeds dynamically updated threshold, indicating a significant improvement under the principle-guiding the model is en- couraged to align its output with the better response and move away from the poorer one. If the difference is less than or equal to the threshold, both responses are considered of similar quality. To fully utilize all generated responses, we incorporate both in the model's training, assigning weights based on the normalized reward scores. Our contributions can be summarized follows:
However, due to the challenges of stability and scalabil- ity involved in the interactions between multiple models in RLHF, researchers have started exploring other more di- rect and efficient methods for model alignment (Rafailov et al., 2023; Yuan et al., 2023; Zhou et al., 2023; Dong et al., 2023). DPO derives an equivalent optimization objective from RLHE, allowing the model to be directly optimized using preference data without the need to train a separate reward model (Rafailov et al., 2023). Similarly, RRHF incorporates the steps of RLHF into the SFT stage by intro- ducing a regularization term, which encourages the model to generate preferred responses with higher probability and poor responses with lower probability (Yuan et al., 2023). LIMA has experimentally demonstrated that when the pre- trained model is sufficiently good, only small amount of high-quality data is needed. By using only SFT, it is pos- sible to obtain well-aligned model without the need for the subsequent complex RLHF steps (Zhou et al., 2023). RAFT similarly posits that using only SFT is sufficient for effective model alignment. They expanded the SFT training set by sampling a batch of high-scoring data based on the scores from the reward model (Dong et al., 2023).
Practically, we are the first to identify that previous alignment methods overlook the coupling between data generation and model training, leading to inef- ficient utilization of generated data. And we propose a novel framework that integrates these two processes, enabling them to work synergistically.
Theoretically, we prove that with the progressively updated threshold strategy, our approach can bound the error rate between the trained model and the optimal model, ensuring convergence within a controlled range.
3. Preliminaries
We first introduce the formal notation for the language model alignment problem. Let V be a vocabulary of a language model. The goal of alignment is to ensure that the language model π X + y generates response y E y that is consistent with human values and preferences given query X, where the query = [z',²² ..... x=" and response y = [ .................. ....... y""
Extensive experimental results validate the effectiveness of our methods over several existing language model alignment approaches.
are sequences of tokens, the input space X' = 1m and the output space y -
2. Related Work
The alignment process typically begins with Supervised Fine- Tine-Tuning (SFT) stage, which adjusts the language model using Maximum Likelihood Estimation on human-labeled
The alignment of language models refers to the process of ensuring that the models behave in ways that are consis- tent with human values, ethical principles, and intended purposes (Leike et al., 2018). The most prominent and ef- fective method currently used achieve this alignment is
2

---

Progressively Label Enhancement for Large Language Model Alignment
How you unihosole money?
menerr shaal
How droo ajúb intervicu?
relaa You the
Lekinal You sohht sould cepor librraatee coodge
I zena sortt with nec. Inticttonee in dlippll and
Ties and lied light claeel pkrt ieservtee Brass
veceee, paret
and
8 How do you embeatte money?
cole pollin laariin
R How should dress for a job interview?
beiiemoo luves and antiestrr de expinion lar
comercative accosr Grne you adde is clea and apprepttxc for ove
tratace scce mere porlasti thet accully
(a)
(b)
Figure 1. Comparison of language model responses with and without principle guidance. (a) Without principles, the model generates an unethical response to query about embezzlement. With principles, the model refrains from providing harmful information and instead offers an ethical response. (b) For a query about job interview attire, both responses are consistent and align with heing informative and helpful.
high-quality dataset Dan =
(1vi yi) Hii:
Therefore, we consider generating additional data during training to expand the dataset. Motivated by the self-align approach (Wang et al., 2023b; Sun et al., 2023), we design a set of principles to guide the model in generating responses that align closely with human preferences:
N
ni
Sat = Σ
Σ
log P [y kg 0),
(=1 ==
where N is the number of training examples, is the length of the i-th target sequence, and / represents the parameters of the language model Te.
Your answers should follow these rules:
ethical: You should actively refrain users on illegal, immoral, or harmful topics.
The goal of language model alignment is to ensure that the model's responses queries align with human prefer- ences. These preferences are typically captured by a reward model R: (2) + R, where higher scores indicate that responses better align with human values and preferences. Conversely, lower scores indicate less alignment. An ideal model maximizes the expected reward:
informative: You should provide users with accurate, relevant, and up-to-date information in its responses. helpful: Your responses should be positive, interesting, helpful and engaging.
which is denoted as p = i'. ....... pp"',, where np is the token length of the principle prompt. As long as the model's input length allows, entries for these principles can be expanded as desired.
π arg max Ean [R(x, y)l,
where πt represents the optimal policy that maximizes the expected reward according to the reward model R.
Let T sft be the SFT-aligned model optimized by Eq. (1) and we use it as the initial model. During training, for each query E D Qquery = { where Ng is the num- ber of queries, the model samples response y <0(-1z)) In addition, the crinciplc-guider model then samples a re- sponse: Prompe ~ To [p. æz) based on the set of principles designed to expect ethical, informative, and helpful output. The reward model R assigns the scores s R(z, y) and sprump R(x,, prumpt).
4. The Proposed Method
In this section, we present our novel framework named PI.F, i... Progressively Label Enhancement for Language Model Alignment. As illustrated in Figure 1, during the sample gen- eration phase, we use carefully crafted principles to guide the model's s outputs. When the reward score difference be- tween the principle-guided output and the original response exceeds dynamically updated threshold, the model is en- couraged to align with the better response and move away from the poorer one. If the difference is less than or equal to the threshold, both responses are considered of similar quality and are assigned weights based on their normalized reward scores for model training.
When the difference between the reward scores, prompt 8, exceeds a threshold т, we consider that the current model has generated a better response based on the principles com- pared to the original response. Therefore, to encourage the model to generate responses closer to the better response and away from the poorer response for the given input æ, we adopt ranking loss. This ranking loss aims adjust the model's parameters that the likelihood of generat- ing the better response is increased while the likelihood of generating the poorer response is decreased by the length-
4.1. PLE
Language model aligmment requires a large amount of high- quality data, which is often impractical many scenarios.
3

---

Progressively Label Enhancement for Large Language Model Alignment
normalized log probability (Yuan et al., 2023; Zhao al., 2023). The formula is as follows:
Algorithm 1 The PLE Algorithm
Input: The SFT training set Dan, query set Dquery, the human-designed principle p, the initial base model mg. the initial threshold To and the decay factor a and the number of iteration I.
Lrakk
En log xo (ypromery))
1: Initialize the model with SFT on Dst with Eq. (1)
Σ
Initialize the training dataset with empty set Dinin 0.
[yyprompt]
(3)
gtamp 72
Ej log Te (y' [w]] [y) x)
3: for each training step = 1 to / do
4: Fetch mini-batch queries Bquery from D Quuery
y
5: for each query æ E queery do
6: Sample a response y ~ no (-(a))
When the difference between the reward scores is less than or equal to the threshold Ti at the training step t, we consider that the response generated by the principle-guides model and the original response are of similar quality. Therefore, both responses are deemed effective for the model's training. We include both responses in the dataset for subsequent training, with their weights determined by the magnitude of their scores, a process known as label enhancement (Xu et al., 2021; 2023c). The formula for this process is as follow:
7: Sample a principle-guided response poompi
no( Te(- a))
Calculate reward scores
R(x, y) and
sprompe R(2,, prompt)
9: Add the generated sample (xz, y, yrompt to Duain Dinin + {(xr,, U, yrompp)) U Dimin
10: end for
11: for each (æ. y: gompp)) E Darain do
12: Update model parameters by Eq. (6)
13: end for
wweigheed-stt
14: Update threshold T. Tt-1
Σ
Σ log To ( xx)
15: end for
gmoml S
(4)
Output: The language model Te.
Prroppp
Σ log ne (yppooeyy o | (ymoopyyess x)
where the weights w and Promppt are calculated as follows to normalize them to the range 1-1, l]:
5. Theoretical Analysis
2
2c a
In this section, we will provide a theoretical analysis to demonstrate that PLE, which uses dynamically updated threshold for data selection and model training, will ulti- mately converge to the optimal model x defined in Eq. (2).
w: 6 + Catee
1,
in promppt
e a
1. (5)
This approach ensures that both the original and the principle-guided responses contribute to the training process, with their influence proportional to their respective reward scores. By incorporating both responses, we enhance the model's ability to generate outputs that align with human preferences and values. Then the final objective function is:
Before proceeding with the proof, we present some ba- sic definitions and assumptions. For two queries x and that satisfy R(z, y " R(z, y=) > R(a,, ya (pr) R(æ, ya), i.e., the margin between the reward score of principle-guided response R( gompp and that of orig- inal response R(z,yz) yz) is larger than that in point æ, the indicator function (=) R(z, g
L Lrank + Leeightdd sft
(6)
In the training process, as the model's output scores for the original responses become increasingly close to the menciple-guided responses, indicating the model's improved capability, we progressively reduce the threshold. This al- lows the loss function to adapt to these smaller variations. Here's how the threshold adjustment can be expressed:
R(z, y2) > R(æ, y (uump)) R(æ, yn) equals 1 if the model's output probabilities for the responses т (y)2z)) and t (yy [z) are inconsistent with the corresponding rank- ing of their reward scores. Then, the gap between the cur- rent model's probability and the optimal model, i.e., the approximation error of the model, could be controlled by the inconsistency between the model's 's output probabilities and the corresponding ranking of their reward scores for all the queries z.
Tr Tt- ar,
where 7, is the threshold at training step / and a E (0, 1) is a decay factor that progressively reduces the threshold over time.
The whole process of PL.F is shown in Algorithm 1.
Therefore, we assume that there exist constants a, 6 < 1,
4

---

Progressively Label Enhancement for Large Language Model Alignment
such that for any æ E X' and y E V
Theorem 5.3. Suppose there exists pure L(eo, Toy R)- level set for the initialized model no, if one runs purification in the PLE algorithm with enough iterations and the initial-
[ n" (y) [2) S aZ yysaau)
(1(
R(z, y y - (((2))22)
(8)
ization: (1) eo 2 a=t
I+a
(2) Gend > E e3) The iteration steps
I 12
61 log( }- then we have:
R(x, y - R(x yv)) 6
+
19T
P Po (1T())2)) x"" (v(=)))
C 2
In addition, for the probability density function d(uu) of the cumulative distribution function of the margin of the reward scores (u)) = P (u(xz) I u), where u(xx)= R(x: yf ) - R( yz) denotes the margin of the reward scores for the query æ. We assume that there exists constants c,, such that c < d(u)= ct. Then, we define the worst- case density imbalance ratio as I e"
(9)
S C46.
The proof of Theorem 5.3 is provided in Appendix A.2. This result provides theoretical support for the effectiveness of our method in aligning language models with generated preferences data.
c.
Motivated by the pure level set in traditional meachine learn- ing (Zhang et al., 2021; Xu et al., 2023a), the region where the language model is reliable, i.e., the region where the model's s output probabilities are consistent with the ranking response and
6. Experiments
6.1. Experimental Configurations
of the reward scores of the principle-guic the original response, can be defined as:
Datasets. We conducted experiments three tasks. (1) For multi-turn dialogue task, we use Anthropic's Helpful and Harmless (HH) dataset as experimental dataset (Bai et al., 2022a). This dataset is designed to evaluate the alignment of language models with human preferences, ensuring that the models produce responses that are both helpful and harmless. For each query in the HH dataset, there are two responses: chosen response and rejected response. The chosen response is preferred based on human evaluators' ratings, while the rejected response is considered less ap- propriate or effective. The dataset consists of 161K K training data points and 8.55K test data points.
Definition 5.1. Pure (e, n, Ry-level set: A queries set
L(e, R) = {ax R(x, ya ) R(x, yu) 2 e} is pure for the model n, if for any x E L(e, R),, π T(ya0m"11 (x) > (z)
We now present lemma to demonstrate that, given an initial non-empty level set, this level set will expand progressively with each iteration of the algorithm. In other words, as the algorithm iterates, the model becomes increasingly re- liable. Specifically, there will be an increasing number of queries where the probability distribution of the response pairs aligns with the reward model.
(2) For controlled text generation task, we use IMDb dataset (Maas et al., 2011). This dataset is widely used for sentiment analysis and consists of movie reviews labeled as either positive or negative. It contains 50K labeled reviews, evenly split between training and testing sets. (3) For summarization task, we use Reddit TL;DR summarization dataset (Völske et al., 2017). It contains user- seer-generated posts paired with concise summaries, providing a challenging benchmark for abstractive summarization tasks. It includes a diverse range of topics and writing styles, making it suitable for evaluating the summarization capabilities of language models.
Lemma 5.2. For given language model n, there ex- ists a pure Lie, π,, Rly-eveel set. For query x E D Qauery. if n (yprrmp xx) T a)) > we add the instance- responses pair into the preference dataset Dirain for cal- culating ranking loss. And assume the updated model Crank (x, prrmmpp y). Let Cyew mineele > O, L(e, R)) is pure for ] and assume
Tnew = arg min E( (umnwrn
that Enew > E. Then.
Ryy (x)-
Enew 2 (1
) prompppp (x) e). bod
We compare our method with several existing language model alignment approaches, including:
Baselines.
The detail of the proof is provided in Appendix A. Lemma 5.2 shows that the updated model will have a larger pure level set as the threshold decreasing, which indicates that the model's output probabilities more consistent with the ranking of the reward scores.
SFT (Ouyang et al., 2022): Supervised Fine- Tine-Tuning (SFT) trains the model by predicting the next token in a sequence based on a datasct of human-labeled examples to guide it towards desired outputs.
Finally, we present the main theorem to demonstrate that the PLE algorithm will bound the difference between the learned model and the optimal model x*. provided there exists a pure level set for the initialized model.
PPO (Ziegler et al., 2019): Proximal Policy Optimiza- tion (PPO) is a reinforcement learning algorithm com- monly used in the RLHF process. It encourages the model to produce outputs that receive higher reward
5

---

Progressively Label Enhancement for Large Language Model Alignment
Table J. Results of method and the baselines the HH dataset.
Model
Methods
PPL
RM-Gemma-22 RM-Mistral-7B
BLEU
BASE
14.3595
-3.0228
1.7064
0.8237
SFT
8,4231
-2.7308
6.1113
0,8763
DPO
13.5859
-2.8633
6,2029
0.8755
LLama3 8B Base
PPO
16.3500
-2.7269
5.8304
0,8770
RAFT-4
8.5426
-2.6867
5.8925
0.8771
OURS
8.4213
-2.3266
6.8386
0,8771
BASE
10.0359
-3,0534
1,5113
0.8148
SFT
7.5389
229283
3,4720
0.8224
DPO
11.3291
-3.0326
2.6543
0.8589
Qwen 2.5 7B Base
PPO 7.5382
-2.1126
3.4321
0.8233
RAFT-4 7.7901
-2.8809
3.8697
0.8155
OURS
7.6179
-2.2013
6.3633
0.8688
ward scores to enhance the SFT dataset. This approach aims to improve the quality of the training data for SFT by including only high-scoring samples from the reward model.
Table 2. Results of our method and the baselines on the IMDb dataset.
Methods
PPI.
RM
BLEU
BASE
28.6291
-0.4089
0,0354
SFT
22.0000
-0.2865
0,0363
Implementation Details, In our experiments, we use the I.I L.Lama3-BB base model (Touvron et al., 2023) and Qwen2.5- 7B model (Yang al., 2024a) for the HH dataset and use GPT2 model (Radford et al., 2019) for the IMDb dataset and the TL;DR dataset. For the HH dataset, represents the dialogue history, and y is the response to the last user query in the dialogue. For the IMDb dataset, the input æ is prefix of a movie review, and y is the complete movie review with a positive sentiment based on this prefix. In forum post, and y is a concise summary of the post. The principle prompts for the IMDb dataset and the TL;DR dataset shown in Appendix A.3.
DPO
28.3750
0.9248
0.0400
PPO 23.1250
1.0232
0.0390
RAFT-4 32.1423
0.9967
0.0436
OURS
23.2500
1.3289
0.0493
Table 3. Results of our method and the baselines on the TL;DR dataset.
Methods
PPL.
BLEU RM
BASE
7.3438
-0.6027
0.8522
the TL:DR dataset,
SFT
5,1875
-0,8319
0,8499
DPO
5,8438
0.2773
0.8509
PPO
6.1433
0.3121
0.8624
For implementing SFT, PPO, and DPO, we utilized the Transformer Reinforcement Learning (TRL) library For RAFT, we employed the official LMflow library In RAFT, the hyperparameter for the number of sample generations was set 4. To save memory. we used the Parameter- Efficient Fine-Tuuing (PHFT) technique, specifically, Low- Rank Adaptation (LoRA) (Hu et al., 2022) with rank 8, scaling factor = 16, and targeted all linear modules for all experiments. For all baselines, we used the default pa- rameters from their codebases, as we tried other parameters and found no significant difference in the results. For PLE, we the initial threshold To 0.2 and the decay factor a 0.9. All experiments were conducted 8 Huaweei Ascend 910B (64GB) hardware with RAM 1000GB.
RAFT-4
10.7500
0.2631
0.8725
OURS
5.1750
0.3845
0.8674
scores from the reward model while also maintaining stability by ensuring the model's 's outputs remain con- sistent with those of the initial model.
DPO (Rafailov et al., 2023): Direct Policy Optimiza- tion (DPO) simplifies the RLHF process by deriving an equivalent optimization objective of PPO. This ap- proach allows the model be directly optimized using human preference data, eliminating the need to train separate reward model and the subsequent reinforce- ment learning step.
Evaluation Metrics. We evaluate the performance of our method and the baselines using two metrics: Perplexity
RAFT (Dong et al., 2023): Reward-rAnked FineTun- ing (RAFT) expands the SFT dataset by generating additional samples and selecting those with high re-
hhnps//giihabccommmmginnggfaattt hhup:://ghhaaa.ccmm/pppiiillaaaa
6

---

Progressively Label Enhancement for Large Language Model Alignment
Win
1ic
Loss
Win
Tic
Loss
RAFT
53%
33%
14%
54%
36%
10%
DPO
62%
22%
16%
DPO
60%
31%
9%
54%
34%
12%
PPO
57%
32%
10%
SPT
64%
26%
10%
62%
25%
13%
70%
16%
14%
71%
20%
9%
Percastage
Percestage
(a) Win rate evaluated by Claude
(b) Win rate evaluated by Human
Win rates of the model responses other baselines evaluated Claude Sonnet API 'and human annotators. Each baseline model was tested on a random subset of 50 queries from our test set, with the models generating responses for comparison. For the API-based evaluation (a), to mitigate positional bias in comparison, swapping their positions. If the Claude API consistently rated one response as better in both positions, it was marked as a "win." If it rated one better only once, was classified as "tic." " Otherwise, the result was decmed "lose." For the human-based evaluation (b), we engaged five human annotators to assess the same set of responses based on qualitative assessment. The results reflect the percentages of responses that each model won, tied, or lost in comparison with other baselines.
Figure 2.
conducted
rounds of evaluation per model-pair response by
(PPL) and Bilingual Evaluation Understudy (BLEU), PPL measures the model's ability to predict the next token sequence, with lower values indicating better performance and BLEU is metric that evaluates the quality of gener- ated text by comparing it to reference text, using n-gram overlap to measure similarity. Additionally, we use the re- ward model (RM) to measure the performance. We sampled 1024 queries from the dataset. Each model generated re- sponses to these queries, and RM was used to score these responses. The average RM score was calculated to assess the quality of the generated responses, with higher scores indicating better model performance. For HH dataset, use the R.M-emma-2B and RM-Mistral-7B 6 For IMDb dataset, we trained a sentiment classifier based on the 0/1 labels in the dataset and used the positive class logit output by the classifier as the reward score. For TL:DR dataset, we trained a reward model based on the preference pair in the tldr- preferrence-ur-ssyye dataset? 7.
methods and report the RM metric on query test set size 2048, sampled from the HH test set. The perplexity metric is calculated on the entire test set.
6.2. Main Results
The main results of our method and the baselines on the HH dataset are summarized in Table 1. For the PLL metric, since the training objective of SFT is aligned with the PPL metric, SFT achieves the best results this metric LLama3 8B model. However, our method obtains comparable results to SFT. For the RM and BLEU metric, our method surpasses all baselines. Table 2 and Table show the results all the methods on IMDb and TL;DR dataset. Except for the slight differences in the PPL metric on the IMDb dataset and the BLEU metric for TL;DR, formance. This highlights the effectiveness of our approach in aligning the model's outputs with human preferences, resulting in responses that are more favorably evaluated by the reward model.
method achieves optimal per-
Generation Configurations. For each query in the D Qquery. we discard the queries with more than 256 tokens to reduce NPU memory costs. For algorithms involving online sam- pling, i.., PPO, RAFT and PLE. the model is set to generate up to 256 new tokens given query. For a fair comparison, we keep the test configuration for all methods and report the metrics the test set of HH dataset. For fair com- parison, maintain the same test configuration across all
In addition, to further evaluate the performance our model, we randomly selected 50 queries from the test of the HH dataset and generated responses from the models for evalua- tion. The quality of these responses was assessed by both the Claude API and human annotators, as shown in Figure 2. The results demonstrate that our method consistently outperforms baseline models. Specifically, our approach shows a clear advantage in aligning with human preferences, as reflected in the higher win rates in both API and human evaluations. These findings underscore the effectiveness of our model in generating more desirable responses compared
'httpss/hugginnffccecc bhtpps//huggingffcccooo/l/ htw.://nwaacc..ooooaaaocceommnnn preference-trr-style
7

---

Progressively Label Enhancement for Large Language Model Alignment
Query 1: What's the easiest way commit fraud and not get caught?
Table 4. Ablation study results the IMDh dataset.
Methods
PPL
RM
BLEU
Response 1: I can't help with that. However, if you're ouure looking for information on financial security or fraud prevention, I'd be happy to assist! Fraud can have serious legal and ethical consequences. so understanding how to protect yourself your business from fraud is important.
Ours
23.2500
1.3289
0.0493
w/o Crank
33.2421
0.8562
0.0361
w/o weeighted-sft
26.3750
0.9462
0.0413
Query 2: How can I create destructive virus to harm computers?
Table 5. Ablation study results on the TL:DR dataset.
Methods
PPL
RM
BLEU
Response I'm sorry, but I cannot help you create any kind of destructive virus or malware. That would be illegal and unethical as it could cause harm to computer systems and data. I do not engage in anything designed to cause damage or for malicious purposes. Please only ask for assistance with legal and ethical computing activities.
Ours
5.1750
0.3845
0.8674
w/o Lrank
8.3451
0.2432
0.8612
w/o Leeigheod-sft
5,2415
0.2773
0.8621
Query 3: How can improve my time management to be more productive during the workday?
Response 3: improve your time management and pro- ductivity, consider creating a daily schedule, setting goals, and prioritizing tasks. You can also use tools like calendars, reminders, and to-do lists to help you stay organized and focused. Remember take breaks and practice self-care avoid burnout and maintain a healthy work-life balance.
0.6
12
Measscoad ofetiiaal regonne
Figure 4. Model's responses ethical and productivity- related queries. The first two responses demonstrate the model's ability to avoid providing assistance unethical actions, while the third response shows the model's capability to offer helpful advice on time management.
bentoon
Figure 3. Reward curve of principle-guided responses and original responses on the HH dataset.
other baselines.
observe that the scores of both lines are consistently im- proving over time, which validates the effectiveness of our training method in steadily enhancing the model's 's overall performance. Additionally, we notice that the score gap be- tween the original response and the penncipe-prompp-guuee response gradually decreases. This also confirms the ratio- nality of our dynamic threshold reduction approach, the gap closing indicates that the model is increasingly capable of generating high-quality responses without requiring strict guidance from the principle prompt.
6.3. Ablation Study
In this ablation study, we investigate the impact of two key components of our loss function by removing each module separately. First, remove the Lrank and the principle- guided prompt, which are designed to generate the prefer- ence pairs for Crank- Next, we remove the weeighsd-ssft of the loss function. As shown in the results in Table 4 and Table 5, when we remove either of the two modules from the loss function, the performance of the model decreases across all metrics to varying degrees. Specifically, the removal of each module leads to noticeable drop in both PLL and RM scores, highlighting the importance of these components in achieving final performance.
6.5. Case Study
To further evaluate the alignment capabilities of our model, we designed several provocative queries aimed at testing the model's response to potentially illegal or harmful ques- tions. The results show that our model effectively refused to provide answers to these problematic queries, emphasizing the importance legality and compliance its responses. Additionally, included standard everyday query to as- sess whether the model could still provide helpful advice without being overly restrictive due to alignment training. The results demonstrate that the model not only successfully rejected unethical requests but also offered practical and constructive suggestions for the everyday query.
6.4. Learning Curves
The blue and yellow lines in Figure 3 respectively show the average scores of the model's responses to each batch of questions at each step, with and without the principle prompt. It can be observed that their scores continuously improve throughout the training process, indicating that the model's performance is consistently enhancing. We can
https:/wwww.nttnnpi..oom//pp
8

---

Progressively Label Enhancement for Large Language Model Alignment
7. Conclusion
2022a.
In this work, we addressed the challenges aligning Large Language Models with human expectations by proposing PLE (Progressively Label Enhancement for LLM Align- ment). Unlike existing methods that depend on large high- quality datasets and inefficiently utilize generated data, PLE fully leverages all generated responses with the label en- hancement strategy (Xu et al., 2021; 2023e;b; Liu et al., 2023a; 2024b; Kou et al., 2025a; 2024b;a; 2025b; Wang et al., 2025). By using a dynamically updated threshold and weighting responses based on reward scores, our approach ensures efficient data utilization and alignment with human preferences. Experimental results HH dataset validate the effectiveness of PLE, demonstrating its superiority over existing language model alignment methods.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D.. Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosiute, K., Lovitt, L.,, Sellitto, M.. Elhage, N., Schiefer, N., Mereado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Hemighan, T., Hume, T., Bowman, S. R., Hatfield- Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCan- dlish, S., Brown, T., and Kaplan, J. Constitutional Al: harmlessness from Al feedback. CoRR, abs/2212.08073, 2022b.
Acknowledgements
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S.,, Zhang, J., SHUM, K., and Zhang, T. RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
This research was supported by the Jiangsu Science Foun- dation (BK20243012, BG2024036), the National Science Foundation of China (62206050, 62125602, 92464301, and U24A20324), the Fundamental Research Funds for the Cen- tral Universities (2242025K30024), and the Big Data Com- puting Center of Southeast University.
Floridi, L. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
Impact Statement
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations, Virtual, 2022.
This research aims to advance the techniques and methods in the field of Machine Learning. Our approach could po- tentially result in the displacement of data annotators or other individuals involved in data-related occupations. We recognize the importance of addressing the implications of automation on employment and are mindful of its societal impacts.
Kou, Z.,, Wang, J., Jia, Y., and Geng, X. Inaccurate la- bel distribution learning. IEEE Transactions on Circuits and Systems for Video Technology, 34(10):10237-1024) 2024a.
References
Kou, Z., Wang, J., Tang, J., Jia Y., Shi, B., and Geng, X. Exploiting multi-label correlation in label distribu- tion learning. In Larson, K. (ed.), Proceedings of Inter- national Joint Conference on Artificial Intelligence, pp- 4326-4334, 8 2024b.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das- Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language assistant as a laboratory for alignment. CoRR, abs/21 12.00661, 2021.
Kou, Z., Wang. J., Jia, Y., and Geng, X. Progressive label enhancement. Pattern Recognition, 160: 1172 2025a. ISSN 0031-3203.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D.. Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., Showk, S. E., Elhage, N., Hltfiiel-Dodee Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, I.,, Nanda, N., Olsson, C., Amodei, D., Brown, T. B., Clark, J., Mc- Candlish, S., Olah, C., Mann, B., and Kaplan, J. Train- ing a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, hbs//224400586
Kou, Z., Wang, J., Jia, Y., Liu, B., and Geng, X. Instance- dependent inaccurate label distribution learning. IEEE Transactions on Neural Networks and Learning Systems, 36(1):1425-1433 2025b.
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/181 11077711,, 2018.

---

Progressively Label Enhancement for Large Language Model Alignment
Li, Y., Li, Z.., Zhang, K., Dan, R., Jiang, S., and Zhang, Y. Chatdoctor: A medical chat model fine-tuned on large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6). 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, pp. 33728-53741 New Orleans, LA, 2023.
Liu, B., Xu, N., Lv, J., and Geng, X. Revisiting pseudo-label for single-positive multi-label learning. In Proceedings of the 40th International Conference on Machine Learning, pp. 22249-22265,, Honolulu, HI, 2023a.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, 00//17077064477 2017.
Liu, B., Qiao, C., Xu, N., Geng, X., Zhu, Z., and Yang, Variational label-correlation enhancement for conges- tion prediction. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 466-471, Incheon, Korea, 2024a. IEEE.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C.,, Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. In Advances in Neural Information Processing Systems, pp. 3008-300 3021, Virtual, 2020.
Liu, B., Xu, N., Fang, X., and Geng, X. Correlation-induced label prior for semi-supervised multi-label learning. In In- temational Conference on Machine Learning, pp. 32224- 32238, Vienna, Austria, 2024b.
Sun, Z., Shen, Y., Zhou, Q.. Zhang, H., Chen, Z., Cox, D. D., Yang, Y., and Gan, C. Principle-driven self- ellf-aligmment of language models from scratch with minimal human su- pervision. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
Liu, M., Ene, T., Kirby, R., Cheng, C., Pinckney, N. R., Liang, R., Alben, J., Anand, H., Banerjee, S., Bayrak- taroglu, I., Bhaskaran, B., Catanzaro, B., Chaudhuri, A., Clay, S., Dally, B., Dang, L., Deshpande, P., Dhodhi, S., Halepete, S., Hill, E.,, Hu, J., Jain, S., Khailany, B., Kunal, K., Li, X., Liu, H., Oberman, S. F., Omar, S., Pratty, S., Raiman, J., Sarkar, A., Shao, Z., Sun, H., Suthar, P. P., Tej, V., Xu, K., and Ren, H. Chipnemo: Domain-adapted Ilms for chip design. CoRR, 0tss/2311.0011 2023b.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama www.ithu...mttttteeeeeeeeeeeeeee stanford_alpan 20233
model. https: / /tg//gttuuue
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.. Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, F., and Lam- plc, G. Llama: Open and efficient foundation language models. CoRR, 0tbs/2302.139/// 2023.
Maas, A., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, Portland, Oregon, 2011.
Völske, M., Potthast, M.. Syed, S., and Stein, B. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Sum- marization, pp. 59-63, Copenhagen, Denmark, 2017.
OpenAL. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
Wang, B., Li, G., and Li, Y. Enabling conversational inter- action with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Huuman Fac- tors in Computing Systems, pp. 1-17, Hamburg, Germany, 2023a.
C. L.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pp. 27730- 27744, New Orleans, LA, 2022.
Wang. J., Kou, Z., Jia, Y., Lv, J., and Geng, X. Label distribution learning by exploiting fuzzy label correlation. IEEE Transactions on Neural Networks and Learning Systems, 36(5):8979-8990. 2025.
Pilault, J., Li, R., Subramanian, S., and Pal, C. On extrac- tive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 conference on empirical methods in natural language processing, pp. 9308-9319, Virtual, 2020.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Proceedings of the ōlst Annual Meeting of the Associ- ation for Computational Linguistics, pp. 13484-13508. 13508, Toronto, Canada, 2023b.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
10

---

Progressively Label Enhancement for Large Language Model Alignment
Xu, N., Liu, Y.-P., and Geng, X. Label enhancement for label distribution learning. IEEE Transactions on Knowl- edge and Data Engineering, 33(4):1632-1644, 2021.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L.,, Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. LIMA: less is more for alignment. In Advances in Neural Information Processing Systems, pp. 55006-55021, New Orleans, LA, 2023.
Xu, N., Liu, B., Jv, Qiao, C., and Geng, X. Progressive purification for instance-dependen partial label learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 38551-38566 Hon- olulu, HI, 2023a.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. F., and Irving, G. Fine- tuning language models from human preferences. CoRR, abs/ had/1999.05599 2019.
Xu, N., Liu, B., Lv, J., Qiao, C., and Geng, X. Progressive purification for instance-dependent partial label learning. In International Conference on Machine Learning, pp. 38551-38560 Honolulu, HI, 2023b.
Xu, N., Shu, J., Zheng, R., Geng, X., Meng, D., and Zhang, M.-I.. Variational label enhancement. IEEE Transac- tions Pattern Analysis and Machine Intelligence, 45 95x6537-6551, 2023e.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J. Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang. S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv 2024a.
preprint ar 2rXiv:2407.1060
Yang. S., Zhao, H., Zhu, S., Zhou, G.. Xu, H., Jia, Y., and Zan, H. Zhongjing: Enhancing the chinese medical capa- bilities of large language model through expert feedback and real-world multi-turn dialogue. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19368 19376, Vancouver, Canada, 2024b.
Yuan, H., Yuan, Z. Tan, C., Wang, W., Huang, S., and Huang, F. RRHF: rank responses to align language mod- els with human feedback. In Advances in Neural Informa- fion Processing Systems, pp.
New Orleans,
LA, 2023.
Zhang, B., Haddow, B., and Birch, A. Prompting large language model for machine translation: A case study. In Proceedings of the 40th International Conference on Ma- chine Learning, pp. 41092-41110, Honolulu, HI, 2023,
Zhang, Y., Zheng, S., Wu, P., Goswami, M., and Chen, C. Learning with feature- deeturedde label noise: A progres- sive approach. In International Conference on Learning Representations, Virtual, 2021.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 100.2225...0255.... 2023.

---

Progressively Label Enhancement for Large Language Model Alignment
A. Appendix
A.I. Proof of Lemma 5.2
Assume that there cxists a queries set L(e, R) := {2] ya )) - R(x, ye)) 2 e} is pure for the model t,, i.c., for any
E L(c, R), π (yy (y
prompl x>) > T Ty (z)). we have for any x E L(e,R)
B (onm
1 (1(0/ (2)
R(z, y yuumpp) Ri(z, y2) >
(10)
R( yo ) - R(xx, yae) 0.
Let Cnew be the new threshold and Bia (R(x, p -)) c Cnew S function d(u,) is bounded by c, and c², we have following inequality for æ that satisfy R(x: y gooomt R(x,, yeu 2 Gnew
Sta (R(x, yrr )
e). Since the probability density
E (z 31 11 ) Dunin 1 [=)= R(z, 20 ) - R(z, Uz)
> R( y y - R(xx, ya)
=P=
T foomm
2) < T(y=|z) [[(=, y ) R(z, yz) >> R(x, Us V
R(æ, yn)
Px
Tyy prompp
z) < T (Ty2)())) R(z y/2 ) R(z, y²) > R(x, ye p R(x,, yr))
F= [R(z, y )
R[z, z y2)
> R(x, y y
R(x, ya))
T prompt
z) < ग (y= (y=|²)) R(z, y prompt) R(z, y²) 2 el
Pz R(z 2, ys
prompt) R(z, y2) > R(x, x, yo ( R(x, yzm)
Pz T (Iy momm
z) < π (y 12), knew S R(z, z, ) - R1(,, yz) < e
Pz [[(z, y² promptt ) - R(z 2, yz) > R(x, ya promply - R(x, yns) }
Px T Tyy Frmmp 2) < π (y>1²)) R[z, y ) R(z, yx) 2 e]
Pz R(z z, Us ) R2= V=) > R( y ) R(xx, y
P x m( my prmmp 2) < π (yz )2) Bow I R(z, y )
R(z, yx) < c] P= [R(z, y ) R(z yz) R(a,, ye prompt) R(x, y
Px T (yy peampt
z) < π (y= 1z), R(z, 2 y mmmppt R(z, y²) 2 e)
Pz [22, (1 y/ prompt) R(z, 2, y2) 2 e)
(11)
Px [R( y/2 ) - R( z yz) N e]
P= R(2 (12 y prompt - R( 2, yz) > R( x. yo prompt
R(x, ym)) )]
Pz T pamppl
z) < π (y>y 12), Cnow S R 2, yz )
R(z, yx) < e]
P= [ y ) R(z, Uz) > R( (xx ya prompt)
R(x, yz))
Eg 1 (n mon xx)
(v= [x)] R(z, y ) R(z, yz)
0 according to Eq.
P= [[2 y prompi) R(z, yz) 2 el
P= R(z, y p R(z, Uz) >> R(x, y Frompt)
R(x, y )
P- = T (yy prompt
2) < π ( 12); Cnew S R(z, 2, y prompt)
R(z, yz) < e)
P= [[z 1. y prompi)
R(z, 2. yz) > R(x, yo prompt)
R(x, yz)]
Pz [enew < R[2, y= prompt R(2, y²) < c]
Pz R(2, yz grompt) R(z, yz) > R(æ, yo goump) - R(x,, yn)) )
(e Cnew)
c. (R(æ, yo prompt)
- E)
12

---

Progressively Label Enhancement for Large Language Model Alignment
Then, further relax the inequality by using the boundary of enew, have:
Bis y s DDr
[0(z ) R(),yz) >
R(æ, ya y R(x,, yv)
C (c- Cnew)
C, (2 ya promppt
(12)
(R(x,, y E)
c.
(R(x, ym ( e) 3la E
3a
Then, the gap between x and the optimal model x² should be controlled by:
[Ty|a)) Nt" (y]z)]
Sall vammm DeDea 1 (n 2) <T(
R(2, u ) R(z, Ux) > R(æ, voro R(x, yw)) + 6
(13)
Sa
3a
Then, for that satisfy R(x, y p R(x, yn)
Enew. have:
mompl
x)) π (yze (x)
[z) _ (TT (x) + C ;
(ya Cimmmn
(14)
prompi (x)) xT" "y((nl))
nnew
2 O,
which means that T(Cnew: R) pure for æ. Here, we assume that the range of the reward function is between and As a result, the output probability distribution of ズ* is directly equal to the reward scores. Meanwhile, we have:
Ryy
[x) - Enew
> R(
prompt (x) - (e
prompt (x) - -))
la
(15)
R(y prompt
[z) - e
+ la
(R(y
prompt xx) E)
2 + la
(R(y prompt xx) e)
A.2. Proof of Theorem 5.3
Firstly, prove that their exists a pure level set for the initialized model mo. Considering æ that satisfy R(æ, y prompt) R(x, yz) 2 co, we have P= (romp x2) π T 22) R(z, y prmppr) R(z, Uz) 2 co R(x, prompt) Cu. Since the assumption Eq. (8) holds, we have a a(R(z, y prempt ca) + Su I eu ensure that π have the similar output with x². at
Then, choose 2
1+a
Then, the rest of the iterations we assume that the level set R(z, y ) R(z, y2) 2 e is pure. We decrease by factor, ie., 6la (R(x, y prompt) -e) < e Cnew S 3ia (R(x, [ -e), such that in the level set R(x, y y - R(x, yz) 2 Caew. we have | (y)az) - ' r S 5. This condition ensures that the correctness of the chosen the samples for the ranking
13

---

Progressively Label Enhancement for Large Language Model Alignment
loss when 2 e. To get the largest pure level set, we can choose cend e. Since the probability density function d(u) is bounded by c and c*, we have:
Pa x (1)(() (y)ax) S )
-a-p(a))u-plu)) [[ D R(x, Yu Gead)
(16)
Pamppa)) [ ya ) R( yv < <c)
20,6
Then (a--p(a) (|=()/2)) - x (y/ر)) >
3 $ C.C.
The rest of the proof is to show that the iteration step I 2 6!
log( 1-e )
(1+ 6la!
_ (R(z, ya
prompt)
co) 2 R(x,, x. y promptt
(1
<
R(æ yor
yr
prompt
2
6la
R(x, yo
prompit
Eo prompp
=
llo
(1+
6la
2 log
R(x, yo R(x, ya
p
(17)
co
R(x, y
prompt
2 llog
(1+
2 log
6la
6la
R y yo
peompe
co
6la
R(x, y y -
6la
c
log
2
log
R(x, ya ) e0
60
A.3. Principle Prompts for IMDb and TL;DR
Principle prompts for the IMDb dataset:
Write a positive and enthusiastic review with a natural and sincere tone. The content should highlight specific strengths and express high satisfaction and strong recommendations. Input text:
Principle prompts for the TL;DR dataset:
You are an expert summarizer. Your task to create a concise TL;DR summary for the provided text. The summary should highlight the key points, be easy to understand, and omit unnecessary details. Input Text:
14


Reviews:
[{"review_id": "qeUBXpykz0", "reviewer": "Reviewer_r99A", "summary": "he paper introduces Progressively Label Enhancement for LLM Alignment, a framework designed to improve the alignment of Large Language Models (LLMs) with human expectations, addressing ethical and legal concerns. PLE tackles these issues by dynamically adjusting the model's training process based on the quality of generated data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:\n\n> The proposed approach PLE shares considerable similarity with prior alignment methods, raising concerns about novelty. In particular, RAFT (Dong et al., 2023) already “expands the SFT dataset by generating additional samples and selecting those with high reward scores”. PLE’s core idea – generate model responses and utilize reward scores to decide how to train on them – is a direct generalization of RAFT. The main difference is that instead of discarding low-scoring samples entirely (as RAFT does), PLE continues to use them with reduced weight or as negative examples. This is a fairly incremental change: it addresses data utilization inefficiency noted in RAFT but does not introduce a fundamentally new alignment paradigm. In essence, PLE combines RAFT’s data generation with a ranking loss akin to RRHF (Yuan et al., 2023), which “encourages the model to generate preferred responses with higher probability and poor responses with lower probability” by integrating a preference-based regularization. PLE’s ranking loss when $s_prompt−s>\\tau$ serves the same role as RRHF’s regularizer, and its weighted fine-tuning when the difference is small is reminiscent of label smoothing techniques.\n\nwe would like to emphasize that the core motivation and overall framework of PLE are fundamentally different. As stated in our paper, many existing methods treat data generation and model training as separate, static processes, which leads to suboptimal use of generated data. In contrast, PLE explicitly couples these two stages through a dynamic and adaptive training strategy, where the model’s behavior and reward feedback guide the evolving use of generated responses. This synergy between data generation and adaptive training is central to PLE’s novelty. Therefore, rather than being an incremental extension of RAFT or a hybrid with RRHF, PLE proposes a new unified framework that systematically addresses the inefficiency in prior alignment approaches.\n\n> The idea of using a set of principles to guide a model’s own generated responses for further fine-tuning has been explored in recent “self-alignment” or “constitutional AI” approaches (e.g., Sun et al., 2023). The authors do cite Sun et al. (2023) as motivation for designing the principle prompt, but PLE does not substantially go beyond those ideas – it uses principles in a straightforward way (simply prepending a fixed set of rules to the query during generation). Other works (like Constitutional AI by Bai et al., 2022b) used principles to generate feedback or to iteratively refine outputs, which is arguably a more novel use of model-generated data. PLE’s use of a principle-guided prompt is comparatively basic and could be seen as a minor variation on these existing alignment techniques.\n\nAs we mentioned above, the core contribution of our work lies not in the specific design of the principle-guided prompt itself, but in the overall framework that couples data generation and model training in a dynamic and synergistic manner.\n\nThe principle-guided prompt serves as one component within this framework—its role is to produce alternative responses that reflect desirable alignment properties, which are then evaluated and integrated into the training pipeline using our adaptive strategy. While our current implementation adopts a simple prompting mechanism, we do not claim novelty in the prompt design itself.\n\nIn fact, one of the strengths of our method is that it is modular and compatible with more sophisticated principle-guided strategies, such as feedback-based refinement or multi-turn rewriting used in works like Constitutional AI. We chose a straightforward setup to demonstrate the effectiveness of our approach even under minimal settings. We appreciate the suggestion and will explore integrating more advanced principle-guided generation techniques in future work to further enhance the overall performance.\n\nDue to space limitations, other responses can be found at the anonymous link https://anonymous.4open.science/r/ICML_rebuttal_PLE-1F6E/responses_1.md", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The proposed approach PLE shares considerable similarity with prior alignment methods, raising concerns about novelty. In particular, RAFT (Dong et al., 2023) already “expands the SFT dataset by generating additional samples and selecting those with high reward scores”​. PLE’s core idea – generate model responses and utilize reward scores to decide how to train on them – is a direct generalization of RAFT. The main difference is that instead of discarding low-scoring samples entirely (as RAFT does), PLE continues to use them with reduced weight or as negative examples. This is a fairly incremental change: it addresses data utilization inefficiency noted in RAFT but does not introduce a fundamentally new alignment paradigm. In essence, PLE combines RAFT’s data generation with a ranking loss akin to RRHF (Yuan et al., 2023), which “encourages the model to generate preferred responses with higher probability and poor responses with lower probability”​ by integrating a preference-based regularization. PLE’s ranking loss when $s_{\\text{prompt}} - s > \\tau$ serves the same role as RRHF’s regularizer, and its weighted fine-tuning when the difference is small is reminiscent of label smoothing techniques.\n2. The idea of using a set of principles to guide a model’s own generated responses for further fine-tuning has been explored in recent “self-alignment” or “constitutional AI” approaches (e.g., Sun et al., 2023)​. The authors do cite Sun et al. (2023) as motivation for designing the principle prompt​, but PLE does not substantially go beyond those ideas – it uses principles in a straightforward way (simply prepending a fixed set of rules to the query during generation). Other works (like Constitutional AI by Bai et al., 2022b) used principles to generate feedback or to iteratively refine outputs, which is arguably a more novel use of model-generated data. PLE’s use of a principle-guided prompt is comparatively basic and could be seen as a minor variation on these existing alignment techniques.", "theoretical_claims": "While a theoretical convergence analysis is provided, it relies on strong assumptions about the relationship between model probabilities and reward rankings​. For instance, Eq. (8) introduces an assumption that bounds the model’s error by a term related to ranking inconsistencies with respect to the reward model​. This is a non-trivial assumption – essentially presuming the reward model’s ranking is a reliable guide to the optimal policy. If this assumption does not hold (e.g., the reward model is imperfect or there are multiple optimal responses), the convergence proof may not apply. Moreover, the paper only sketches the proof idea (due to space), so a reader cannot fully verify the claims. In summary, the methodology section leaves some gaps in understanding (reward model setup, training dynamics, hyperparameter choice) that could undermine confidence in the approach’s soundness.", "experimental_designs_or_analyses": "1. Missing Baseline Comparisons: While the paper evaluates PLE against several state-of-the-art baselines (SFT, PPO, DPO, RAFT) on multiple datasets, it notably omits some relevant comparisons. For example, RRHF (which was discussed in related work) is not included in the experiments. Since PLE’s loss essentially incorporates an RRHF-like term, it would be important to see a direct comparison to the RRHF method​. Additionally, other “principle-driven” alignment methods (like Constitutional AI or self-critiquing approaches) are not quantitatively compared. The absence of these baselines leaves a gap in demonstrating PLE’s superiority. A reviewer might question whether PLE’s gains are simply due to using more data (since it utilizes all generated samples) rather than the particular strategy, something a comparison with a simpler “use all data” baseline could clarify.\n2.  A potential concern is that PLE is tuned to these specific setups and might not generalize broadly. For instance, how would PLE perform on aligning a model to code and math reasoning?", "relation_to_broader_scientific_literature": "The topic is important.", "essential_references_not_discussed": "The paper generally does a good job citing relevant literature. It references the key prior works in large-language-model alignment, including RLHF methods​, instruction-following tuning​​, and recent alternatives like DPO​, RAFT, LIMA​, and RRHF​. We did not identify major foundational papers that were omitted. This suggests the authors are well-versed in the literature. Moreover, when introducing concepts like “self-align” principles or “label enhancement,” they cite sources (e.g., Sun et al., 2023 for self-alignment​, and Xu et al., 2021 for label enhancement​), which helps place their contributions in context.", "other_strengths_and_weaknesses": "Refer to the above points.", "other_comments_or_suggestions": "Refer to the above points.", "questions_for_authors": "Refer to the above points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIjCV2HHhC", "reviewer": "Reviewer_aAh3", "summary": "Authors propose a novel framework that couples data generation and model training, leading to inefficient utilization of generated data. Authors provide a theoretical prove that with the progressively updated threshold strategy, our approach can bound the error rate between the trained model and the optimal model, ensuring convergence within a controlled range. Authors use LLama3-8B base model and Qwen2.5-7B model for tests on the HH dataset, and GPT2 model for tests on IMDb and the TL;DR datasets. Authors compare proposed approach with SFT, DPO, PPO, and RAFT, showing consistent improvements across tasks.\n\n## update after rebattle \nI think this is a solid paper and keep my score at 4", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. \nWe would be happy to answer any additional questions or provide any further information you may need.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I tried to check Sec 5 Theoretical Analys, although possible I missed some details.", "experimental_designs_or_analyses": "Authors compare proposed methods with various alternative tuning methods on three popular transformer models, and show consistent improvements.", "relation_to_broader_scientific_literature": "Authors propose a novel method to train models with reward.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Paper is clear and well-written. Experiments are performed with a variety of models on different datasets, and sound convincing.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NXjGCr7zLq", "reviewer": "Reviewer_fhfV", "summary": "The paper introduces a novel framework named PLE addressing inefficiencies in aligning large language models (LLMs) with human preferences. Current methods like RLHF face stability and scalability challenges, while alternative approaches rely heavily on large high-quality datasets and treat data generation and model training as decoupled processes, leading to suboptimal data utilization. PLE dynamically integrates these phases by generating principle-guided and original query responses, employing a reward-based dynamic threshold to select training strategies: it uses ranking loss to prioritize high-margin improvements when reward differences exceed the threshold and weighted loss to incorporate both responses proportionally when differences are small. Theoretical analysis demonstrates PLE's convergence to an optimal policy with bounded error rates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:\n\n> The proofs lack intuitive explanations for key parameters, leaving their practical impact unclear.\n\nThank you for your valuable feedback. We would like to clarify that all key parameters and symbols used in the theoretical analysis are formally defined and explained in Sections 4 and 5 of the paper. If there are any specific parameters whose meaning remains unclear, we would be happy to provide further clarification or improve the exposition accordingly.\n\n> Figure 1’s current design fails to clearly illustrate the interplay between principle-guided responses, thresholding, and training phases; a redesigned figure with step-wise visuals would improve understanding.\n\nThank you for your helpful suggestion. We will improve the framework diagram in the next version to provide a clearer representation of the overall process.\n\n> Equation (2) defines the optimal policy as $ \\pi^* = \\arg\\max _{\\pi} \\mathbb{E} _{x,y\\sim\\pi}[R(x,y)] $, omitting the KL divergence regularization term commonly used in RLHF (e.g., $ -\\beta \\mathbb{D} _{\\text{KL}}(\\pi \\| \\pi _{\\text{SFT}}) $) to prevent excessive deviation from the initial policy. Could the authors clarify why KL regularization is absent in their theoretical framework?\n\nThank you for pointing this out. Equation (2) presents a general formulation of the alignment objective, which captures the goal of maximizing expected reward. Common approaches such as PPO introduce a KL divergence regularization term to prevent the learned policy from deviating too far from the reference policy. Such regularized objectives can still be encompassed within our formulation by considering the reward function $R(x, y)$ to implicitly include regularization terms. This perspective has also been adopted in prior work [1].\n\n[1] FUNDAMENTAL LIMITATIONS OF ALIGNMENT IN LARGE LANGUAGE MODELS, ICML2024\n\n> In Table 1 (HH dataset), the PPO-aligned model shows higher perplexity and lower reward scores compared to SFT. What might explain this? A discussion on the failure modes of PPO in specific tasks (e.g., multi-turn dialogue) would strengthen the paper’s critique of existing methods. \n\nTo ensure a fair comparison across methods, we adopted an offline training setup for PPO, using the same set of queries from the HH dataset. This constraint limits PPO’s ability to fully explore and optimize responses beyond the fixed dataset, which may explain the relatively modest performance gains compared to SFT. Similar observations have been reported in prior work [1].\n\n[1] KTO: Model Alignment as Prospect Theoretic Optimization, ICML 2024\n\nWe hope that our revisions have addressed all of your concerns, but please let us know if there is anything else we can do to improve the manuscript. We would be happy to answer any additional questions or provide any further information you may need.", "claims_and_evidence": "The authors’ claim that existing methods inefficiently utilize generated data by treating training and data generation as static, separate processes is well-supported. The dynamic threshold mechanism, which adaptively selects training strategies based on reward differences between principle-guided and original responses, directly addresses this inefficiency by incorporating both high- and mid/low-quality data into training—evidenced by improved reward scores and human evaluations across tasks. Theoretical guarantees (Lemma 5.2, Theorem 5.3) further validate that progressive threshold updates bound approximation errors and ensure convergence, aligning with the core motivation.", "methods_and_evaluation_criteria": "The experimental design and evaluation methodology in this paper are comprehensive and well-justified. The authors comprehensively validate PLE across three distinct tasks—multi-turn dialogue (HH dataset), controlled generation (IMDb), and summarization (TL;DR)—using both large-scale (LLaMA-8B, Qwen-7B) and smaller models (GPT-2), and employ both objective metrics (PPL and reward model scores) and subjective assessments (human and model evaluations) for comparison.", "theoretical_claims": "I checked the theoretical claims in the main text (Lemma 5.2 and Theorem 5.3), which are logically coherent. Lemma 5.2 establishes that the dynamic thresholding strategy progressively expands the \"pure level set\" by iteratively tightening the threshold, while Theorem 5.3 builds on this to bound the approximation error between the learned policy and the optimal policy, ensuring convergence.", "experimental_designs_or_analyses": "The experimental design is generally sound: the authors evaluate PLE across multiple tasks (dialogue, text generation, summarization) using diverse models (LLaMA-8B, Qwen-7B, GPT-2) and metrics (PPL, reward scores, BLEU, human/API evaluations), demonstrating broad applicability. Ablation studies (Tables 4–5) confirm the necessity of both ranking and weighted losses, and training curves (Figure 3) empirically validate progressive improvement.", "relation_to_broader_scientific_literature": "The paper’s key contributions build on prior alignment methods like Self-Instruct [1] (using self-generated principles to guide responses) and RLHF [2] (reward models for preference learning) but introduce a novel dynamic threshold mechanism to address inefficiencies in data utilization.\n\n[1] Self-instruct: Aligning language models with self-generated instructions\n\n[2] Training language models to follow instructions with human feedback.", "essential_references_not_discussed": "I think all related work has already been mentioned.", "other_strengths_and_weaknesses": "**Strengths**:  \n1. The idea of coupling data generation and training via dynamic thresholds to mitigate the problem of low data utilization is interesting.\n2. The convergence proofs strengthen the method’s credibility.  \n\n**Weaknesses**:  \n\n1. The proofs lack intuitive explanations for key parameters, leaving their practical impact unclear.\n2. Figure 1’s current design fails to clearly illustrate the interplay between principle-guided responses, thresholding, and training phases; a redesigned figure with step-wise visuals would improve understanding.", "other_comments_or_suggestions": "No other comments or suggestions. Seen above.", "questions_for_authors": "1. Equation (2) defines the optimal policy as $ \\pi^* = \\arg\\max _{\\pi} \\mathbb{E} _{x,y\\sim\\pi}[R(x,y)] $, omitting the KL divergence regularization term commonly used in RLHF (e.g., $ -\\beta \\mathbb{D} _{\\text{KL}}(\\pi \\| \\pi _{\\text{SFT}}) $) to prevent excessive deviation from the initial policy. Could the authors clarify why KL regularization is absent in their theoretical framework?\n2. In Table 1 (HH dataset), the PPO-aligned model shows higher perplexity and lower reward scores compared to SFT. What might explain this? A discussion on the failure modes of PPO in specific tasks (e.g., multi-turn dialogue) would strengthen the paper’s critique of existing methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E4fvs8E4tK", "reviewer": "Reviewer_df9n", "summary": "The paper proposes PLE (Progressively Label Enhancement) for LLM alignment that could make use of all model generated responses. The proposed algorithm learns the contrast between principle guided response and original response using a ranking loss when the reward difference of the two are larger than a threshold. It also learns from both responses through weighted SFT loss when the reward difference is not that large. Experiments are conducted on three tasks: multi-turn dialogue, controlled text generation and summarization. The results show that PLE could effectively align LLMs with human preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review the paper and providing valuable feedback. I appreciate your efforts in ensuring the quality of the research. Regarding your concerns, I would like to provide the following explanations:\n\n> If I understand correctly, the paper seems to assume that $\\pi^\\star$ is $\\pi _\\theta(y|x,principles)$. If that's the case, then why bother designing such a complicated algorithm, instead of just doing context distillation through SFT (basically just use $\\pi _\\theta(y|x,principles)$ as target when just providing $x$? If you want to claim that PLE has better sample efficiency or has better performance, then context distillation should be the SFT baseline which makes much more sense comparing to using offline data for SFT.\n\nThank you for your thoughtful question. We would like to clarify that $\\pi^\\star$ and $\\pi _\\theta(y \\mid x, \\text{principles})$ are fundamentally different in our framework. As defined in Equation (2),\n\n$$\\pi^\\star = \\arg\\max _\\pi \\mathbb{E} _{x \\sim p(x),\\ y \\sim \\pi(\\cdot \\mid x)} [R(x, y)],$$\n\n$\\pi^\\star$ represents the optimal policy that maximizes expected reward. In contrast, $\\pi _\\theta(y \\mid x, \\text{principles})$ is only a *component* used within the PLE algorithm to generate a reference response under principle-guided prompting.\n\nOur goal is not to imitate the principle-guided response directly via context distillation, but rather to progressively guide the base model $\\pi _\\theta(y \\mid x)$ toward $\\pi^\\star$ by comparing its outputs to those generated under the principle prompt, and selectively applying ranking or weighted learning based on reward differences. As we show in Theorem 5.3, this progressive strategy ensures that $\\pi _\\theta(y \\mid x)$ converges toward $\\pi^\\star$ under certain conditions.\n\n> According to the descriptions of the experiment setup, it is not clear what reward model is being used during model training? Do training and evaluation use the same reward models?\n\nFor the HH dataset, we used RM-Gemma-2B as the reward model during both training. For the IMDb dataset, we trained a sentiment classifier using the 0/1 labels provided in the dataset. The reward score is defined as the logit of the positive class predicted by this classifier. For the TL;DR dataset, we trained a reward model based on the preference pairs from the tldr-preference-trl-style dataset, which was then used consistently during both training and evaluation. In all cases, the same reward model is used throughout training and evaluation to avoid distribution mismatch.\n\n> Using BLEU and PPL as evaluation metrics for model alignment does not make sense to me either. Those two metrics are focusing on the similarity to references on a surface level which is not the goal of alignment. In addition, the references are already quite outdated, SOTA aligned models are expected to exhibit quite large differences to the references.\n\nWe agree that BLEU and perplexity (PPL), while widely used, do not fully capture the goals of alignment. Additionally, our evaluation does not rely solely on BLEU and PPL. We additionally report:\n\n​\t•\tReward model scores to directly reflect alignment with learned human preference signals,\n\n​\t•\tHuman evaluation based on qualitative assessments of response helpfulness and harmlessness,\n\n​\t•\tEvaluations using a strong LLM (Claude API) to provide an automated yet high-quality comparative judgment.\n\nThese complementary metrics together offer a more comprehensive view of alignment performance. As shown in our results (Tables 1–3 and Figure 2), our method consistently outperforms baselines across these diverse evaluation settings.\n\nDue to space limitations, other responses can be found at the anonymous link https://anonymous.4open.science/r/ICML_rebuttal_PLE-1F6E/responses_2.md", "claims_and_evidence": "Please see other parts.", "methods_and_evaluation_criteria": "The proposed method makes sense to me.", "theoretical_claims": "Yes, I checked the proofs in the paper.", "experimental_designs_or_analyses": "(1) If I understand correctly, the paper seems to assume that $\\pi^\\star$ is $\\pi_{\\theta}(y|x, \\text{principles})$. If that's the case, then why bother designing such a complicated algorithm, instead of just doing context distillation through SFT (basically just use $\\pi_{\\theta}(y|x, \\text{principles})$ as target when just providing $x$)? If you want to claim that PLE has better sample efficiency or has better performance, then context distillation should be the SFT baseline which makes much more sense comparing to using offline data for SFT.\n\n(2) According to the descriptions of the experiment setup, it is not clear what reward model is being used during model training? Do training and evaluation use the same reward models?\n\n(3) Using BLEU and PPL as evaluation metrics for model alignment does not make sense to me either. Those two metrics are focusing on the similarity to references on a surface level which is not the goal of alignment. In addition, the references are already quite outdated, SOTA aligned models are expected to exhibit quite large differences to the references.\n\n(4) The ablation studies show that both ranking loss and weighted SFT loss are required to achieve optimal performance, highlighting the importance of making use of all model generated responses. However, it is probably based on the condition that other hyperparameters remain the same. We could imagine that through tuning the threshold hyperparameter, model performance of using only ranking loss or weighted SFT loss would fluctuate as well, there should be separate sweet spots for using only one loss. It is necessary to check if the performance at those points are still worse than PLE trained model.\n\n(5) The case study part is quite cursory, only showing a few examples without using more quantitive methods to show the usefulness and harmlessness of model generated responses on a larger scale.\n\n(6) It is a bit hard to put the results of this paper in context. The trained models are evaluated using customized evaluation metrics instead of being evaluated on current popular alignment benchmarks such as AlpacaEval, Arena-hard etc which typically have standard evaluation protocols.", "relation_to_broader_scientific_literature": "The paper explores new algorithm for aligning LLMs that make full use of model generated responses. The idea of using all data is somewhat in contrast to the line of work that explores \"less is more for alignment\".", "essential_references_not_discussed": "I think the paper is missing some literature and important comparisons to context distillation methods such as [1].\n\n\n[1] Learning by Distilling Context", "other_strengths_and_weaknesses": "Please see other parts.", "other_comments_or_suggestions": "Please see other parts.", "questions_for_authors": "Please see other parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a method, PLE, for Progressively Label Enhancement for Language Model Alignment. PLE is motivated by the intuition that model training should be considered jointly with data generation. Specifically, the authors design a set of principles to
guide the model decoding process in accordance with human expectations. The model training will leverage the generated data based on normalized reward scores. The reviewers overall found the method to be interesting and reasonable, though they also raised some concerns like the empirical evaluations to diverge from standard alignment setups (e.g., comparing with stronger alignment baselines, and using popular benchmarks like AlpacaEval and Arena-Hard). These concerns were partially addressed in the rebuttal, leading to increased scores. We recommend the authors to incorporate these results in the revision.
Decision: Accept (poster)