ICML 2025

Progressively Label Enhancement for Large Language Model Alignment
Biao Liu 1 2 Ning Xu 1 2 Xin Geng 1 2
Abstract
Large Language Models (LLM) alignment aims to
prevent models from producing content that mis-
aligns with human expectations, which can lead
to ethical and legal concerns. In the last few years,
Reinforcement Learning from Human Feedback
(RLHF) has been the most prominent method for
achieving alignment. Due to challenges in sta-
bility and scalability with RLHF stages, which
arise from the complex interactions between mul-
tiple models, researchers are exploring alternative
methods to achieve effects comparable to those
of RLHF. However, these methods often rely on
large high-quality datasets. Despite some meth-
ods considering the generation of additional data
to expand datasets, they often treat model training
and data generation as separate and static pro-
cesses, overlooking the fact that these processes
are highly interdependent, leading to inefficient
utilization of the generated data. To deal with
this problem, we propose PLE, i.e., Progressively
Label Enhancement for LLM Alignment, a frame-
work that dynamically adjusts the model’s train-
ing process based on the evolving quality of the
generated data. Specifically, we prompt the model
to generate responses for both the original query
and a set of carefully designed principle-guided
query, and then utilize a dynamic threshold to
determine the appropriate training approach for
both responses based on their corresponding re-
ward scores. Experimental results demonstrate the
effectiveness of PLE compared to existing LLM
alignment methods.
1School of Computer Science and Engineering, South-
east University, Nanjing, China.
E-mail: {liubiao01, xning,
xgeng}@seu.edu.cn. 2Key Laboratory of New Generation Artifi-
cial Intelligence Technology and Its Interdisciplinary Applications
(Southeast University), Ministry of Education, China. Correspon-
dence to: Ning Xu <xning@seu.edu.cn>.
Proceedings of the 42 n...