Towards Faithful and Controllable Personalization via Critique-Post-Edit
  Reinforcement Learning
Authors: name: Chenghao Zhu; name: Meiling Tao; name: Tiannan Wang; name: Dongyi Ding; name: Yuchen Eleanor Jiang; name: Wangchunshu Zhou
Published: 2025-10-21 17:40:03

Abstract: Faithfully personalizing large language models (LLMs) to align with
individual user preferences is a critical but challenging task. While
supervised fine-tuning (SFT) quickly reaches a performance plateau, standard
reinforcement learning from human feedback (RLHF) also struggles with the
nuances of personalization. Scalar-based reward models are prone to reward
hacking which leads to verbose and superficially personalized responses. To
address these limitations, we propose Critique-Post-Edit, a robust
reinforcement learning framework that enables more faithful and controllable
personalization. Our framework integrates two key components: (1) a
Personalized Generative Reward Model (GRM) that provides multi-dimensional
scores and textual critiques to resist reward hacking, and (2) a
Critique-Post-Edit mechanism where the policy model revises its own outputs
based on these critiques for more targeted and efficient learning. Under a
rigorous length-controlled evaluation, our method substantially outperforms
standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an
average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses
the performance of GPT-4.1. These results demonstrate a practical path to
faithful, efficient, and controllable personalization.