Title: Towards Faithful and Controllable Personalization via Critique-Post-Edit Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
Keywords: Personalization, Reinforcement Learning, LLMs

Abstract: Faithfully personalizing large language models (LLMs) to align with individual user preferences is a critical but challenging task. While supervised fine-tuning (SFT) quickly reaches a performance plateau, standard reinforcement learning from human feedback (RLHF) also struggles with the nuances of personalization. Scalar-based reward models are prone to reward hacking which leads to verbose and superficially personalized responses. To address these limitations, we propose \textbf{Critique-Post-Edit}, a robust reinforcement learning framework that enables more faithful and controllable personalization. Our framework integrates two key components: (1) a \textbf{Personalized Generative Reward Model (GRM)} that provides multi-dimensional scores and textual critiques to resist reward hacking, and (2) a \textbf{Critique-Post-Edit} mechanism where the policy model revises its own outputs based on these critiques for more targeted and efficient learning. Under a rigorous length-controlled evaluation, our method substantially outperforms standard PPO on personalization benchmarks. Personalized Qwen2.5-7B achieves an average 11\% win-rate improvement, and personalized Qwen2.5-14B model surpasses the performance of GPT-4.1. These results demonstrate a practical path to faithful, efficient, and controllable personalization.

TL;DR: We propose a method to enable faithful and controllable personalization of LLMs.