Title: Preference Distillation via Value based Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Preference distillation

Knowledge distillation

LLM with RL
Abstract: Direct Preference Optimization (DPO) is a powerful paradigm to align language models with human preferences using pairwise comparisons. 
However, its binary win-or-loss supervision often proves insufficient for training small models with limited capacity.
Prior works attempt to distill information from large teacher models using behavior cloning or KL divergence.
These methods often focus on mimicking current behavior and overlook distilling reward modeling.
To address this issue, we propose \textit{Teacher Value-based Knowledge Distillation} (TVKD), which introduces an auxiliary reward from the value function of the teacher model to provide a soft guide. 
This auxiliary reward is formulated to satisfy potential-based reward shaping, ensuring that the global reward structure and optimal policy of DPO are preserved. 
TVKD can be integrated into the standard DPO training framework and does not require additional rollouts. 
Our experimental results show that TVKD consistently improves performance across various benchmarks and model sizes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Teacher Value-based Knowledge Distillation (TVKD), a novel method for aligning small language models with human preferences by leveraging the value function of a larger, DPO-aligned teacher model. Unlike traditional approaches that rely on coarse binary feedback or behavior cloning, TVKD uses the teacher's value function to provide fine-grained, sequence-level reward signals via Potential-based Reward Shaping. It is a fully offline method that requires no additional rollouts. Experiments demonstrate TVKD's superiority over baselines across multiple alignment benchmarks like MT-Bench and AlpacaEval.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Authors use the teacher model's value function as a soft reward signal, effectively address the coarse binary feedback limitation of standard DPO. They frame this as a Potential-based Reward Shaping (PBRS) problem, which bypass the need for online rollouts.\n2. Authors provide theoretical proofs and analysis on the method, which establishes strong theoretical guarantees.\n3. Experimental results show the effectiveness of this method on several models and benchmarks.\n\nWeakness:\n\n1. The paper could benefit from more discussion of the relationship between the hyperparameter $\\beta$ in DPO and the distillation strength $\\alpha$, as these parameters likely interact in important ways.\n2. While the paper highlights the advantage of being an offline method compared to online approaches like ADPA, it does not quantify the computational overhead TVKD introduces over standard DPO. It would be helpful to include a direct comparison on training time and resource usage.\n3. Compared to MT-Bench and Alpaca Eval, Arena Hard may be a more suitable benchmark nowadays.\n\nQUESTIONS:\nThe use of a value function as a shaping reward introduces a risk of reward hacking, where the student learns to exploit artifacts in the teacher’s value estimates rather than genuinely improving preference alignment. What are authors' opinions on this aspect?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Teacher Value-based Knowledge Distillation (TVKD) to enhance preference alignment in small language models by leveraging the value function of large language models. Direct Preference Optimization (DPO) relies on coarse-grained binary feedback, which is insufficient for small models with limited capacity. Existing distillation methods (e.g., KL divergence) mimic teacher behavior but overlook reward modeling, risking misalignment with human preferences. TVKD introduces potential-based reward shaping (PBRS) by treating the teacher model’s value function as an auxiliary reward signal, preserving DPO’s optimal policy while providing fine-grained guidance. Experiments across benchmarks MT-Bench, AlpacaEval, Open LLM Leaderboard show consistent gains over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Theoretically sound: Leverages token-level optimal value of soft value-based RL (Lemma 1), with an invariance guarantee of the optimal policy under PBRS (Lemma 2), avoiding issues of action-dependent shaping (Corollary 2.1).\n* Empirical performance: Comprehensive experiments on diverse model sizes (Table 2) and datasets (DPO-MIX-7K, HelpSteer2) demonstrate TVKD’s superiority over 14 baselines across RM, MT-Bench, AlpacaEval, and Open LLM Leaderboard metrics.\n\nWeaknesses:\n* reward hacking risk from teacher bias: Distillation from a DPO-trained teacher, whose \"optimal\" policy might contain spurious biases from preference data, increases the risk of transferring these biases and impairing student generalization on out-of-domain datasets.\n* teacher model \"optimality\" assumption: The \"optimality\" of the teacher's value function is derived from its training on potentially imperfect human preference data, which might not perfectly reflect true human preferences, thus limiting the distilled signal.\n\nQUESTIONS:\nQuestion:\n* How can the potential for transferring spurious biases from the teacher model be further quantified or mitigated? Have alternative methods been explored, such as teacher ensembles or teachers trained with more robust alignment techniques?\n* Beyond current benchmarks, how does TVKD's generalization hold on highly out-of-domain or adversarial datasets to truly assess its robustness against reward hacking?\n* A more detailed analysis of the hyperparameter α's sensitivity is needed. How does its tuning affect the balance between teacher guidance and student learning from direct DPO, and what are practical tuning guidelines?\n* Can the TVKD framework be generalized or adapted to distill from teachers aligned using other RLHF methods, such as those employing an explicit reward model (e.g., PPO-trained teachers)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Teacher Value-based Knowledge Distillation (TVKD), a method to distill preference alignment from a teacher model. Instead of focusing solely on behavior cloning or KL divergence, TVKD incorporates a value-based auxiliary reward that adheres to potential-based reward shaping principles, preserving the global reward structure and optimal policy. The approach integrates seamlessly into standard DPO training without requiring additional rollouts. Experimental results show that TVKD yields consistent performance gains across multiple benchmarks and model sizes.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The authors provide detailed theoretical analyses supporting the design of their method, which enhances its credibility and soundness.\n2. The paper includes comprehensive experiments with a wide range of baseline comparisons, demonstrating thorough empirical evaluation.\n\n**Weakness**\n1. According to the Qwen3 Technical Report, SOTA models typically undergo multiple preference optimization stages beyond DPO alone. This raises concerns about whether a DPO-based distillation method can sufficiently capture the full spectrum of preference knowledge from a powerful teacher model.\n\n2. The proposed reward shaping technique using the teacher's value function appears similar to Equation 10 in the paper “From r to Q*: Your Language Model is Secretly a Q-Function.” The authors should clarify the conceptual and practical differences between their approach and that prior work to avoid ambiguity and better position their contribution.\n\n3. Experimentally, the proposed method does not outperform the ADAP baseline by a significant margin. Given that evaluation relies on LLM-based judges—known to exhibit randomness and occasional misalignment with true human preferences—it is unclear whether the observed improvements are robust.\n\nQUESTIONS:\nWhich theoretical results or proofs are original contributions by the authors, rather than adaptations or direct reuse from prior work?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of preference alignment for small language models (SLMs) by proposing a novel knowledge distillation approach termed Teacher Value-based Knowledge Distillation (TVDK).  \n\nWhile current mainstream methods such as Direct Preference Optimization (DPO) align models using binary comparisons, the resulting coarse-grained signals may be insufficient for small models with limited capacity. Traditional knowledge distillation approaches primarily focus on mimicking the behavior of the teacher model, neglecting the richer internal reward modeling contained within it.\n\nThe core idea of TVKD lies in moving beyond mere imitation of the teacher’s output, instead distilling the teacher’s internal \"value judgments.\" This is achieved by introducing an auxiliary reward term that satisfies the theory of Potential-Based Reward Shaping (PBRS). As a result, the student model receives more fine-grained guidance while the original optimization objective and optimal policy of DPO remain unchanged.\n\nSTRENGTHS AND WEAKNESSES:\nThe method's principal advantage is its theoretically-grounded framework that distills a teacher model's internal value function via potential-based reward shaping, thereby enriching the student's learning with fine-grained supervision while provably preserving the optimal policy. Conversely, its core limitation is the fundamental dependence on the teacher model's quality, as any inaccuracies or biases inherent in the teacher's approximated value function are directly propagated to the student, capping performance and potentially transferring undesirable behaviors.\n\nQUESTIONS:\n1. On what data was the teacher model trained using DPO?\nThe manuscript mentions that the teacher model πϕ is one that has \"already been trained using DPO.\" But the specifics of this training process are quite vague.\n2. Definition and Interpretation of \"Accuracy\" in Value Function Analysis (Table 3).\nIn the analysis presented in Section 4.3, Table 3 showcases a metric named \"Accuracy(%)\", which is utilized to demonstrate that its shaping term can better identify preferred trajectories. Nevertheless, both the implication and the results of this metric are perplexing.The accuracy of the \"Ours\" method indicated in the paper is 25.82%. This figure, on its own, appears remarkably low, making it difficult from an intuitive standpoint to regard as \"accurate\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}