Preference Distillation via Value based Reinforcement Learning
Authors: Minchan Kwon
Junwon Ko
Kangil Kim
Junmo Kim
Venue: NeurIPS 2025 poster ()
Keywords: Preference distillation
Knowledge distillation
LLM with RL
Primary area: applications
TL;DR: Using teacher value function and PBRS, propose a theoretically grounded method for preference distillation

Abstract: Direct Preference Optimization (DPO) is a powerful paradigm to align language models with human preferences using pairwise comparisons. 
However, its binary win-or-loss supervision often proves insufficient for training small models with limited capacity.
Prior works attempt to distill information from large teacher models using behavior cloning or KL divergence.
These methods often focus on mimicking current behavior and overlook distilling reward modeling.
To address this issue, we propose \textit{Teacher Value-based Knowledge Distillation} (TVKD), which introduces an auxiliary reward from the value function of the teacher model to provide a soft guide. 
This auxiliary reward is formulated to satisfy potential-based reward shaping, ensuring that the global reward structure and optimal policy of DPO are preserved. 
TVKD can be integrated into the standard DPO training framework and does not require additional rollouts. 
Our experimental results show that TVKD consistently improves performance across various benchmarks and model sizes.

Preference Distillation via
Value based Reinforcement Learning
Minchan Kwon
Junwon Ko
Kangil Kim
Junmo Kim
Korea Advanced Institute of Science and Technology (KAIST)
Gwanggj Institute of Science and Technology (GIST)
(kmc0207, jojuuewon, jumoo kim) ac.kr, kikim01@gist..ac.nt
Abstract
Direct Preference Optimization (DPO) a powerful paradigm to align language models with human preferences using pairwise comparisons. However, its binary win-or-loss supervision often proves insufficient for training small models with limited capacity. Prior works attempt to distill information from large teacher models using behavior cloning or KL divergence. These methods often focus on mimicking current behavior and overlook distilling reward modeling. To address this issue, we propose Teacher Value-based Knowledge Distillation (TVKD), which introduces auxiliary reward from the value function of the teacher model provide a soft guide. This auxiliary reward is formulated to satisfy potential-bart reward shaping, ensuring that the global reward structure and optimal policy of DPO are preserved. TVKD can be integrated into the standard DPO training framework and does require additional rollouts. Our experimental results show that TVKD consistently improves performance across various benchmarks and model sizes.
Introduction
Direct Preference Optimization (DPO) 2 is a powerful paradigm for aligning large language models (LLMs) with human preferences. Unlike Reinforcement Learning with Human Feedback (RLHF) 22), DPO enables efficient learning through pairwise comparisons, without requiring additional rollouts and reward model training. However, since DPO supervision is provided only in terms of binary win-or-lose preferences over entire sequences, it becomes difficult evaluate how good given response is. This coarse-grand caaseegrained feedback offers limited insight into the magnitude or quality of preference, making challenging assess fine-grained improvements or to interpret alignment quality beyond simple pairwise comparison. In small language models (SLMs) which have limited capacity and language understanding, this issue becomes more crucial 9 [6].
To address this issue, several methods have attempted to leverage the rich information from large teacher models through diverse ways such KL-divergence [6] online rollouts 30 2]
However, most prior works focus on imitating the behavior of the teacher, without modeling its long- term reward structure. This issue has been widely discussed in the inverse RL [ [33 21 literature. where it is known that mimicking behavior without reward modeling can lead to compounding errors and poor generalization. Moreover, prior methods often disregard the potential conffict between the reinforcement learning (RL) objective and teacher information, leading to misalignment with the intended preference signal in datasets. For example, simply integrating KL--divergence with the
CCo-supervising authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

teacher in the RL objective may result in conflict between following the teacher policy and optimizing for human preferences in the offline dataset. This undermines the original intent of DPO, which is to align policies with human preferences expressed in data.
In this work, we propose Teacher Value-based Knowledge Distillation (TVKD), method that integrates soft reward labels from the teacher without interfering with the DPO reward structure and its global optimal policy. The key idea is to introduce a novel auxiliary reward term that leverages the value function of the teacher. These value functions provide estimates of the value function that serves as a proxy for the internal reward modeling of the teacher model. We add this value function to the RL objective in a form that satisfies Potential -baeed Reward shaping (PBRS) [20]. This formula theoretically guarantees that the comparison of the action-level Q-functions will not change, maintaining the optimal policy. We introduce these value functions as a soft sequence-level reward, enabling DPO to account not just for which sequence wins, but by how much. This adds more informative supervision to the originally binary feedback. Practically, it can be integrated into the DPO loss with minimal modification, providing a simple yet effective way to incorporate internal signals from the teacher model.
We demonstrate that TVKD achieves strong performance across multiple benchmarks for preference distillation, including MT-Bench (3I) Alpacalival [6], and the Open LLM Leaderboard [8]. Notably, our method requires no additional rollouts and only leverages teacher outputs on an existing offline DPO dataset, making it compatible with the standard training frameworks of DPO. In addition, we demonstrate that TVKD performs consistently across student models ranging from 0.5B 3B parameters, and remains robust under wide range of ablation.
Our main contributions are summarized as follows:
We introduce Teacher Value-based Knowledge Distillation, a preference distillation method that leverages the value function of the teacher as a proxy for its reward model.
We present a form of auxiliary reward term that satisfies potential based reward shaping, suggesting method for adding teacher information without conflict with the original DPO reward structure.
We empirically validate TVKD on multiple preference benchmarks, demonstrating consistent improvements across various student model sizes and datasets.
2
Related Work
Preference Optimization Preference optimization is a training framework for aligning LLM outputs with human preferences. The standard approach, Reinforcement Learning from Human Feedback [22), involves training a reward model and fine-tuning the policy using policy gradient methods such as proximal policy optimization (26.. However, due the high cost and instability of these online settings, offline alternatives have become popular in recent work. DPO (24) eliminates the reward modeling phase by directly optimizing the log-ratio margin between preferred and dispreferred responses. Extensions such as SimPO[[9] and WPO [32 further address issues such as length or sampling bias. To overcome the limitations of relying on sequence-level preference signals, recent work has proposed incorporating token-level supervision into preference optimization [7 29). In particular, (25) shows that -PO-trained models implicitly learn a soft Q-function over token-level actions, revealing potential path for finer- mmer-grained alignment.
Preference Distillation Preference distillation is a technique that uses knowledge distillation (KD) [1] for preference optimization. Preference distillation can be categorized into online and offline settings. In online settings, rollouts from either the teacher or student model can be used to guide training. PaLD 30) uses DPO with teacher responses as the selected answers and student responses as rejected answers. DPKD [16] use in the same setting from PaLDD, but replaces the reference term of DPO with the KL-divergence of teacher model, encouraging the student to follow the teacher closely. ADPA (9) leverages the difference between to generate feedback from student rollouts, which used to train the student via policy gradient. While effective, such online methods often suffer from computational overhead and stability issues due to sampling during training. Offline preference distillation offers the advantage of eliminating rollouts during training, as it learns from a pre-collected dataset of human preferences. DCKD [9]
and SFT-trained teachers

---

minimizes KL divergence between student and teacher outputs on both preferred and dispreferred responses. DDPO 7 makes the reward margin equalize between the student and teacher, which changes the RI objective to squared form. However, the offline setting has conflicts between the internal dataset reward and teacher reward, and stability concerns because it learns an off off-policy environment. We propose to solve this problem by using the value function of the teacher as a soft target for the first time to the best our knowledge
Reward Shaping in Reinforcement Learning Reward shaping is a widely studied technique in RL for accelerating training and improving sample efficiency by providing additional, informative feedback beyond sparse environmental rewards. Potential-based reward shaping, introduced by Ng et al. (20], offers a thcoretical guarantee that shaping terms can guide learning without altering the optimal policy. Subsequent work extended this idea various RL settings, including temporal difference methods (28), partially observable MDPs [5), and multi-agent coordination [4] We adapt this idea of potential-base shaping to preference distillation for language models.
3 Method
In this section, we introduce our method, Teacher Value-based Knowledge Distillation (TVKD). We first define language generation as token-level Markov Decision Process (MDP), and re-visit value-based RL and DPO. We then define value function from the DPO-trained teacher model, and incorporate it into the RL objective. We present derivation of our final loss from the RL objective, along with an analysis of its behavior.
3.1 Preliminary
Token-level MDP for Large Language Model We formulate the language generation task as token-level Markov Decision Process (MDP). Unlike the view of text generation as static decision problem (e.g., multi-armed bandit), this formulation allows us to model the token-wise policies and their optimization over trajectories. Formally, We define the token-level MDP tuple M (S, A. f. Po). where the state space S consists of all tokens generated so far and the action space A is the vocabulary V of tokens. The dynamics f are the deterministic transition model between the tokens f(s, a) = s/a, where is concatenation. The initial state distribution Po is a distribution prompts x, where initial state Sy is comprised of the tokens from x. The reward function r(s, a) assigns a scalar reward to each state-action pair. This MDP structure provides a natural way to analyze and improve the generation policy To sequences by optimizing token-level decisions. Following previous token-level MDP settings, [24] 25 we fix the discount factor y = 1.
Value-based RL To find policy that maximizes the reward in given token-level MDP, can use the following objective:
max ENST
Σ D(ss, + +
where H(((-Iss)) = - (a|s;) log (a]s;) is the Shannon entropy of the policy and 3 balances the reward and exploration. This objective is called the Maximum Entropy RL (MaxEnt RL) objective.
Value- based RL offers framework for solving MaxEnt RL problems by learning value functions that estimate expected future rewards. These functions act as surrogates for long-term returns and enable efficient policy improvement through dynamic programming or bootstrapping.
In this context, the state value function V (*))s is defined as the expected cumulative reward when starting from state and following policy π.:
V = Ex
Σ x ((ss)))))
where 7 E [0, the action:
is a discount factor. The action value function (= a) additionally conditions on
(s) a() + E., (s)].
3

---

In the MaxEnt RL framework, the optimal state value function satisfies the soft Bellman consisteney condition as follows:
(*( = log Σ exp (s,a))
which corresponds to the partition function over the exponentiated Q-values, where Q"and V* is optimal soft Q and value functions.
can The optimal policy æ value-based RI. can represented by Boltzman distribution of Q" and V functions:
exp(Q"(s.a) raaa En exp(Q"(s.a")) exp (8" (x)),
(2)
(s)) Eg exp(Q" a)). This form ensures that actions with higher Q-values are selected more frequently, while preserving stochasticity for exploration.
where the normalization constant is given by the soft value function exp(V*(s))
Direct Preference Optimization While value-based RL are appealing because they explicitly model future returns, they are difficult to apply in practice without an online environment or an explicit reward model. Direct Preference Optimization (DPO) [24] offers a practical alternative by directly optimizing the policy from human preference data without requiring an explicit reward model. In typical preference learning settings, data are collected in the form of prompt-responn response comparisons D [(x, y." y N, where y" the preferred (or "winning") response and y' is the less preferred (or "losing") response for the same prompt x.
Each response is interpreted trajectory composed of token-level decisions. The Bradley- model is used to express the probability that the winning trajectory is better than the losing one:
exp exp (r(r"))
p" (T" " 7-) exp (r(r")) + exp (r(7'))
where = = r(s,, ae) is the total reward along trajectory.
Rather than learning r(s, a) directly, DPO derives a closed-form objective that allows preference likelihood to be optimized without explicit reward model training. By reparameterizing the reward in terms of the log-ratio between the learned policy To and a reference policy Tets the resulting loss takes the following form:
(T(a
( 3
To (a/ls})
(Tg, D) = -E log a
3
Σ
log Trar
> log
(s
(ay) s
where (-) denotes the sigmoid function and 3 controls the scale of preference sensitivity.
3.2 Teacher Value-based Knowledge Distillation (TVKD)
Overview We consider a setting where a DPO-style preference dataset D and a large, well-aligned teacher model & are available. The teacher has been trained using DPO. Our goal is to train smaller student model Tee on D, leveraging sequence- guidance from the teacher to improve alignment and generation quality.
Soft Value Function of a DPO-trained Teacher DPO supervision is limited to binary sequence- level comparisons, offering no information about why a response is preferred or how much better it is. This lack of granularity makes it difficult to guide the model toward generating more aligned content. To address this, we propose to use the value function of the teacher model, which reflects the internal reward model.
Following the interpretation by Rafailov et al. (25]. we treat DPO-trained policy as a soft-optimal solution to teven-level MDP. In this framework, the token-level logits (s a) are viewed soft -alluuss and the corresponding soft value function is defined according to MaxEnt RL principles.
Lemma 1 (Soft value function of DPO-trained policy (5)). Let Qo a) denote the token-level logits of DPO-trained model na and let B > be the temperature of the Boltzmann policy. Then the soft value function is given by:
Vo(s) log Σ
exp(Qg(s,a)/3).
aEV

---

In Appendix C.1 we provide full proof.
From functional perspective, the soft value function be interpreted soft measure of the number of high-quality choices available at a given state. It assigns higher values to states with many promising tokens, thereby indicating more favorable decision point.
Reward Shaping with Teacher Value We propose a method for incorporating information from the value function into the original DPO objective to provide sequence-level soft guidance. We define a shaping term based on the value function of the teacher:
(5s a) Vs('') Va(s),
where s' = [( a) denotes the next state. This shaping term captures the change in the expected future return and serves as proxy for the utility of the action from the perspective of teacher.
Following the theoretical guarantee Ng et al. [20], adding this value-based shaping term to the original DPO reward preserves the optimal policy:
Lemma 2 (Optimal Policy Invariance under Teacher Value-based Shaping). Let M
(S.A, f.r, yr,po Pu) be token-level MDP, where r(s, a) denotes the implicit reward function derived from a dataset. Let n be a OPOrerdined teacher policy, and let Vø(s) denote the state-value function of T-
Define the potential-bune -haseidll---edddddee shaping function ()(s (22 S1+1) as:
(S,, ax) (5((5+1) (st).
Then, the optimal policies of the original MDP M and the shaped MDP M' = (s are the same.
Proof Sketch (see Appendix C.2 return differs from the original return by state-dependent constant. As result, the action that maximizes the expected return remains unchanged.
By telescoping the shaping terms over a trajectory, the total shaped
Failure of Action-dependent Shaping contrast to the value-based shaping term in Lemmal 2 shaping functions that depend jointly on state and action, such as log Ta {) or Qa a), violate
the sufficient conditions for policy invariance (20) These action- eeeionndependentt terms introduce implicit
preferences over actions that cannot be factored out across trajectories, potentially distorting the underlying optimization.
Corollary 2.1 (Action-based Shaping Breaks Policy Invariance). Let (s, = r(s,a)
(s,,)), where (sss,, a.) is composed of the function dependent with both state and action. (s, (ax log (a(a) 8), aQ&(+ (s, a)), then the resulting reward violates the policy invariance guarantee and may alter the optimal policy.
We defer a formal proof this corollary to Appendix C.3
where we show that action-dependent shaping terms fail to cancel out and lead trajectory-level policy distortion.
we incorporate teacher reward signals into student learning beyond binary supervision. This shaping term augments the original DPO reward, resulting in the following RL objective:
Teacher Value-based Knowledge Distillation Following Lemma 2
max B(,,o)=<o (r(s,a) + + (b),,)))
where a is hyperparameter that controls the strength of distillation.
Then we drive our final loss term from the new RL objective:
To(a" |x²))
no(a! 1st)
LTvKD Dj Ta) = (F(D log
Blog
3 log
exp (8"0 (M))))
exp (s at)
We provide full derivation in Appendix B.1 formula for simplicity, which can naturally be incorporated into the current formula. full MaxEnt RL interpretation is in Appendix B.3
We have removed the MaxEnt RL term from the entire
5

---

Gradient Analysis To further analyze how the TVKD loss function works, we define rewards over the preferred and less-preferred trajectories.
(T a) == 8 log ne (a s) - (s,(s,)) a),
where Va (8T) terminal value of the value function of the teacher.
The total trajectory-level preference margin computed as:
1=|-1 Σ
M TTVKD aa")
TTvKD(s' (
which accounts for the potential length mismatch between the preferred and less dess-preferred responses. Then, the gradient of the TVKD loss becomes:
Iwul-1
Inl-1 Σ
VeLTvKD = o(-
Vg log (o(a (s))
Ve log To st)
This expression shows that the gradient encourages increasing the sig-pobbabiltyy of tokens in the preferred trajectory while decreasing that of tokens in the less- dess-preferred one, modulated by the total preference margin M. Notably, the M incorporates both student likelihoods and teacher-derive shaping, effectively blending supervised learning with policy shaping based on external value signals.
4 Experiments
4.1
Experiments Setting
Backbone Architectures and Training Dataset We conduct the preference distillation experiments using three Small Language Models (SLM) as a student: LLaMA-3.2
IB, -LaMA-3,2-33- [IO] and Danube3-500M (23 For teacher models, use Mistral-7B-vO. [13] and LLaMA-3.1 3.1--B (10).
Following 9 we apply supervised fine-tuning (SFT) to both student and teacher models using Deita- 10k-V0 18, dataset of 10k high-quality instruction-response pairs. For preference distillation, use two datasets: (1) DPO-MIX-I - curated collection high-quality pairwise preference data, and (2) HelpSteer2 (27), which is designed to improve helpfulness in LLMs. When using HelpSteer2, we exclude samples in which positive and negative responses have identical helpfulness scores.
Validation and Evaluation Framework Following (9] use Refair-LLaMA-RME 1 (RM)
to select the best checkpoints during training. RM generates scalar score for the responses generated by the model for a given prompt, which has proven to perform well in RewardBench [5]. For evaluation, we evaluate our models on four benchmarks: RM, MT-BBechh (MT) [11, AlpacaEval (AE) 66) and the Open LLM Leaderboard (OLL). For RMM. we use the test set for evaluation, note that this is distinct from the validation set used for checkpoint selection. This gives the average score for the test prompts on the DPO-MIX-7K and Helpsteer2. For MT-Bench, we adopt the single-turn evaluation setup using the 10--4--miii2222-077 compute win rates against ADPA M. also using 200-44-min--224--077
model | as the evaluator. In AlpacaEval, we
the evaluator. For the Open LLM Leaderboard, follow the evaluation protocol defined by the Gao et al. [8].
Baselines We compare our method with diverse baselines, which can be broadly grouped into three categories. First, consider KD methods. VanillaKD [12] and DCKD [ are offline KD methods that rely solely on KL--diergence between the teacher and the student output. In contrast, SeqKD [14] and GKD (2 perform distillation in an online setting incorporating teacher or student rollouts, respectively. Second, we consider offline alignment methods. This group includes DPO[24] and its variants, SimPO [19] and WPO 332 which align model outputs with human preferences without teacher model information. TDPO utilizes a token-level extension of DPO, which is from
http:://plaatoom.oppna.. http:///ppppaaaoooooooo/oooooa//

---

Table Overall preference distillation performance. We use LLaMA-3.2-IB (student) and LLaMA-
3.1-8B (teacher). We report the Fsfair-LLaMA3RRM-VV (RM) rating, the alpaca eval win rate(AE)
against ADPA, the average MT-hench ratings (MT), Open LLM Leaderboard (OIL) scores. The best performances are highlighted in bold, while second- best performances are underline.
DPOMIX
Helpsteer2
Method
RM(1)
MT(()
AE
OLL(T)
RM(()
MT(1)
AE(()
OLL(())
DPO Teacher
-0.77
5.74
73.12
53.45
-0.88
5.63
72.87
57.95
SFT Teacher
-0.94
5.58
71.02
53.15
-1.23
5.58
70.31
53.15
SFT
-1.55
3.56
43.62
40.18
-1.78
3,43
45,49
41.06
DPO
-1,41
3.70
47.03
35.48
-1.22
3.77
44.5
41.24
SimPO
-1.22
3.65
46.12
41.5
-1.37
3.75
49.98
41.33
WPO
-1.51
3.53
39.23
41.36
-1.62
3.62
48.35
41.19
TDPO
-1.45
3.71
41.83
40.95
-1.84
3.61
46.11
41.14
VanillaKD
-1.66
3.46
38.88
41.17
-1770
3.53
44.06
41.09
SeqKD
-1.38
3.40
37.16
41.17
-1.75
3.20
38.47
41.17
DDPO
-1553
3.61
37.3
41.23
-1.64
3.58
43.97
41.23
DPKD
-1557
3.43
37.99
41.35
-1.73
3.48
43.04
41.14
GKD
-1.61
3.52
36.63
41.19
-1.74
3.41
40.78
40.92
DCKD
-1.60
3.55
36.79
41.33
-1.46
3.51
49.98
41.09
ADPA
-1.21
3.73
50.00
40.29
-1.43
3.57
50.00
41.26
TVKD(Ours)
-1.15
3.97
52.18
41.61
-1.18
3.98
54.85
41.35
Question: How does photosynthesis work?
DPO teacher
DPO student
student
In photosynthesis, plants convert sunlight, water, and carbon dioxide into glucose and oxygen using chlorophyll capture energy.
In photosynthesis, plants convert sunlight, water, and carbon dioxide into glucose and oxygen using chlorophyll to capture energy.
In photosynthesis, plants convert sunlight, water, and carbon dioxide into glucose and oxygen using chlorophyll capture energy.
Figure 1: Visualization of shaping terms. The top-3 tokens are highlighted in red.
self-ju jelf-judgment for token-level weight. Third, we consider preference distillation methods. DDPO and DPKD [T6] are fully offline approaches that do require student rollouts. In contrast, ADPA performs one round of student rollout over the entire dataset. More detailed training configurations can be found in Appendix We report the results we have reproduced.
4.2 Main Results
Quantitative Results We show the performance table when using LLaMA-3.1-8B the teacher and LLaMA- 3.2-1B as the student, training and evaluated on the DPOMIX and Helpsteer2 datasets in Table [ First, our method consistently outperforms all baselines on both the RM and MT-Bench. These benchmarks evaluate response quality using L..M-based or GPT--ased grading. Our method surpasses ADPA by 0.29% points on DPOMIX and 0.41% % points on Helpsteer2 in MT-Bench. Notably, unlike ADPA, our approach requires no additional stu- dent rollouts or online evaluation by the teacher. This shows that TVKD has strong generation performance un- der a fully offline setting. In addition, compared to DDPO, which aligns the overall reward margin between the teacher and student, our method yields better performance. This suggests rather than simply mimicking the teacher's re- ward, our value function serves more effective in-
Writing
Humanities
Rolepley
STEM
Math
Extraction
Coding
Reasaning
DPO
TVKD
ADPA
Figure 2: Visualization of task-wise per- formance on MT-bench.

---

Table 2: Results of ablation for TVKD using DPOMIX dataset. The best performances are highlighted in bold, while second best performances are underline.
Mistral-7B-->aanubee-
Mistral-7B >> Llama-1B IB
1lama-8---Llam--3
RM(t)
MT(()
RM(t)
MT(f)
RM(t)
MT(f)
DPO -3.03
3.21
-1.61
3.28
-1.10
5.08
SimPO
-2.91
2.96
-1.55
3.32
-1.12
4.90
DCKD
-2.08
2.84
-1.72
3.07
-1.25
4.86
Ours
-1.99
3.22
-1.36
3.38
-1.05
5.19
Table 3: Accuracy of identifying preferred trajectories using different reward formulations, log probabilities (DPO), length-normaan normalized log probabilities (SimPO), and our shaping term (Ours).
Scoring term Log prob. Log prob. with LC Shaping term(Ours)
Accuracy(%) (f)
19.56
19.72
25.82
dicator of human preference. Second, our method achieves strong performance under pairwise comparison-basel evaluation using AE. On both datasets, it is the only method that surpasses ADPA in win rate, highlighting its strength in direct pairwise comparisons beyond scalar scoring. This means that TVKD produced higher-quality sentences than the previous best model, ADPA, even in a relative evaluation that is more precise than a sentence-by by-eebbbb--eeeeeeeee evaluation. Finally, we show competitive performance a variety of domains. We report only the average results in the main table, while full results are provided in Appendix [F.]] Our method outperforms all baselines in OLL, indicating that TVKD achieves strong alignment without sacrificing performance in other areas.
OLL benchmarks, demonstrating that method generalizes well across
To further investigate the effect of value-based distillation, we analyze the performance of each task in MT-bench, as shown in Figure 2 We compare our method with DPO and ADPA, representing methods without and with distillation, respectively. Comparisons with other methods can be found in Appendix F22 TVKD outperforms both baselines across all areas. In particular, our method outperforms DPO in Writing and Roleplay. A similar trend is observed in ADPA, indicating that distillation helps transfer the teacher's 's writing ability. On the other hand, compared to ADPA, TVKD shows strengths in STEM and Extraction. This is where DPO has advantage over ADPA, which shows that TVKD takes the strengths both DPO and ADPA.
Qualitative Results Figure visualizes the shaping term, defined the difference in value functions between consecutive tokens. The top-3 tokens with the highest values are highlighted in red. For TVKD, which is trained using the value function of DPO teacher, the highlighted tokens such as sunlight, chlorophyll, and capture match those selected by the teacher. These tokens are conceptually central the explanation of photosynthesis, indicating that the value function effectively captures semantically meaningful tokens. While the DPO student shares the same training objective as the DPO teacher, its smaller model size leads differences in the learned value function. This suggests that TVKD successfully inherits the teacher's reward modeling behavior.
Model Ablation In Table 2 we evaluate TVKD across various teacher-student configurations on the DPOMIX dataset. For computational limitations, we compare only offline methods. TVKD consistently outperforms all baselines, demonstrating strong generalization across model scales and architectures. Notably, it improves performance by 0.3% points even when distilling from Mistral-7B into the compact Danube2. 1-50MM, showing that our value guidance remains effective for smaller models. TVKD also demonstrates robust performance across teacher architectures, confirming the transferability of teacher-derived value signals. Moreover, TVKD performs robustly when the size difference between the teacher and student model is smaller.

---

Table 4: Analysis of value function alignment and MT-Bench performance across models. We compare the KL-divergence between teacher and student value functions.
Metric
DPO Teacher
DPO Student
Ours
KL-div, with Teacher Value (4)
0.000
1.245
1.202
MT-Bench (t)
5.74
3.70
3.97
Table 5: Comparison of various auxiliary rewards on the same setting in Table (AE), we use our method as baseline.
In Alpaca- Epal
Type
Name
Auxiliary Reward
Margin Acc.(t)
MT-hencch())
AE(1)
Action
Logits
Q(a, s)
18.29
3.61
37.18
Dependent
Log Probability
log (a
18.31
3.43
32.50
Max
max (
28.86
3.79
36.07
Margin
" (x)
T (s)
(2)
30.23
3.90
48.81
State Dependent
Σ Ta ()Q
Expectation
30.23
3.45
49.42
log Σ exp (Qa(a)
Ours
30.23
3,97
50.00
4.3 Analysis
Value Function Analysis To investigate whether the value function is good at giving pairwise soft labels, we perform additional analysis in Table Specifically, we compute sequence-level rewards by summing token-wise values derived from different signals: tog-pobbaiilities (DPO), log- pog-pobbabilities with length normalization (SimPO), and shaping terms (Ours). Our shaping term achieves higher accuracy than both the raw and the length-normalized log probabilities, suggesting that the teacher value function provides more accurate estimate of human preference. Because our method telescoping sum, naturally reduces the impact of sequence length without requiring explicit normalization. As result, it performs better than methods that rely on length normalization. This shows that our shaping term can act as a reliable and consistent sequence-lever soft label.
Value Alignment Analysis In Table we analyze whether TVKD simulating the value function of the teacher well. We provide MT--ench performance and value function distance with DPO teacher for three models, DPO teacher, DPO student, and TVKD. For value function distance, measured for chosen responses and questions from the train set of DPOMIX and measures the KL-divergence between the output of the teacher value function and the target value function. TVKD shows lower KL-divergence for the teacher value function compared to DPO students. This shows that the student model trained with TVKD has a value function that more similar to the DPO teacher. This shows that TVKD is implicitly modeling the teacher's reward modeling during training, even though it does not have an object that directly models the value function.
Reward Conflict Analysis In Table 5 we conduct detailed analysis to investigate the effects of PBRS and the design of auxiliary reward functions. Specifically, we evaluate various auxiliary reward functions intended to reflect the reward signal of the teacher. To check the conflict between the reward from the dataset and the auxiliary reward, we demonstrate margin accuracy. The margin accuracy is computed following the protocol in (19), measuring the proportion of test samples in which the chosen response receives higher reward than the rejected one. Action-dependent functions. such as logits or log probabilities, fail to preserve the optimal policy theoretically (see Corollary 2.1 Empirically. they also result in lower margin accuracy, confirming their misalignment with preference data. In contrast, all state-dependen functions satisfy PBRS by design, but their performance varies depending on the quality sequence-level supervision. practice, all state-dependent functions exhibit high margin accuracy. However, our value function achieves the highest performance, showing that theoretical guarantees alone are insufficient. These results suggest that effective alignment requires accurate and semantically meaningful reward shaping.

---

Conclusion
In this paper, we tackled the challenge of aligning SLMs with human preferences, task made difficult by the limited capacity of such models and the coarse granularity of sequence-level feedback. To address this, we introduced Teacher Value-based Knowledge Distillation, a novel offline distillation framework that interprets preference alignment through the lens of reward shaping. By extracting value functions from a preference-aligned teacher model, TVKD reshapes the DPO objective to enable fine- fine-grained, trajectory-consiste supervision without altering the optimal policy. Extensive experiments across various benchmarks confirm that TVKD consistently enhances performance.
Acknowledgments
This work was supported by Institute of Information communications Technology Planning Eval- uation (ITTP) grant funded by the Korea government (MSIT) (No.2022-0-00955,, Development of Uncertainty-Aware Agents Learning by Asking Questions).
10

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly indicated our claims and contributions in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our paper in detail in the LIMITATIONS section of the APPENDIX.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?

---

Answer: [Yes]
Justification: We present a rough outline our theoretical lemma in the main text, and present the full proof and assumptions in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We cover the detailed experimental setup in the main text and in the appendix. We will also release the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
12

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We publish our code in supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co. ctipp.ccc/uubli///gr guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We present the full experimental details in the text and appendix. They can also be found the published code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We do not provide statisties for the full data due the cost GPT-evaluation in addition to the prohibitive cost of the experiment, but we do present error bars for our methods in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
13

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify our experimental setup, computational environment, and costs in detail in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///eaanppp..c/opaiii/tttae
Answer: [Yes]
Justification:
We follow all ethical considerations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method is way to make good use of offline datasets, and we don't 't expect much social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
14

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: By only studying distillation, we do not introduce any risk beyond the tradi- tional model.
Guidelines: This is a distillation paper and does not contain any risk beyond the model that already exists.
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We accurately attribute the datasets and models we use in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
15

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not include human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our paper does not include human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
16

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We use LLM for editing purposes only.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
References
[1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning, ICMI. '04, page 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385. doi: 10.1145/1015330.U0 1015430.
1111110.11111111115311 1015430
[2] Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. arXiv preprint arXiv:2306. 13649, 2023.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
[4] Tim Brys, Jörg Denzinger, and Ann Nowé. Multi-agent reinforcement learning using shared potential- advice. International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 1071-1078 2014.
[5] Sam Devlin and Daniel Kudenko. Theoretical considerations of potential-based reward shaping for multi-agent systems. The Knowledge Engineering Review, 26(2):165-174, 2011.
[6] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.044/ 2024.
[7] Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19311 2024.
[8] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL 12608602
org/ records/
17

---

[9] Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, and Qifan Wang. Advantage-guider distillation for preference alignment in small language models. arXiv preprint arXiv22502.79227, 2025.
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1XX3::5503.02533 2015.
[13] Fengqing Jiang. Identifying and mitigating vulnerabilities in IIm-integrated applications. Mas- ter's thesis, University of Washington, 2024.
[14] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1317-1327, 2016.
[15] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
[16] Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, and Furu Wei. Direct preference knowledge distillation for large language models. arXiv preprint arXiv:2406. 19774, 2024.
[17) Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang, Jiulong Shan, Al- bin Madappally Jose, Xiaojiang Liu, Lijie Wen, et al. Tis-dpo: Token-level importance sampling for direct preference optimization with estimated weights. arXiv preprint arXiv:2410.04350. 2024.
[18] Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He, What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:23/2.15665 2023.
[19] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 17:124198-222235, 2024.
[20] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Icml, volume 99, pages 278-287, Citeseer, 1999.
[21] Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[23] Pascal Pfeiffer, Philipp Singer. Yauhen Babakhin, Gabor Fodor, Nischay Dhankhar, and Sri Satish Ambati. H2o-danube3 technical report. arXiv preprint arXiv:2407.09276, 2024.
[24] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53745 2023.
[25] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to gN: Your language model is secretly f-function. 2024.
18

---

[26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
[27] Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpssteer--prefeeene::: Complementing ratings with prefer- ences, 2024. URL http:/::///xxxvoog/aaaaa/
01257
[28] Eric Wiewiora, Garrison Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In ICML, volume pages 792-799, 2003.
[29] Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token- level direct preference optimization. arXiv preprint arXiv:2404. 2024.
[30] Rongzhi Zhang, Jiaming Shen. Tianqi Liu, Haorui Wang, Zhen Qin, Feng Han, Jialu Liu, Simon Baumgartner, Michael Bendersky, and Chao Zhang. Plad: Preference-bee based large language model distillation with preffeeereeeeenceeeee pairs. arXiv preprint 2rXiv:2406.0088m 2024.
[31] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 16:46595-46623, 2023.
[32] Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted pref- erence optimization. arXiv preprint arXiv:2406.1 1827, 2024.
[33] Brian D. Ziebart, Andrew L. Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pages 1433-1438, 2008
19

---

A Limitation
Although our method theoretically guarantees optimal policy invariance under value-based shap- ing [20], in practice, the shaping term can negatively affect training dynamics if the teacher's 's value function severely misestimated or noisy. Although our method guarantees optimal policy invariance, it can be reach in suboptimal. Our method assumes that the teacher value function provides reliable signals. If the value is highly inaccurate or noisy, the shaping term may interfere with learning and lead to suboptimal policies. We do not test cases with weak or mismatched teachers, and the method's robustness in such settings unclear.
B Mathematical Derivation
B.1 Mathematical Derivation of Equation 3
begin with the value-guided shaping framework introduced our main objective:
max E. (-lee)) TT
(r(s;, ax) + a (s, ail)
(5)
where the shaping term is defined as:
(s) =
and s is the next state resulting from action state 8.
To connect this to the DPO setting, we follow the formulation in (25 which relates the reward r(s, a)
soft Q-function Q" (s) In our setting, the shaped Q-function takes the form:
(" ae) = r(st, 0z) + (oa at) + V' (st+1),
(6)
assuming 8141 is not terminal. At terminal states, V*(s)) =
Using this, we express the total cumulative reward across trajectory T = (so, ao. .0....,.77 (,.,,, ST) as:
T-1
T-1
Σ r(s,, 0r) =
Σ (Q" - -
= QQ' ao) + V*(ss)
Σ (s)))
- (*()) - a as
Applying the DPO interpretation of the soft Q-function as log bog-brobabilities 25 I. we approximate:
Q'(s at) (s(+1) log (e(a) s1) and substitute (s) = Vo(s++1
Va Vo(ss) obtain
the shaped log-prob objective:
T-1
T-1
To(a: | 8c)
Σ
r(s,, ax) = Σ
3 log
+ (*p)))
t=
=
exp
Thus, the TVKD loss over a pairwise preference dataset becomes:
Ir -1
Ta (a. s)
LIvKo(TO.Di = -B(Tus MD
logo
Σ
8 Blog
1=
exp
( a
In|-1
Te(at st)
8 Blog
(gpolsh
exp
where (s, a) Vo() Vo(s).
Note that all conditions based on the initial state 8p are shared between Tue and 77 and therefore are canceled out in the preference-based loss.
20

---

B.2 Mathematical Derivation of the Equation 4
We begin by defining the shaped reward at each timestep using the teacher's potential based shaping function Vh. which is independent of the student parameters #:
( 8e) 3 log (e(ac) st) ( az).
For pair of trajectories one preferred (w) and one less preferred (1-we we define the total shaped reward as:
Inl-1
Rd
"Tv R' =
PTvKD(a,, st).
t=
t=0
Then, the preference loss given by the negative log-sigmois of the total reward margin:
LIVRD log o R²),
where the scalar margin is:
1rul-1
In|-1
M
R A²
(av s())
( st).
Since de is computed only from the teacher and does not depend on 0 its gradient vanishes:
Vodaloe, mn) = 0.
Therefore, the gradient of the margin M becomes:
T'-1
VoM= =
Vo log To(a"" 8T))
Σ
Vo log (a st)
Applying the identity , log (xx) = (-x), the gradient of the loss becomes:
VOLTVKD -V Vollggo(M) (- VOM
-B-
Vo (a s
Σ Vo log (a st)
This formulation generalizes to trajectories of unequal lengths and cleanly separates the teacher shaping from the student optimization path. The teacher's value function contributes to the reward margin, while the gradient flows solely through the student model's oogprobbabllitiss
B.3 Mathematical Derivation of Equation 3 in MaxEnt RL setting
We extend our value-guided objective to the entropy-regullrized reinforcement learning (MaxEnt RL) framework. The resulting objective is:
max E Ea
(r(s,, a,) DDKr.
+ t (
(7)
where the shaping term is defined as vis (s, a) = Vo(s') - Vo(s). consistent with the formulation in our main objective.
We follow 225] which describes the recursive soft Q-function in token-level DPO as:
Tr(s,, ae) + 3 log (st + (*((4)) a (ss,,,), 3 is not terminal,
Q" (St, 0.)
r(se, ar) + 3 log (ar (8)) Vo(s)),
if 8441 is terminal.
(8)
21

---

Summing over a trajectory
(su. ao .0, the total reward becomes:
(8 a,)
10" (st, a4) - -Blog ( V (8++1) a (3))))
(9)
[ log no (1slss)) - 3 log mer(()))))) (st,
(10)
To (o(ar(se)
T-1
B log
Σ
a (st ac).
(11)
(1)
The final TVKD loss under MaxEnt RL:
Tul-11
(a(a ((t))
CDVKD(TO, Brefe T²) -E (Tw
log a
log
expl xxp(yvol (o 0² Tre (a)
1=0
(12)
|n|-1
Ta(a\(s\)
B log
exp(Sng th (s% at) ( (at st))
C Proof of Lemma
C.1 Proof of Lemma
We build on the interpretation introduced by Rafailov et al. [25), which characterizes DPO-trained policies soft-optimal solutions in a deterministic token-level MDP. Specifically, they show that the token-level logits (s a) of a DPO-trained model correspond to the soft Q-function of an optimal Boltzmann policy under the maximum entropy reinforcement learning (MaxEnt RL) framework.
Assumption (from
25
Let ( s) be the DPO-trained policy defined as:
exp(Qg(s,a)/3)
aa s) E. exp(Qo(s,a')/8)
for fixed temperature 0. Then Te soft-optimal policy with Q-function Qa a), and its corresponding soft value function is defined by:
(s(s) = E a)] + {.[T=( s)],
where M[[r] - -. We now derive a closed-form expression for (s) based on the Boltzmann structure of mg. Derivation.
E. (a) log (a) denotes the entropy of the policy.
First, recall the identity:
log (a 8) Qs(ssa)
log Σ ex))())(s.c')13)
B
Thus, the entropy term becomes:
H[ms]
Σ Ts () log (olgo
Σ Ts(a (s))ss) Q0(s,a) log exp((o(ssaa)/3)
1 Σ (a s)Q a) + log Σ exxp(Qo(s.
8
22

---

Now substitute back into the soft value function definition:
Va(s) = Σ Ta(a s)Qu(s,a) + 3H[
(-11 1 <
Σ (a ()) +
Tola 8)Qa(8, + log E Eexp(Qo(s,a))19)
log Σ exp(Qo(s.a))/)
aEV
This completes the proof.
C.2
Proof of Lemma
In Section 3 we stated that augmenting the DPO reward with the Teacher Temporal Difference term (+(+++1) - (st) does not change the optimal policy. Below we provide formal proof.
Proof. Consider trajectory (so, ao; $1, -1 Let the original return be:
T-1
G(T) = Σ a()
Now, define a potential-bure shaped reward:
F(s,, Gz. 8t+1) r(s,, a,) + (((+41) ((sx),
and let the corresponding shaped return be:
T-1
G(T) = Σ
[r(3,, (((++)) - ((s))]
1=
By telescoping,
Σ ( = +(ST) (s0))
so the shaped return becomes:
G(r) = G(T) ( - (
This additive term depends only on the start and end states, not on the specific actions taken. Therefore, for any policy the expected shaped state-action value is:
Q (s, o) = Q" (s, + Evwrttss [2())) ((s).
Since the difference between shaped Q-values at any state only depends on (sx) and the expected (s'',, the ranking actions remains unchanged:
arg max (* () = arg max Q* a).
Hence, the optimal policy is invariant under this form of reward shaping.
C.3 Proof of Corollary 2.1
In Section 3 we claimed that action-based shaping terms such (s,sa)) a log (o(a s) or ()(s, a) = ( a) violate the conditions required for potential- shaping and may alter the optimal policy. We now provide formal justification.
Proof. Suppose augment the DPO reward with an action-dependent shaping term:
F(st, ax) r(st. ax) + ((5), where (),,,, az) 4(s)) ((5+1)).
23

---

Then the shaped return for a trajectory
(s0, du. $1.... ST) becomes:
G(T) Σ (r)) +s).
Unlike the state-based shaping case in Lemma there is no telescoping cancellation of the added shaping terms because (5 0x) depends explicitly the action each step and not solely on the state. As result, the shaped return ((T) differs from the original return G(T) by an amount that depends on the entire sequence of actions:
G(T) - + Σ (s)
where E. (88 0e) varies with the policy and trajectory. This action-dependent distortion implies that:
Σ (5 (s)
("(s.))=Q"sssd) + =000000000000000 8.00ma)
which can shift the relative ranking of actions at state s, thereby changing arg max, (*
Hence, the optimal policy under 7 may differ from the under the original reward violating policy invariance.
D Relation to Advantage Function in RL
Our method can be interpreted as a special form of salue-based reward shaping, closely related the advantage function in reinforcement learning. In standard RL, the ad advantage function is defined as:
A(s, a) = r(s, a) + (s') - (s),
which quantifies the relative benefit of taking action state compared the expected return from the state baseline (ss).
In our formulation, we do not use explicit reward r(s, a). as it is typically unavailable in offline preference datasets. Instead, we define the reward shaping term using the teacher's value function:
(ss a) Vo(s') Vs(,,,
where Vo is the value function learned by the DPO teacher model, and s' denotes the next state (i.e., the updated token-level context after applying action a).
This formulation implicitly captures notion of advantage by measuring the change in future utility as estimated by the teacher. It provides directional feedback for the student model without relying on noisy or hard-to-estimate token-level rewards.
This connection offers theoretical justification for the stability and effectiveness of our method, as value-based shaping of this form is known to preserve the optimal policy under standard condi-
tions (20
E Experiment Details
E.1 Training Details
Training Details The SFT for both student and teacher models is conducted over epochs, using a learning rate of 2 x 10 and batch size of 128. The DPO teacher is trained with 3 = 0.05, 10-7, a batch size of 128, and for epochs. For distillation methods, we employ context distillation 3 to improve efficiency. Specifically, we precompute the top 50 probabilities and save them to use for distillation following (9]. In TVKD, we experiment with [0.1, 0.2, 0.5, 0.7, 1.0, 1.5] and B [0.1, 0.2, 0,
a learning rate of
24

---

Table 6: Overall performance of Open LLM Leaderboard trained with DPOMIX datasets.
Method
Average
ARC-easy
GSM8K
Hellaswag
MMLU
TruthfulQA
Winogrande
DPO teacher
53.448
76.800
47,900
61,480
60.230
(4.200
70.080
SFT teacher
53.152
76.200
47,300
60.844
60.070
(4,500
70.000
SFT
40.176
59.040
05.500
49.570
37,950
29.488
59.510
DPO
35.475
59.000
06.300
49.760
37.840
00.600
59.350
SimPO
41.497
64.942
06,975
50.129
37.943
29.009
59,984
WPO
41.356
64.850
06.369
49.642
37.958
29.253
60.063
TDPO
40.950
64.310
06.431
49.991
37.560
27.907
59.747
VanillaKI
41.166
64.980
05.838
49.353
37.922
28.764
60.141
SeqKD
41.166
64.980
06.141
48.840
37.820
29.002
60.210
DDPO
41.225
65.000
06.141
49,492
37.840
29.131
59.747
DPKD
41.354
64.890
06.388
49.542
37.986
29.253
60.063
GKD
41.191
64.770
06.670
49.070
37.900
28.274
60.460
DCKD
41.328
65.110
05.990
49.340
37.850
29.540
60.140
ADPA
40,288
60.185
06.670
48.626
37.450
28.760
60.039
TVKD (Ours)
41.605
64.550
05.760
49.650
38.540
29.880
61.250
Detailed baseline setting For ADPA, following the original implementation we first generated single student rollout over the entire dataset. We then computed and saved the differences between the DPO teacher and the SFT teacher, which were used during training. For other online- based KD method(SegKDD GKD), we use only single student or teacher rollout over the entire datasets following [9. For online-based preference distillation method, we use ground-truth chosen and rejected answer instead of teacher and student rollouts following [9]. It can be different original implementation in theoretically. the case of our method, we saved the DPO teacher's ' logits over the original dataset once and used them for training.
Computational Efficiency We conducted experiments using four Nvidia RTX 4090 GPUs. The total wall time for TVKD was approximately 2.5 hours. In our method, we precompute and store the DPO teacher's logits during training at the cost increased storage usage. In contrast, methods such as GKD, SeqKD, and ADPA require generating additional student or teacher rollouts across the entire dataset and subsequently computing and storing teacher logits for each those rollouts. This results storage overhead and slightly higher computational ADPA requires not only the DPO-trained teacher but also the SFT-trained model to compute the logit differences, further increasing computational resource demands.
the entire dataset once, which reduces computational overhead
larger compared our approach. In particular,
F
Additional Results
F1
Open LLM Leaderboard
We show the overall score of the OLLs the main table as shown below in Table 6
Table
F22
MTThench
We represent the hexagons in Figure 3 that represent the performance of each task on MTTbench. Our method shows consistently high performance compared to other methods, except for coding.
F.3 Robustness to Distillation Strength
In Figure 4a we conduct an ablation study on the hyperparameter o, which controls the influence of the teacher's 's value signal during training. We observe that TVKD achieves optimal performance on both the RM score and MTT-bench when a 0.7, with performance dropping at both lower and higher values. This indicates that our method offers a tunable mechanism to balance teacher guidance and data-driven learning. small a may underutilize the teacher's future preference signal, while an excessively large suppress the student's 's capacity to fit dataset-specific patterns. The ability
25

---

Table 7: Overall performance of Open LLM Leaderboard trained with Helpsteer2 datasets.
Method
Average
ARC-easy
GSM8K
Hellaswag
MMLU
TruthfulQA
Winogrande
DPO teacher
57.949
78.746
46.745
61.085
60.109
30.290
70.718
SFT teacher
53.152
76.200
47.300
60.844
60.070
04.500
70.000
SFT
41.064
64.983
06.369
48.676
37.787
29.376
59.195
DPO
41.243
64.680
05.980
49.572
37.972
29.253
60.000
SimPO
41.308
64.680
06. 161
48.974
37.958
30.110
59.984
WPO
41.190
64.915
06.141
48.645
38.043
28.860
60.537
TDPO
41.130
64.140
02.126
49.114
36.767
28.886
60.805
VanillaKD
41.089
65.194
06.065
48.645
37.081
29.009
60.537
SeqKD
41.169
65.194
05.838
48.964
38.200
29.009
59.809
DDPO
41.234
65. 609
06.293
49.522
37.958
29.009
59,511
DPKD
41.136
64.948
06.358
49.422
37.892
28.764
59.433
GKD
40.924
65.025
05.075
49.104
37.858
29.131
59.353
DCKD
41.092
65.125
06.520
48.516
37.253
28.519
60.616
ADPA
41.259
65.446
05.898
49.104
38.007
28.641
60.458
TVKD (Ours)
41.352
64.229
05.681
48.974
38.157
30.772
60.300
Writing
STEM
Mach
Extrection
Coding
Ressoning
dped
TUKD
ad59
2005
Figure Visualization of token-wise preference
to adjust provides practical flexibility, enabling calibrate the distillation strength suit various datasets, teacher qualities, or deployment constraints.
E.4 Robustness to Temperature Parameter
We present the robustness evaluation with respect the temperature parameter / in Figure 4b Our method achieves the highest performance 8 = 2, while overall demonstrating robust behavior across a wide range of values. This suggests that although higher temperatures yield sharper value function and thus increase discriminability. the influence of 3 remains limited since our method captures not the absolute value itself, but the difference in value before and after the current action.
E.5 Teacher Model Analysis
We show additional experiments Table and Table 9 to demonstrate that TPKD is robust distillation teacher quality. We compare the distillation performance three different teachers: the default teacher model SFTed on Peita-10k- VO, a teacher model trained with Helpsteer2 as a DPO, and Instruct teacher, the instruct tuning version posted on Huggingface. Our method outperforms DCKD on all teacher settings. This shows that TPKD relatively robust to the quality of the teachers. Note that Instruct teacher performs poorly RM using helpsteer2's test set, but performs well as DPO teacher on MTTbennc, which a comprehensive evaluation.
26

---

0.00
5.00
-0.50
4.50
8
-0.50
4.50
RM Score
-0.75
4.25
RM Score
-0.75
4.25
-1.00
4.00
-1.00
4.00
MT-benc
3.75
1.50
RM Score
3.50
-1.50
3.50
MT-aench Score
3.25
-1.75
Optimal
MT-oench Score
3.25
-2.00
1.4
3.00
-2.00
0.25
0.50
1.50
1.25 1.50
1.75
2.00
3.00
a in TVKD loss
B in TPKD loss
(a) Robustness to Distillation Strength
(b) Robustness Temperature Parameter
Figure 4: Sensitivity analysis of distillation hyperparameters.
Table RM (higher is better) comparison between DCKD and TPKD across teacher types.
Teacher Type
DPO
SFT
Instruct
DCKD
-1.46
-170
-1.68
TPKD
-1.21
-1.46
-1.56
Table MT-bench (higher is better) comparison between DCKD and TPKD across teacher types.
Teacher Type
DPO
SFT
Instruct
DCKD
3.51
3.38
3.60
TPKD
3.98
3.73
3.96
27


Reviews:
[{"review_id": "ixRpUaxbGB", "reviewer": "Reviewer_Sqwj", "summary": "This paper addresses the challenge of preference alignment for small language models (SLMs) by proposing a novel knowledge distillation approach termed Teacher Value-based Knowledge Distillation (TVDK).  \n\nWhile current mainstream methods such as Direct Preference Optimization (DPO) align models using binary comparisons, the resulting coarse-grained signals may be insufficient for small models with limited capacity. Traditional knowledge distillation approaches primarily focus on mimicking the behavior of the teacher model, neglecting the richer internal reward modeling contained within it.\n\nThe core idea of TVKD lies in moving beyond mere imitation of the teacher’s output, instead distilling the teacher’s internal \"value judgments.\" This is achieved by introducing an auxiliary reward term that satisfies the theory of Potential-Based Reward Shaping (PBRS). As a result, the student model receives more fine-grained guidance while the original optimization objective and optimal policy of DPO remain unchanged.", "questions": "1. On what data was the teacher model trained using DPO?\nThe manuscript mentions that the teacher model πϕ is one that has \"already been trained using DPO.\" But the specifics of this training process are quite vague.\n2. Definition and Interpretation of \"Accuracy\" in Value Function Analysis (Table 3).\nIn the analysis presented in Section 4.3, Table 3 showcases a metric named \"Accuracy(%)\", which is utilized to demonstrate that its shaping term can better identify preferred trajectories. Nevertheless, both the implication and the results of this metric are perplexing.The accuracy of the \"Ours\" method indicated in the paper is 25.82%. This figure, on its own, appears remarkably low, making it difficult from an intuitive standpoint to regard as \"accurate\".", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The method's principal advantage is its theoretically-grounded framework that distills a teacher model's internal value function via potential-based reward shaping, thereby enriching the student's learning with fine-grained supervision while provably preserving the optimal policy. Conversely, its core limitation is the fundamental dependence on the teacher model's quality, as any inaccuracies or biases inherent in the teacher's approximated value function are directly propagated to the student, capping performance and potentially transferring undesirable behaviors.", "was_revised": true, "final_justification": "The manuscript and subsequent rebuttal present detailed experiments and results, and the authors addressed some of my concerns. However, I remain concerned about the strong reliance on the teacher model and the associated biases. I look forward to future work. This is my final rating.", "rebuttal": "We sincerely appreciate the reviewer Sqwj’s recognition of our theoretical analyses and comprehensive experiments, and we are grateful for the constructive comments provided. Below, we respond to the specific points raised:\n\n### **W1,2.**\n\n---\n\n> Conversely, its core limitation is the fundamental dependence on the teacher model's quality, as any inaccuracies or biases inherent in the teacher's approximated value function are directly propagated to the student, capping performance and potentially transferring undesirable behaviors.\n> \n\n### **R1.**\n\n---\n\nWe understand the concern that errors or biases in the teacher’s value function may be propagated to the student, which is a well-known issue in distillation methods. To address this issue, TVKD is designed **to utilize the rich representations of the teacher** model while explicitly **preserving the human-preference structure** encoded in the dataset, as described in lines 36–37 of the Introduction.\n\n**Lemma 2 provides the theoretical foundation** for this design. By incorporating the teacher’s value function as a potential within the PBRS framework, TVKD preserves the optimal policy of standard DPO and reduces the influence of bias present in the teacher. **This is supported by empirical results in Table 5**, where the state-dependent variant that satisfies the PBRS condition achieves higher DPO-margin accuracy than the action-dependent variant that violates it.\n\nAdditionally, **Appendix F.5 compares distillation using teachers before and after DPO fine-tuning**. When the teacher model is of lower quality, such as before undergoing DPO, TVKD remains more **robust than baseline methods**. This shows that TVKD is more **robust to low-quality teachers** by prioritizing supervision signals in the dataset over teacher guidance.\n\nTaken together, these theoretical and empirical results demonstrate that TVKD can effectively leverage the benefits of the teacher model while mitigating the risks of negative bias transfer.\n\n---\n\n### **Q1.**\n\n---\n\n> On what data was the teacher model trained using DPO? The manuscript mentions that the teacher model $π_ϕ$ is one that has \"already been trained using DPO.\" But the specifics of this training process are quite vague.\n> \n\n### **A1.**\n\n---\n\nThank you for pointing out the lack of clarity in our description. As described in **Section 4.1 and Appendix Section E.1**, the DPO teacher model was first **supervised fine-tuned (SFT) on the Deita-10k-v0 dataset**, and then **further trained with DPO for 2 epochs on the same dataset as the student model (DPOMIX or HelpSteer2)**. This setup was chosen to **ensure fairness by following the same configuration as the ADPA paper**. The performance of the trained DPO teacher, which consistently surpasses both the student model and the SFT teacher, can be verified in **Table 1**. All training details can be verified in the provided codebase.\n\n---\n\n### **Q2.**\n\n---\n\n> Definition and Interpretation of \"Accuracy\" in Value Function Analysis (Table 3). In the analysis presented in Section 4.3, Table 3 showcases a metric named \"Accuracy(%)\", which is utilized to demonstrate that its shaping term can better identify preferred trajectories. Nevertheless, both the implication and the results of this metric are perplexing.The accuracy of the \"Ours\" method indicated in the paper is 25.82%. This figure, on its own, appears remarkably low, making it difficult from an intuitive standpoint to regard as \"accurate\".\n> \n\n### **A2.**\n\n---\n\nWe acknowledge that our explanation of Table 3 was insufficient and appreciate the reviewer’s thoughtful comment. **Table 3 reports results on the test split of DPOMIX, not on the training data.** The relatively low accuracy primarily reflects the inherent difficulty, rather than an issue with insufficient training.\n\nFor clarity, we report below the accuracy on the **training split**, where all methods achieve **above 70%**, indicating that the models are well-trained. In both the training and test datasets, our method consistently demonstrates higher accuracy compared to the baselines.\n\n| Method | Log Prob | Log Prob with LC | Ours |\n| --- | --- | --- | --- |\n| Test Accuracy (%) | 19.56 | 19.72 | **25.82** |\n| Training Accuracy (%) | 75.54 | 77.49 | **83.81** |\n\nWe promise to include this table in the Appendix and provide a clearer explanation in the camera-ready version."}, {"review_id": "auxj6JVLb7", "reviewer": "Reviewer_uzh2", "summary": "This paper introduces Teacher Value-based Knowledge Distillation (TVKD), a method to distill preference alignment from a teacher model. Instead of focusing solely on behavior cloning or KL divergence, TVKD incorporates a value-based auxiliary reward that adheres to potential-based reward shaping principles, preserving the global reward structure and optimal policy. The approach integrates seamlessly into standard DPO training without requiring additional rollouts. Experimental results show that TVKD yields consistent performance gains across multiple benchmarks and model sizes.", "questions": "Which theoretical results or proofs are original contributions by the authors, rather than adaptations or direct reuse from prior work?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The authors provide detailed theoretical analyses supporting the design of their method, which enhances its credibility and soundness.\n2. The paper includes comprehensive experiments with a wide range of baseline comparisons, demonstrating thorough empirical evaluation.\n\n**Weakness**\n1. According to the Qwen3 Technical Report, SOTA models typically undergo multiple preference optimization stages beyond DPO alone. This raises concerns about whether a DPO-based distillation method can sufficiently capture the full spectrum of preference knowledge from a powerful teacher model.\n\n2. The proposed reward shaping technique using the teacher's value function appears similar to Equation 10 in the paper “From r to Q*: Your Language Model is Secretly a Q-Function.” The authors should clarify the conceptual and practical differences between their approach and that prior work to avoid ambiguity and better position their contribution.\n\n3. Experimentally, the proposed method does not outperform the ADAP baseline by a significant margin. Given that evaluation relies on LLM-based judges—known to exhibit randomness and occasional misalignment with true human preferences—it is unclear whether the observed improvements are robust.", "was_revised": true, "final_justification": "Thanks for authors to address my concerns, I will increase my score.", "rebuttal": "We sincerely thank reviewer uzh2 for acknowledging the theoretical contributions and extensive experiments in our work, and we appreciate the thoughtful and constructive feedback provided.\n\n### **W1.**\n\n---\n\n> According to the Qwen3 Technical Report, SOTA models typically undergo multiple preference optimization stages beyond DPO alone. This raises concerns about whether a DPO-based distillation method can sufficiently capture the full spectrum of preference knowledge from a powerful teacher model.\n> \n\n### **R1.**\n\n---\n\n**In principle, TVKD is compatible with any teacher model trained under the Maximum Entropy RL framework(MaxEnt RL).** While the Qwen3 Technical Report does not provide full algorithmic details of all training stages, we have confirmed that GRPO is explicitly mentioned in their pipeline. Beyond GRPO, methods like PPO and SAC also fall under the MaxEnt RL family and are theoretically compatible with TVKD via the soft value function formulation in Lemma 1.\n\nMoreover, **when the teacher model provides an explicit value function (some variants of PPO and GRPO) TVKD can directly leverage it without relying on Lemma 1**, further simplifying implementation.\n\nAlthough our experiments focus on DPO-trained teachers, this choice was made to ensure methodological consistency between teacher and student models. **TVKD itself is not limited to DPO teachers** and can, in principle, be extended to a broader range of RLHF-trained models.\n\nIf there are additional preference optimization methods used in Qwen3 or elsewhere that we have not accounted for, please join the discussion and let us know. **We would be happy to discuss their compatibility with TVKD**. We appreciate the reviewer’s perspective and welcome further dialogue on this point.\n\n### **W2.**\n\n---\n\n> The proposed reward shaping technique using the teacher's value function appears similar to Equation 10 in the paper “From r to Q*: Your Language Model is Secretly a Q-Function.” The authors should clarify the conceptual and practical differences between their approach and that prior work to avoid ambiguity and better position their contribution.\n> \n\n### **R2.**\n\n---\n\nWe appreciate the reviewer’s comment and agree that clarifying the relationship with *“From r to Q: Your Language Model is Secretly a Q-Function” (rtoQ)* is important. \n\nFirst, we would like to emphasize that the primary goal and contribution of our work is to **utilize teacher information** in a way that does not interfere with the reward structure of the **offline DPO dataset during distillation**. In contrast, rtoQ is primarily an **analytical work** that reinterprets DPO from a Q-learning perspective and does not propose a new training method or distillation technique.\n\nOur training loss may looks similar to Equation 10 in rtoQ because **both share the PBRS condition** (originally from Ng et al., ICML 1999 [1]). However, while rtoQ employs this equation solely for **theoretical analysis**, TVKD explicitly introduces the teacher value function $V_\\phi$ as a shaping term to **directly guide student training**. \n**PBRS is a widely used theoretical tool in RL** ([2],[3],[4]), and many reward shaping methods have been developed based on this principle. Our theoretical analysis and method are also built upon this foundation, leveraging PBRS to provide a principled understanding of the distillation process.\n\n[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" *Icml*. Vol. 99. 1999.\n\n[2] Wiewiora, Eric, Garrison W. Cottrell, and Charles Elkan. \"Principled methods for advising reinforcement learning agents.\" Proceedings of the 20th international conference on machine learning (ICML-03). 2003.\n\n[3] Asmuth, John, Michael L. Littman, and Robert Zinkov. \"Potential-based Shaping in Model-based Reinforcement Learning.\" AAAI. 2008.\n\n[4] Gao, Yang, and Francesca Toni. \"Potential Based Reward Shaping for Hierarchical Reinforcement Learning.\" IJCAI. 2015.\n\n---\n\n### **W3.**\n\n---\n\n> Experimentally, the proposed method does not outperform the ADAP baseline by a significant margin. Given that evaluation relies on LLM-based judges—known to exhibit randomness and occasional misalignment with true human preferences—it is unclear whether the observed improvements are robust.\n> \n\n### **R3.**\n\n---\n\nWe acknowledge that LLM-based judges can sometimes be misaligned with true human preferences, so we aimed to mitigate this by employing a diverse set of evaluation methods. Specifically, we used a **reproducible open-source reward model (RM), GPT-judge-based MT-bench(MT), 1-on-1 pairwise comparisons (AE) with ADPA, and the Open LLM Leaderboard (OLL)**, which evaluates general commonsense reasoning beyond the training set. **As shown in Table 1, TVKD consistently outperforms ADPA across all these evaluations.**\n\nTo ensure statistical reliability, we report the standard deviations from three independent trainings of the experiments in Table 1. Due to computational cost, we only provide standard deviation values for the comparison of ADPA and TVKD. **TVKD outperforms ADPA with statistical significance on all benchmarks.** \n\n| DPOMIX | RM&nbsp;($\\uparrow$) | MT&nbsp;($\\uparrow$) | AE&nbsp;($\\uparrow$) | OLL&nbsp;($\\uparrow$) |\n|--------|---------------------|----------------------|----------------------|-----------------------|\n| ADPA   | -1.2418 ($\\pm$ 0.037) | 3.7445 ($\\pm$ 0.056) | 50.00 | 40.2883 ($\\pm$ 0.0394) |\n| Ours   | **-1.1327** ($\\pm$ 0.033) | **3.9864** ($\\pm$ 0.087) | **53.09** ($\\pm$ 0.127) | **41.6237** ($\\pm$ 0.0465) |\n\nAdditionally, we emphasize that **TPKD does not require online rollouts**. ADPA evaluates student outputs with the teacher at every epoch, resulting in increased storage and computational overhead as both the dataset and teacher model size grow. In contrast, TPKD requires only a single computation of the teacher value function over the entire dataset, making it significantly more computationally efficient. **A detailed analysis of computation efficiency can be found in Appendix E.1.** \n\nAnd we also report CPU wall time of comparable methods. Our method demonstrates a shorter wall time than ADPA, despite achieving higher performance in all benchmarks.\n\n| Method | CPU wall Time |\n| --- | --- |\n| DPO | **3h 20m** |\n| GKD | 3h 42m |\n| DCKD | 4h 22m |\n| ADPA | 5h 2m |\n| TVKD(Ours) | 3h 41m |\n\n### **Q1.**\n\n---\n\n> Which theoretical results or proofs are original contributions by the authors, rather than adaptations or direct reuse from prior work?\n> \n\n### **A1.**\n\n---\n\nWe appreciate the opportunity to further clarify our original contributions.\n\nWe introduce TVKD, a novel method for **incorporating the rich features of a teacher model** while **minimizing interference with the reward structure of the offline dataset**. We also provide a theoretical guarantee in **Lemma 2**, proving that TVKD preserves the optimal policy under the PBRS condition. In addition, we present **extensive empirical results** demonstrating that TVKD consistently outperforms strong baselines across various settings, highlighting its robustness and effectiveness.\n\nAs our work builds upon the theoretical foundations of reinforcement learning, some proof techniques and equation forms may naturally resemble those used in prior work. However, identifying limitations in DPO-based distillation through theoretical analysis, proposing a new method to address them, and empirically validating its effectiveness are clear contributions of our work.\n\nWe believe this contribution is original and offers a new perspective on DPO-based distillation, both theoretically and practically, which we hope will be valuable to the community."}, {"review_id": "Lhs1OmYC9H", "reviewer": "Reviewer_oStu", "summary": "The paper introduces Teacher Value-based Knowledge Distillation (TVKD) to enhance preference alignment in small language models by leveraging the value function of large language models. Direct Preference Optimization (DPO) relies on coarse-grained binary feedback, which is insufficient for small models with limited capacity. Existing distillation methods (e.g., KL divergence) mimic teacher behavior but overlook reward modeling, risking misalignment with human preferences. TVKD introduces potential-based reward shaping (PBRS) by treating the teacher model’s value function as an auxiliary reward signal, preserving DPO’s optimal policy while providing fine-grained guidance. Experiments across benchmarks MT-Bench, AlpacaEval, Open LLM Leaderboard show consistent gains over baselines.", "questions": "Question:\n* How can the potential for transferring spurious biases from the teacher model be further quantified or mitigated? Have alternative methods been explored, such as teacher ensembles or teachers trained with more robust alignment techniques?\n* Beyond current benchmarks, how does TVKD's generalization hold on highly out-of-domain or adversarial datasets to truly assess its robustness against reward hacking?\n* A more detailed analysis of the hyperparameter α's sensitivity is needed. How does its tuning affect the balance between teacher guidance and student learning from direct DPO, and what are practical tuning guidelines?\n* Can the TVKD framework be generalized or adapted to distill from teachers aligned using other RLHF methods, such as those employing an explicit reward model (e.g., PPO-trained teachers)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Theoretically sound: Leverages token-level optimal value of soft value-based RL (Lemma 1), with an invariance guarantee of the optimal policy under PBRS (Lemma 2), avoiding issues of action-dependent shaping (Corollary 2.1).\n* Empirical performance: Comprehensive experiments on diverse model sizes (Table 2) and datasets (DPO-MIX-7K, HelpSteer2) demonstrate TVKD’s superiority over 14 baselines across RM, MT-Bench, AlpacaEval, and Open LLM Leaderboard metrics.\n\nWeaknesses:\n* reward hacking risk from teacher bias: Distillation from a DPO-trained teacher, whose \"optimal\" policy might contain spurious biases from preference data, increases the risk of transferring these biases and impairing student generalization on out-of-domain datasets.\n* teacher model \"optimality\" assumption: The \"optimality\" of the teacher's value function is derived from its training on potentially imperfect human preference data, which might not perfectly reflect true human preferences, thus limiting the distilled signal.", "was_revised": true, "final_justification": "The author's response address my concern on the optimality requirement of the teacher's value, with additional results and further clarification. Thus I decide to raise my score.", "rebuttal": "We appreciate reviewer oStu’s thoughtful comments and are grateful for the recognition of both our theoretical analysis and experimental validation.\n\n### **W1.**\n\n---\n\n> reward hacking risk from teacher bias: Distillation from a DPO-trained teacher, whose \"optimal\" policy might contain spurious biases from preference data, increases the risk of transferring these biases and impairing student generalization on out-of-domain datasets.\n> \n\n### **R1.**\n\n---\n\nWe acknowledge the reviewer’s concern regarding reward hacking risks arising from teacher bias in DPO-based distillation.\n\nWe identify **two sources of reward hacking** in this context. The first stems from the sparsity of **binary preference labels in the DPO dataset** combined with the **limited capacity of the student** model, which may lead the student to overfit **misleading signals**. Our distillation framework directly addresses this by **leveraging richer supervision from the teacher model**.\n\nThe second source arises from the potential **biases encoded in the teacher model** itself, which can be **transferred to the student**. To mitigate this, we introduce an **auxiliary reward term that satisfies the PBRS condition**, ensuring that the student learns a policy **aligned with the original DPO objective**. This theoretically guarantees the preservation of the optimal policy and protects against harmful bias propagation. Supporting evidence can be found in Table 5, where PBRS-compliant (state-dependent) shaping leads to higher margin accuracy.\n\nAs for out-of-domain generalization, **Open LLM leaderboard(OLL) in Table 1** approximates broad language abilities that contain largely disjoint from the training dataset. TVKD outperforms strong baselines on OLL as well, suggesting enhanced robustness in generalization compared to other methods.\n\n### **W2.**\n\n---\n\n> teacher model \"optimality\" assumption: The \"optimality\" of the teacher's value function is derived from its training on potentially imperfect human preference data, which might not perfectly reflect true human preferences, thus limiting the distilled signal.\n> \n\n### **R2.**\n\n---\n\nWe appreciate the concern, but TVKD does not assume that the teacher’s value function is optimal. Instead, it uses the teacher signal solely as a potential-based shaping term, which theoretically preserves the optimal policy of the original DPO objective (Lemma 2).\n\nThis design ensures that the dataset-derived preference signal remains the primary supervision, and the teacher value merely provides auxiliary guidance. As shown in Appendix F.5, **TVKD shows greater robustness than baseline methods even when the teacher is trained suboptimally**, confirming that our approach does not rely on the optimality of the teacher model.\n\n### **Q1.**\n\n---\n\n> How can the potential for transferring spurious biases from the teacher model be further quantified or mitigated? Have alternative methods been explored, such as teacher ensembles or teachers trained with more robust alignment techniques?\n> \n\n### **A1.**\n\n---\n\nTo approximate the effect of transferring teacher bias, we evaluated how much the **original DPO margin accuracy(Margin Acc.) degrades after distillation** in Table 5. This serves as an indirect indicator of how strongly the student is influenced by teacher information. Through experiments with various distillation terms, we observed that **terms satisfying PBRS(State dependent) tend to follow the original DPO dataset’s preference signal more faithfully**. This suggests that the original preference signal in the DPO dataset is more strongly preserved than the potential biases of the teacher, indicating that the student model remains primarily aligned with the dataset rather than being dominated by teacher bias.\n\n### **Q2.**\n\n---\n\n> Beyond current benchmarks, how does TVKD's generalization hold on highly out-of-domain or adversarial datasets to truly assess its robustness against reward hacking?\n> \n\n### **A2.**\n\n---\n\nTheoretically, TVKD is designed to follow the preference structure encoded in the DPO dataset, regardless of the teacher’s performance, thereby aiming to **preserve the original reward signal**. However, due to the inherently unstable training dynamics of reinforcement learning, incorrect auxiliary rewards can lead to slower convergence or suboptimal local minima.\n\nThe case suggested by the reviewer could occur in a hypothetical scenario where a teacher is trained to intentionally mislead students. Although this setting differs from a typical distillation setting and we cannot be certain, we predict that TVKD does not guarantee success in this scenario.\n\n### **Q3.**\n\n---\n\n> A more detailed analysis of the hyperparameter α's sensitivity is needed. How does its tuning affect the balance between teacher guidance and student learning from direct DPO, and what are practical tuning guidelines?\n> \n\n### **A3.**\n\n---\n\nWe present a sweep over **α** in Appendix F.3 and Figure 4. As expected, **a large α makes the student overly follow the teacher’s value signal, weakening the DPO objective, while a very small α reduces TVKD to standard offline DPO.** Within the range of **0.1–1.5**, we observed stable performance gains without significant drops, but we cannot rule out performance degradation for more extreme α values. As a practical guideline, we suggest starting with **α ≈ 0.5** and adjusting within the tested range.\n\n### **Q4.**\n\n---\n\n> Can the TVKD framework be generalized or adapted to distill from teachers aligned using other RLHF methods, such as those employing an explicit reward model (e.g., PPO-trained teachers)?\n> \n\n### **A4.**\n\n---\n\nIn principle, **TVKD can be naturally extended to teacher models trained with explicit Q-functions**, such as PPO or Soft Actor-Critic (SAC), because these methods inherently provide a value function. To our knowledge, however, **no prior work has explored distillation across fundamentally different RLHF frameworks (e.g., DPO, PPO, SAC) simultaneously**, largely due to their reliance on distinct data structures. Such an experiment would require a dataset where each query has both pairwise preference comparisons and scalar rewards.\n\nIn such scenarios, we are not certain whether a DPO-trained or PPO-trained teacher would serve as a better source for distillation, as the outcome would likely depend heavily on the characteristics of the dataset, such as label format, reward signal fidelity, and distributional coverage. We believe this opens up an interesting avenue for future research into how different RLHF paradigms interact with distillation dynamics."}, {"review_id": "FPogyoEUKT", "reviewer": "Reviewer_5RhB", "summary": "The paper introduces Teacher Value-based Knowledge Distillation (TVKD), a novel method for aligning small language models with human preferences by leveraging the value function of a larger, DPO-aligned teacher model. Unlike traditional approaches that rely on coarse binary feedback or behavior cloning, TVKD uses the teacher's value function to provide fine-grained, sequence-level reward signals via Potential-based Reward Shaping. It is a fully offline method that requires no additional rollouts. Experiments demonstrate TVKD's superiority over baselines across multiple alignment benchmarks like MT-Bench and AlpacaEval.", "questions": "The use of a value function as a shaping reward introduces a risk of reward hacking, where the student learns to exploit artifacts in the teacher’s value estimates rather than genuinely improving preference alignment. What are authors' opinions on this aspect?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Authors use the teacher model's value function as a soft reward signal, effectively address the coarse binary feedback limitation of standard DPO. They frame this as a Potential-based Reward Shaping (PBRS) problem, which bypass the need for online rollouts.\n2. Authors provide theoretical proofs and analysis on the method, which establishes strong theoretical guarantees.\n3. Experimental results show the effectiveness of this method on several models and benchmarks.\n\nWeakness:\n\n1. The paper could benefit from more discussion of the relationship between the hyperparameter $\\beta$ in DPO and the distillation strength $\\alpha$, as these parameters likely interact in important ways.\n2. While the paper highlights the advantage of being an offline method compared to online approaches like ADPA, it does not quantify the computational overhead TVKD introduces over standard DPO. It would be helpful to include a direct comparison on training time and resource usage.\n3. Compared to MT-Bench and Alpaca Eval, Arena Hard may be a more suitable benchmark nowadays.", "was_revised": true, "final_justification": "I have read the responses from authors and they address my concerns. I will keep my score.", "rebuttal": "We would like to thank reviewer 5RhB for their constructive feedback and for highlighting the value of our theoretical and experimental contributions.\n\n### **W1.**\n\n---\n\n> The paper could benefit from more discussion of the relationship between the hyperparameter $\\beta$ in DPO and the distillation strength $\\alpha$, as these parameters likely interact in important ways.\n> \n\n### **R1.**\n\n---\n\nAs shown in Equation (3), **β controls the sensitivity to preference margins between the winning and losing responses**, while **α determines the strength of the teacher’s auxiliary reward signal**.\n\nWe further explore their interaction in Appendix F.4 through a series of sweep experiments. The results show that **TVKD remains relatively robust across a wide range of α and β values**. \n\nJointly tuning both parameters can be computationally expensive. To address this, we recommend first selecting β using standard DPO without distillation, and then adjusting α with β fixed. This strategy is motivated by the observation that β has a greater impact on the overall reward scale, whereas α primarily modulates the influence of the teacher signal. For each β, it is possible to find a suitable α that ensures effective learning. We will include this recommendation in the camera-ready version.\n\n### **W2.**\n\n---\n\n> While the paper highlights the advantage of being an offline method compared to online approaches like ADPA, it does not quantify the computational overhead TVKD introduces over standard DPO. It would be helpful to include a direct comparison on training time and resource usage.\n> \n\n### **R2.**\n\n---\n\nAs shown in Appendix E, TVKD requires **only a single additional forward pass** of the teacher model over the entire dataset compared to standard DPO. This introduces a minor overhead relative to DPO, but the **cost of a one-time forward computation is relatively small.** Moreover, this additional cost is justified by the performance improvements achieved through distillation.\n\nWhen compared to **common distillation baselines such as GKD and SeqKD, TVKD incurs the same computational cost**, as all these methods require precomputing teacher outputs once. We also report a table comparing the **CPU wall time** of standard DPO and other methods. As the table shows, the time difference between DPO and TVKD is less than 25 minutes, and GKD and TVKD take almost the same time. In contrast, DCKD and ADPA incur significantly higher computational costs.\n\n| Method | CPU wall time |\n| --- | --- |\n| DPO | **3h 20m** |\n| GKD | 3h 42m |\n| DCKD | 4h 22m |\n| ADPA | 5h 2m |\n| TVKD(Ours) | 3h 41m |\n\n### **W3.**\n\n---\n\n> Compared to MT-Bench and Alpaca Eval, Arena Hard may be a more suitable benchmark nowadays.\n> \n\n### **R3.**\n\n---\n\n| Method | Arena-Hard-Auto Win Rate(%) |\n| --- | --- |\n| **TVKD(Ours)** | **54.4%** |\n| ADPA | 50.0% |\n| DPO | 34.8% |\n\nWe appreciate the reviewer’s suggestion and agree that Arena-Hard is an increasingly relevant benchmark. We report below the win rates of TVKD, ADPA, and DPO on the Arena-Hard-Auto [1] benchmark, where TVKD continues to demonstrate a performance advantage.\n\nWe will make an effort to include as many Arena-Hard results as possible in the final camera-ready version of the paper to further strengthen our empirical evaluation.\n\n[1] https://github.com/lmarena/arena-hard-auto\n\n### **Q1.**\n\n---\n\n> The use of a value function as a shaping reward introduces a risk of reward hacking, where the student learns to exploit artifacts in the teacher’s value estimates rather than genuinely improving preference alignment. What are authors' opinions on this aspect?\n> \n\n### **A1.**\n\n---\n\nSimply using the teacher value function directly for distillation can lead to reward hacking. However, TVKD mitigates this risk by incorporating the teacher value function into the student reward term in a way that satisfies the PBRS condition. PBRS theoretically guarantees that the optimal policy remains the same as when using only the offline DPO dataset. As shown in Table 5, performance degrades when using similar terms that do not satisfy the PBRS condition, which provides empirical evidence of the importance of PBRS. This is one of the key contributions of TVKD, as it demonstrates a safe and effective way to leverage the teacher value function."}]

Author rebuttal:

Meta-review: This paper introduces a method to distill preference alignment from a teacher model, called Teacher Value-based Knowledge Distillation (TVKD). The latter incorporates a value-based auxiliary reward that adheres to principles from potential-based reward shaping to make sure that the global reward structure and optimal policy can be preserved. The approach integrates into standard DPO training without the need for additional rollouts. Experimental results, in which the authors compare to multiple benchmarks and model sizes, are promising.

Overall, the reviewers are quite positive about the paper. In their reports, they raised a couple of comments and concerns, however, these could essentially be resolved in the rebuttal and subsequent discussion.
Decision: Accept (poster)