Pretrain Value, Not Reward: Decoupled Value Policy Optimization
Authors: Chenghua Huang
Lu Wang
Fangkai Yang
Pu Zhao
Qingwei Lin
Dongmei Zhang
Saravan Rajmohan
Venue: ICLR 2026 Poster (2026)
Keywords: LLM;RLHF;Value Model
Primary area: foundation or frontier models, including LLMs
TL;DR: DVPO pretrains and freezes a global value model from preference data, turning RLHF into stable policy-only optimization and matching or beating SOTA on major benchmarks.

Abstract: In this paper, we explore how directly pretraining a value model simplifies and stabilizes reinforcement learning from human feedback (RLHF). 
In reinforcement learning, value estimation is the key to policy optimization, distinct from reward supervision. 
The value function predicts the \emph{return-to-go} of a partial answer, that is, how promising the partial answer is if it were continued to completion. 
In RLHF, however, the standard pipeline first pretrains a reward model and then learns a value function online, even though no new reward signals are available once preference data is collected. 
This makes critic learning redundant, as the process of training a reward model and then deriving a value model is informationally equivalent to directly pretraining a value model.
Importantly, this requires no additional supervision, and our value model is trained on exactly the same data used for reward modeling. 
Building on this insight, we introduce \emph{Decoupled Value Policy Optimization} (DVPO), a framework that pretrains a \emph{Global Value Model} (GVM) offline and freezes it as a universal critic for policy learning. 
The GVM provides stable, fine-grained credit assignment without critic drift or trajectory sampling. 
Experiments across MT-Bench, Alpaca-Eval, and Arena-Hard demonstrate that DVPO matches or surpasses state-of-the-art RLHF methods. 
These results highlight RLHF can be reframed as policy-only optimization guided by a single pretrained value model. The implementation code for our method is available in \url{https://github.com/microsoft/DKI_LLM/tree/main/dvpo}

Published as a conference paper at ICLR 2026
PRETRAIN VALUE, NoT REWARD: DECOUPLED VALUE POLICY OPTIMIZATION
Chenghua Huang"; Lu Wang*! Fangkai Yang", Pu Zhao Qingwei Lin Dongmei Zhang ,Srraaan Rajmohan"
Schooo of Computer Science, Fudan University Microsoft
huangch22@m. fudan .dd. en
[wlu, fangkaiyana puzhao, dongmeiz,
ABSTRACT
In this paper, explore how directly pretraining value model simplifies and stabilizes reinforcement learning from human feedback (RLHF). In reinforcement learning, value estimation is the key to policy optimization, distinet from reward supervision. The value function predicts the return-to-go of a partial answer, that is, how promising the partial answer is if it were continued completion. In RLHE, however, the standard pipeline first pretrains a reward model and then learns value function online, even though no new reward signals are available once preference data is collected. This makes critic learning redundant, as the process of training a reward model and then deriving a value model informationally equivalent to directly pretraining value model. Importantly, this requires no additional supervision. and our value model is trained on exactly the same data used for reward modeling. Building this insight, we introduce Decoupled Value Policy Optimization (DVPO), a framework that pretrains a Global Value Model (GVM) offline and freezes it universal critic for policy learning. The GVM provides stable, fine-grained credit assignment without critic drift or trajectory sampling. Experiments across MT-Bench, Alpaca-Eval and Arena-Hard demonstrate that DVPO matches or surpasses state-of-the-at RLHF methods. These results highlight RLHF can be reframed as policy-only optimization guided by single pretrained value model. The implementation code for our method is available in https: //github. wwt.://...mcc.coomc BttC//DDDIIIIMMMSSEE wttM//tee//eeeeeeeeeeeeee mm htt///aan///////////////m n/dvpo
INTRODUCTION
The alignment of large language models (LLMs) with human preferences has become one of the core challenges of modern natural language processing. While scaling up L.MSs has led to remarkable gains performance and generality AAchiam et al.) [023]: Bubeck al. (2023), scale alone does not ensure that models behave in ways users actually desire, especially in complex reasoning, summarization, dialogue tasks. Reinforcement Learning from Human Feedback (RLHF) Christiano et al. (2017); Bai et al. (2022b); Song et al. (2024) has emerged critical step in shaping model behavior, enabling systems that are safer, more accurate, and more useful (Bai et al. 2022a| Ganguli et al., (002)).
A central component of RLHF is reinforcement learning (RL), where policies optimized with respect to value estimates of return-to-go. Existing RL.FF frameworks, however, have taken a circuitous path toward value estimation. First, because LLMSs lack interactive environment that provides ground-trute rewards, the community trains a reward model (RM) from preference data (Ziegler al. 2019 Ouyang al., 2022 Wang al. 2024). Second, this reward model is used to supervise either an online-trained critic (as in PPO (Schulman et al. (017) and its variants) or to indirectly estimate values through trajectory sampling (as DPO (Rafallv al. 2024). ReMax (Li et al. 2023). and GRPO (Shao et al. 2024 Both approaches are costly and unstable: critics drift
"Wor is done during an internship at Microsoft CCorrsponning author.

---

Published as a conference paper at ICLR 2026
during joint training, while sampling- based methods discard token-level credit assignment and suffer from high variance. (Yao et al. 2023 Huang et al. (2024)
This work explores simpler and more direct alternative, motivated by two idcas. First, if no new ground-truth rewards are available during policy training. then learning a value model online from a fixed reward model adds no fundamentally new information. Pretraining reward model followed by critic training is therefore informationally equivalent to directly pretraining a value model. Second, in open-ended tasks, rewards are largely policy-invariant: whether policy A or policy produces solution, the return is determined by correctness or preference, not by policy-specific stochasticity. This opens the door to amortizing value estimation into a Global Value Model (GVM), pretrained once on diverse trajectories and reused as frozen critic across policies.
We call this approach Decoupled Value Policy Optimization (DVPO). DVPO eliminates online critic training entirely: the GVM is pretrained offline to predict token-level returntooggo, frozen, and then used to guide policy optimization via a standard RL objective. Crucially, the GVM is trained on exactly the same preference data that would ordinarily be used to pretrain a reward model, requiring no additional supervision or annotations. This design avoids critic drift, reduces compute and memory overhead, and provides fine-grained interpretability through token-level attributions.
We present empirical evaluations across standard RLHF benchmarks, including MT-Bench ZZeng et al. (2023), Alpaca-Eval Dubois al. (2024), and Frena-Hard Li et al. (2024). Our results show that DVPO achieves performance on par with or surpassing state-of-the-ar RLHF methods while reducing GPU memory usage by 30-40% and training time by 30-45%. This indicates that rather than repeatedly retraining unstable actor-critic pairs or discarding value functions altogether, a single offine-pretrained value model can serve as stable, scalable, and generalizable critic for RLHE
2 RELATED WORK
Reinforcement Learning in Language Model Optimization. Reinforcement learning has emerged as prevalent method for fine-tuning large language models (LLMs), with Proximal Policy Optimiza-
tion (Schulman et al. 2017) and its variations (Ramamurthy et al. 2022 Wu et al. 2023) being the most prevalent methods. These techniques largely adhere to the actor-critic paradigm (Sutton 2018).
This approach alternates between training a value estimator for the current policy and leveraging it to enhance policy performance. This bilevel process may result in suboptimal policy, as demonstrated by empirical studies (Gao et al., (003). Moreover, the alternating optimization of the policy and critic models, along with the use of rewards provided by the reward model as environmental feedback, necessitates loading four models (including the reference model) simultaneously during training. This significantly increases training complexity and computational resource consumption Yao et al. 2023 Hu et al. 2024).
Many recent studies have sought to mitigate the computational complexity and re- source consumption of the reinforcement learning (RL) step in RLHF Methods such as DPO (Rafalloy
Efficiency RLHE
al. 2024) and its variants (Meng al., 2024 Ethayarajh et al. 2024 Hong et al., 2024) bypass reward modeling and the actor-critic learning framework by directly learning from preferences. However, existing research indicates that due their offline nature, these approaches exhibit
performance gap compared to online RL (Xu et al.] 2024). Some recent works have proposed a reward-only approach to reduce the training cost of the RL phase (Li et al., 2023 Gunter al. 2024. Shao et al.) 2024 AAmaddia et al., 20244 Yu et al. 2025). However, this method lacks value estimation and assigns the same reward score to each token, leading to high variance and instability during training (Hu) 2025). Unlike these approaches, our method pre-trains global value model (GVM) and leverages to guide RL training, providing token-level supervision signals. This not only reduces training resource consumption but also stabilizes the training process, achieving performance comparable to the original PPO.
Some recent works have attempted to learn value function and use it to guide the decoding phase of LLMs, thereby bypassing the RL optimization stage (Han et al. 2024 Kong et al., 2024 Mao et al.) 2024). However, this approach significantly increases inference complexity and raises inference costs. Some studies (Yuan et al.] 2025) also leverage pre-trained value models to guide policy optimization; however, they still employ actor-critic architecture, incurring substantial computational overhead.
2

---

Published as a conference paper at ICLR 2026
Model
Modela
RM Training
PPO
Raferenc Modd
Honward
Nookl
GVM Training
DVPO
Polay Model
Oi
Pollcy
Vilua
Mocel
Value
Polly
Nocul
Figure 1: Overview of DVPO. PPO requires pre-trained reward model to provide environmental feedback. It combines this reward signal with a learned value function estimate the action-value Q(s,a) In contrast, DVPO trains a Global Value Model (GVM) on the same offline data to produce fine-grained value estimates. This design removes the need for additional ground -trnthhhrhhhhh rewards during policy training and decouples policy learning from value learning.
In contrast, we leverage the learned value model guide RL training, where the pre-trained value model helps the policy model converge more stably (Noukhovitch et al. 2024).
METHOD
We propose DVPO for RLHF in which GVM trained once and then fixed guide policy updates. This approach removes the need for joint policy-value training and mitigates the associated computational overhead and instability. As shown in Figure [ our method comprises two primary stages: (1) Train GVM: Use offline trajectories (states, actions, returns, and policy-specific data) to learn a policy-conditioned action-value function Qs- Notably, returns can be derived from either preference
used in standard RLHF settings- or pretrained reward model. For detailed comparison of these sources, see Appendix G (2) Decoupled Value Policy Optimization: Freeze Qú e and optimize a policy using standard RL objective (e.g., PPO), taking advantage estimates from the fixed GVM.
3.1 PROBLEM SETTING
We model the sequence generation task in NLP as a Markov Decision Process (MDP). The response where y² y and y is the vocabulary. We assume 50 []. indicating an empty prefix. Given prompt x and the first t 1 tokens y <t the language model (LM) predicts the probability distribution for the next token as: To(: [xr, y (J)).
consists of T tokens, denoted by y y TT [v', ..2.....y'" ......y",
In this MDP formulation, the state is defined as 8; = [a. S* representing the prompt (i.e., a)) and the generated response up to step f. The action is the next generated token a, = y². The ground-truth reward at the sentence level, denoted by r(z, y). is provided by human feedback. To transform the sentence-level reward into token-level rewards r(s az), we apply simplified form of temporal difference (TD) learning. Specifically, we assign a reward of to each intermediate step and use the final step's reward as the sentence-level reward value.
The
By this definition, the cumulative return from step onwards is expressed as: G. Et k (())))))), which simplifies to G, TT"=r(2x y) when all intermediate rewards are zero. In RLHF, we rely on offline data D, which contains state action trajectories, returns, and policy behaviors. The dataset is defined as: D = {(T,, st, af, G where т; represents the i-th policy sequence of question answer pairs, (s), a;) is state-action pair sampled under that policy. Note that T; and (sf, a;) may refer to different or identical question-ansee answer pairs under the same policy. No additional ground-truth rewards collected during training.
behavior, typically expressed
3

---

Published as a conference paper at ICLR 2026
3.2 TRAINING A GLOBAL VALUE MODEL (GVM)
Our key insight is to learn a policy-conditioned action-value function QØ(T, 8, a) from offline data, where represents sampled trajectory capturing how the policy behaves in unrelated contexts. This conditioning approximates how well particular policy, embodied by T, would perform when taking action a in state s.
Policy Conditioning via Trajectories. Traditional actor-critic methods require online adaptation of the value function to the actor's evolving behavior. In contrast, we aim for a single, globul Qa that generalizes across different policies, thus avoiding iterative re-learning. Instead of conditioning on explicit policy parameters, we leverage trajectories T randomly sampled from the policy in question. Each is sequence of question-anse pairs (in LLM tasks) other interactions that reveal distinct policy traits (c.g., stylistic tendencies, correctness, domain expertise).
Formally, we parametrize: Qo(T,S,a) E EE rft r(s; ae)
so = s, ag = a, where
approximating. In practice, train the global value model (T a) using Temporal Difference (TD) learning. The target return G; is estimated based on future rewards and value predictions. Specifically, G, is computed
implicitly determines which policy TT- s)
G = r(t,, ar) + QQ 0++1),
where r(s;, at) is the reward for taking action a in state 8t, and Qo S1+1. a(+1) the predicted value of the next state-action pair.
The training objective for the global value model is minimize the TD loss:
(o) =
[I+e + 7Q 88+1, (++1) Qo(T, 86, a+))
(I)
This TD-based loss drives Q4 to iteratively adjust its estimates of the expected return, ensuring that the value function accurately reflects both immediate and future rewards. This approach is particularly suitable for offline RLHF scenarios, as it can handle sequences with deterministic state transitions.
3.3 DECOUPLED- VALUE POLICY OPTIMIZATION
In traditional actor- critor methods, both the policy (actor) and value (critic) are trained simultaneously. which introduces instability due to their interdependence. This issue is exacerbated in offline RLHF settings, where no environment rewards can be collected correct misaligned updates. To address these challenges, we propose a decoupled-valus policy optimization approach: the global value model Qo is trained and fixed before policy optimization, decoupling the learning dynamics of the actor and critic.
Policy Optimization Objective. Once the global value model Q+ converges, we fix its parameters and use it to guide policy updates. Let xo (a s) be the policy to be optimized. We adopt clipped PPO objective, which stabilizes policy updates by limiting the impact of large policy changes:
(Ppo(0) mimmm.inn (r.(0)A, clip(r.()), 1 - -e, +.
where (0) (aels))) is the importance sampling ratio, and is clipping parameter to prevent (0 2x) overly large updates.
The advantage function measures the relative quality of an action compared to the expected value of state. We define the advantage using the fixed value model:
A, = Qu(T, 8z, az),
where Qs is the value estimate computed during the GVM training stage (see $3.2). In offline RLHE, this static advantage definition provides a reliable signal for policy improvement without requiring dynamic value updates.
Our decoupled-valle value approach offers two key advantages: (1) By fixing Q we eliminate the "moving target" problem inherent in actor-critic methods, leading to more stable and predictable policy updates.
4

---

Published as a conference paper at ICLR 2026
(2) Since additional environment rewards can be collected, the static Qo provides all necessary supervisory information, enabling efficient reuse of the offline dataset for policy optimization.
In addition, DVPO does not require stronger assumptions than standard PPO-style RLHF. Because the GVM is pretrained once and fixed, policy optimization proceeds with stable guidance and avoids critic drift. We provide empirical evidence of this stability in Appendix [ smooth and reliable training curves compared to PPO.
where DVPO exhibits
3.4 EQUIVALENCE OF REWARD-CRITIC TRAINING AND VALUE PRETRAINING
In RLHF with fixed feedback, we first train a reward model Rg on preference dataset D. Existing pipelines then estimate values from this trained reward-eithe by learning an online critic from Rs (PPO-style) or by sampling many outputs and normalizing Rs scores Crucially, both routes consume no supervision beyond DD the value signal is entirely derived from the already-trained Rag- Therefore, the two-stage procedure (reward training) (value estimation from the trained reward) is informationally equivalent to directly pretraining a value model Q on the same D. This is the equivalence we claim: not that rewards and values are identical, but that deriving value signal from a fixed, pretrained reward adds no new information compared to pretraining the value model directly. DVPO leverages this by pretraining Global Value Model once and discarding the redundant estimation stage.
3.4.1 PRELIMINARIES
Let r(s,a) a) denote the unknown ground-truth reward function, and let 7 E [0. 1] be the dis- count factor. We assume access to an offline dataset D, from which we pretrain the following models. (1) Reward Model [s, a). which approximates r(s, a) with error bounded by ER- (2) Global Value Model Qp (s, a). which approximates the action-value function (=, =
E,, 1i 7'r((), a.) So s, do a], with approximation error bounded by eQ.
We merge any explicit conditioning (e.g.. trajectory т) into the state definition to simplify notation. Since no rewards are observed post-deployment, both R. and Q serve as static supervision signals throughout training.
Let (a s) be the trainable policy. We define the policy gradient objectives under each supervision signal as:
vOJR(TT)) =
EsraTo aaits
V log (a 8) AR(ss, a)
VoJO(TT) = Eanado aowgy
V log (a( 8) Ag(s, a)):
where T the stationary distribution under policy Ta. and the advantage functions are defined as:
An(s. a) as) b(s). b(s). AQ(s, = Qv b(s),
with b(s) baseline (e.g., state value function V(s)) reduce variance. The estimate 2 a) is computed by solving the Bellman equation using R2. following standard value iteration procedures. This structure parallels typical actor-critic methods, as discussed in (Sutton 2018). In contrast, (s(s, a) is fixed global value model pretrained offline. This is key distinction: unlike standard actor-critic methods (e.g., PPO), where the value function updated alongside the policy at each training step.
3.4.2 EQUIVALENCE LEMMA.
We formalize the observation that, under fixed preference data, (reward pretraining value estimation) provides no additional information beyond direct value pretraining.
Lemma 3.1 (Equivalence under fixed feedback), Let D be a fixed preference dataset and let Rs be a reward model trained on D. Assume: For all (s, a) in D, [R,(s, a) r(s,a)) | S en. 2. For the same distribution, both the reward- induced value R and the pretrained GVM Q2 satisfy
(x( (s. SEQ:
[Qp(s,)) (s)()))) SeQ.
5

---

Published as a conference paper at ICLR 2026
3. (Fixed feedback) No new rewards are observed during training. Then the induced policy gradients satisfy
V VoJog((T)) -
VoJqg (To) S k(CR,CQ):
with k((n, (Q) 0 as en, 'q - 0. Thus, reward-critic training and direct value pretraining are informationally equivalent on D.
The complete proof, including Bellman-based derivations and bounding arguments, provided in Appendix
Convergence Corollary. Building on Lemma! 3.1 we now consider DVPO policy updates. If policy updates are regularized (e-g.. KL--lliped as in PPO) and the GVM approximation error is bounded, then DVPO inherits the same monotonic improvement guarantee as PPO. In other words, replacing the online critic with a fixed pretrained GVM does not weaken PPO's local convergence properties, while eliminating critic drift in practice.
EXPERIMENT
4.1 TRAINING DATASETS
Stagel: We collected offline trajectories data train GVM. Specifically, selected the Ul- raaeeddbcc dataset(Cui et al. [202)), in which, for given prompt, multiple models generate responses that are annotated with scalar scores. Training instances for GVM take the form (prompt, response, r.ward,). We also explored training directly from pairwise preferences by assigning score of the chosen response and -1 to the rejected response. The resulting per- formance was comparable to training with scalar rewards, highlighting GVM's compatibility with different supervision formats; see the Appendix Gfoor details. Stage2: To prevent data leakage, we conduct RL training on the 10K prompts retained from the previous step, with GVM providing training feedback.
4.2 TRAINING SETTINGS
Base setting. We fine-tune a base model (LL.MAA--BB-use and LLaMA3.2-3B-Buss to obtain an SFT model. The data are split into three disjoint parts: SFT, reward-moder training, and RL. We then apply reinforcement learning to the SFT model. This setup follows the standard RL stage and is transparent and reproducible.
Instruction setting. We select an off-the-shelf instruction-tuned model as the SFT model. To facilitate comparison with related work(Li et al. 2023] Rafailov et al.) [2024), we choose the Mistral- 7B- 7B-Instruct--0.2 and LlaMA3-BB-Insie
These models have undergone extensive instruction fine-tuning processes based on the base model, resulting stronger capabilities compared the SFT model in the Base setting. For more training details, please refer to Appendix C
4.3 BASELINE AND EVALCATION
Baseline. For GVM, we compare it with scalarRM trained the same data. We also compare with the value model in PPO training. In PPO, scalarRM provides the supervision signal. For the policy model, we compare with the backbone model and with other RI. algorithms, including PPO, GRPO, DPO, and ReMax. In all these baselines, scalarRM provides the supervision signal.
Evaluation. To evaluate GVM, we use a held-out test set and reward-related benchmarks (e.g.. RewardBench). Because GVM produces token-level estimates for each response, we recover response-leve reward by applying a Bellman-style backup. We discuss our choice evaluation metrics in the Appendix. We evaluate the response quality of the policy models using the most popular benchmarks: MT-Bench(Zheng et al.) 2023). Arena Mera--Hard(Li et al.) 2024), and Alpaca- Eval (Dubois
al. 2024 Additionally, compare the ML.Ma----judde win rate the held-out test

---

Published as a conference paper at ICLR 2026
Table 1: Result of Base setting, Both methods were initialized from SFT and optimized. DVPO achieved the best benchmark scores across models with different parameter sizes. MT--eech scores range from 0-100
Llama3.2-3B-Ban
Llama3-8B-
Model
Mthench
Arena hard
AlpacaEval2
Mthench
Arena hard
AlpacaEval2
SFT
5.22
10.4
8.19
4.87
10.3
9.79
PPO
5.33
13.5
11.54
4.98
11.7
11.14
Remax
5.23
12.2
12.02
4.94
8.9
9.62
GRPO
5.46
13.4
10.86
4.91
10.4
10.27
DVPO
5.73
15.1
12.33
5.01
11.8
11.33
A
+5.10%
+4.7%
+4.14%
+1.40%
+1.5%
+1.54%
EXPERIMENT RESULTS
5.1
DVPO ON BASE SETTING
As mentioned earlier, to clearly evaluate the feasibility of DVPO compared to existing methods, conducted experiments on publicly available datasets. The datasets were split proportionally for supervised fine-tuning (SFT) and reward learning (value learning). Subsequently, the resulting reward model (value model) was used to guide the policy optimization for preference learning. Given the relatively small amount of data each stage (with the RL phase accounting for approximately 20% of the entire dataset), our focus in the Base setting is on relative improvement rather than absolute performance metrics. The results are presented in Table
DVPO demonstrates superior optimization performance. In experiments conducted on LLaMA3- 3B and LLaMA3-SB, DVPO consistently outperforms SFT significantly. Specifically, under the Base setting of LLaMA3-8B, DVPO achieves improvements of 0.2 on MT-Bench, 1.5 on Arena-Hard Hard, and 1.74 on Alpaca-Eval compared
SFT model. The improvements are even more pronounced with LLaMA3-3B, where DVPO achieves 0.51 increase MT--Bench, a 5-point increase Arena- Arena-Hard, and a 4.1int increase on Alpaca-Evee Eval. Similarly, as shown in Figure 2 on the held-out test set, DVPO achieves higher win rate compared to the SFT model. These results validate the robustness and effectiveness of DVPO.
The pre-trained value model provide sufficiently high performance ceiling. According to the experimental results in the Base setting, the final convergence performance of DVPO is very close to that of standard PPO (and in some cases, slightly exceeds it), indicating that the supervision signals provided by the pre-trained value model can support a sufficiently high performance ceiling. As actor-critic method, PPO trains the policy model and value model simultaneously, with the reward model providing environmental feedback. The value model, as bridge between immediate rewards (sentence-level) and long-term returns, offers finer-grained supervision signals noken-level) compared to immediate rewards alone. In DVPO, the global value Model(GVM)) is pre-trained on responses from different policies, enabling it to provide supervision feedback to various policies during the RL phase. The experimental results demonsirate that the fine-grained environmental feedback provided by DVPO can achieve a performance ceiling comparable to that of the actor-critic framework while significantly reducing training complexity and instability.
5.2 DVPO ON INSTRUCTION SETTING
To evaluate the effectiveness of DVPO in most RLHF scenarios (optimization starting from pre-aligned model), we selected Distral-7B-Instrucce model. An existing UltraRM reward model was used to collect labeled data for training the value model, value model also initialized from SFT model. The trained value model then guided the preference optimization of the SFT model. The results presented in Table 2 Experimental findings demonstrate that our method exhibits significant advantages over other approaches. Our main findings are as follows:
and LlaMA3-$B-Instrru as the SFT
7

---

Published as a conference paper at ICLR 2026
Table Result of Instruction setting. Instruction refers Mistral-7B-Instrue--- and LlaMA3- 8B-Instruct. All methods are initialized from their respective instruction models. Results marked with an (*) are sourced from the original paper or inference from the open-source release.
Mistral 7B
Llama3 8B
Method
Mtbench
Arena hard
AlpacaEval2
Mthench
Arena hard
AlpacaEval2
Instruction
6,60*
12.6
17.11*
6.90*
20.6*
34.38*
ReMax
6.67*
21.9*
20.55*
7.62
30.8
36.04
DPO
6.30
16.3
26.80
7.01
32.6
40.32
GRPO
6.31
21.8
27.19
7.53
34.1
35.17
PPO
6.55
19.4
19.62
7.55
36.3
34,98
DVPO
6,79
24.7
27.43
7.72
39.2
42.59
A
+1.90%
+12.1%
+10.32%
+8.20%
+18.6%
+8.21%
DVPO significantly enhances the performance instruction models. Compared the original In-ttl7t---stt DVPO consistently demonstrates performance improvements across all benchmarks. Specifically, achieves 0.19 improvement on MT-Bench and increases the win rate by 12.1% on Arena-Hard. Additionally, it enhances the conttlccottolleed win rate by 10.32% Alpaca- Eval. Furthermore, compared to DPO, DVPO also exhibits notable advantages, on the Mistral setting, outperforming 8.4% in Arena-Hart and 0.49 in MT-Bench On the Llama setting, outperforming 0.71 in MT-Bench. These results indicate that the pre-trained value model can effectively and reliably guide the optimization of policy models.
Finer-grained feedback signals lead to superior performance. Compared to reward-only methods such ReMax and GRPO, DVPO demonstrates significant performance advantages on Mt-Bench, Arena-hard and Alpaca-Eval. In ReMax and GRPO, the feedback signal is at the sentence level from the reward model, assigning a single reward score to the entire sentence and treating all tokens as equally contributing to the overall score. This design prevents the policy model from learning fine-grained preferences during the RL phase, resulting suboptimal performance. In contrast, the GVM trained with TD-learning over prefixe--assigns assigns different values to different parts of a response; as shown in Figure 3 decisive reasoning tokens receive high values while misleading continuations receive low values. By assigning return value to cach token, GVM facilitates more effective training of the policy model. In addition to providing fine- fine-graines returns, DVPO retains the on- ae-policy characteristics of the original PPO, enabling a larger exploration space and offering higher performance ceiling.
6 ANALYSIS
6.1
COMPARSION OF SCALAR RM IN REWARD BENCH
To verify that GVM can effectively learn state-value estimates from offline trajectories, we aggregate its token-level values into final, outcome-level reward and evaluate reward-related benchmarks. We also compare against scalar reward models (ScalarRM) trained on same data. As shown in Table the average RewardBench (Lambert et al. 2024) scores of GVM and ScalarRM are compara- ble overall, but their performance profiles differ. On the Chat (casy) subset, ScalarRM consistently outperforms GVM, suggesting that sequence-level prediction with pairwise Bradley-Terry (BT) loss readily captures global objectives and models overall preference on easier cases. In contrast, on the Chat-Hare subsets, GVM trained with value cormattion-genraliizze better to more complex problems. In sum, given identical data, GVM and ScalarRM achieve similar aggregate performance, but GVM provides finer-grained and more accurate supervision at the token level; relative to distributing single sequence-leve. reward down to tokens, these direct value signals improve policy learning. Additionally, we provide illustrative GVM case studies in Appendix
(TD) learning for token-level
6.2 COMPARSION OF ACTOR-CRITIC VALUE MODEL.
Beyond our comparison with ScalarRM, we also analyze the value model the actor-criti critrcc paradigm. As shown in Table 4 in-distribution data GVM significantly outperforms the A/C value model,
8

---

Published as a conference paper at ICLR 2026
Table In the RewardBench results, GVM and ScalarRM are trained on the same data.
Table 4: Result of Ultrafeedback testset. A/C denotes the PPO critic (value model), evaluated PPO training convergence.
Model
Method
Chat
Chat-Hard
Safety
Reason.
Llama3 8B
GVM
88.3
67.5
66.8
79.2
SclarRM
95.5
58.5
73.9
65.4
Method
Llama3 8B
Mistral 7B
Mistral 7B
GVM
75.4
61.4
66.7
64,4
GVM A/C
68.1
64.5
SclarRM
93.8
52.4
64.1
60.3
60.6
57.6
Table Memory footprint and computational cost analysis. M training time memory (parame- ters, optimizer states, activations); Minteri inference-time memory; Ggene" computational cost for generating n responses; Cback: computational cost for backpropagation and optimizer update.
ScalarRM
GVM
PPO
ReMax
GRPO
DVPO
Musin
Merain
Murain
2 X Murain
Mtaiin
Mirain
Miaiin
Minfer
2 X Minfer
2 Minfer
x Minfer
2 Minfer
Ogene
Ogene
X Cgene
12 X Cgene
Cgene
Cback
Cback
Cback
2 Cback
Cbak
Cback
Oback
indicating that a value model pretrained directly from preference data more effectively estimates response state values than the critic jointly trained with PPO. Moreover, this advantage transfers across different base (backbone) models. A/C value model is trained to fit returns under the current policy. It is tightly coupled to the policy, and its data distribution shifts during training, so it best predicts returns for outputs likely under that policy. In contrast, GVM learns from large-scale preferences which state transitions increase decrease reward. It is policy-agnostic and therefore yields more accurate value estimates.l In this sense, our value model is "global" in that (i) it is policy-agnostic rather than coupled to single behavior policy, and (ii) it evaluates each action from trajectory-level perspective by considering how the entire trajectory contributes to the final outcome.
To further compare the performance of GVM and A/C under dis- tribution shifts, we evaluate the GVM and A/C value model from Ultrafeedback experiment on a new distribution, new prompts in the test split of the HH-RLHF dataset. The result presented in Tabel [ We also compute the accuracy metrics. On the new distribution, both GVM and A/C show lower performance, but GVM still leads. This indicates that GVM's pretrained value estimates remain robust under shift. Pretraining offline trajectories helps the value model Iearn value estimation more faithfully and improves generalization.
Table 5: Result of HH- RLHF testset.
Method Llama3 8B
Mistral 7B
GVM A/C
63.3
60.8
57.5
6.3 COMPARSION COMPUTATION
As the Table 6 shows, training GVM and ScalarRM requires nearly the same GPU memory. Both use the same base model with a single linear head (hidden size - 1). Their training time is also similar, since each step performs one backpropagation. Overall, the compute budgets for GVM and ScalarRM are essentially equivalent. Among RL methods, PPO is the most messurce-intensiie because it trains two models in parallel (policy and value). ReMax, GRPO, and DVPO have comparable memory costs, but group-based methods such as GRPO must generate multiple responses per prompt, which increases training time. In sum, DVPO offers the best balance of memory and time and, with token-leved supervision from GVM, outperforms the other methods. We provide detailed analysis of computational resources and the training curves the Appendix
7 DISCUSSION
Conclusion. We propose Decoupled Value Policy Optimization (DVPO), framework that eliminates joint actor-critic training in RLHF by leveraging a pretrained global value model (GVM). Our theoretical analysis proves the functional equivalence of reward and value models under the constraint

---

Published as a conference paper at ICLR 2026
of reward feedback, justifying the use of fixed GVM for efficient and scalable optimization. Empirical results demonstrate that DVPO achieves comparable or surpasses performance to state-of- the-art RLHF methods multiple benchmarks. Future work will focus refining the value model's training process to enhance estimation performance.
Limitations. DVPO assumes that the offline preference dataset provides adequate coverage of relevant trajectories and that the Global Value Model (GVM) can approximate token-level returns within bounded error. However, the need for sufficiently diverse preference data inherent reward learning, not specific to the GVM. Unlike joint actor-critic training, DVPO does not adapt the critic as the policy evolves; thus its guidance is fixed and may fail highly con-stationary settings. Our equivalence analysis also holds only under the constraint of no new reward signals- if additional human or environment feedback becomes available during training. a static GVM cannot fully exploit it. A promising future work to mitigate this limitation is to extend DVPO to semi-online regime by periodically refreshing the GVM with newly collected preference data, while carefully controlling the update frequency preserve the stability of the training dynamics. Finally, although DVPO reduces computation and improves stability, it may inherit biases from the reward-preference data itself, which remain open challenge for all RLHF methods.
ETHICS STATEMENT
We adhere to the ICLR Code of Ethics. This study did not personally identifiable information sensitive attributes, and involved no direct interaction with human subjects. Our institution determined that the work does not constitute human-subjects research (or qualifies for exemption); therefore, IRB approval was not required. There are no competing interests or sponsorships that could unduly influence the results.
REPRODUCIBILITY STATEMENT
Our study uses open-source datasets; their names and download URLs are listed in the relevant sections of the paper. Our complete training and evaluation code is provided in wn/ntpss:/ / /ttp:://githie thub com//iccossott//
LLM/treed /ateee/aan////e dvpo The appendix details the full hyperparameter settings and dataset splits. Where permitted, we release model checkpoints to facilitate verification of the reported results. A permanent public repository will be made available upon acceptance.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 2rXiv:2303.08874. 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin. Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in Ilms. arXiv preprint arXiv:2402. 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022b.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 0441v22400.04445. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky,, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2442.01306, 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Chemn Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075. 2024.
Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization. arXiv preprint arXiv:2405.06639 2024.
Jiwoo Hong, Noah Lee, and James Thore. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 11170-11189, 2024.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266 2025.
Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An casy-to-use, scalable and seigh-erffrmance rlhf framework. arXiv preprint arXiv:2405.11143 2024.
Shengyi Huang, Tianlin Liu, and Leandro Von Werra. The n implementation details rlhf with ppo. In The Third Blogpost Track at ICLR 2024, 2024.

---

Published as a conference paper at ICLR 2026
Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang. Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: A control perspective. arXiv preprint wrxXvv22406009990 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 1939, 2024.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effec- tive, and efficient method for aligning large language models. arXiv preprint arXiv:2310.1050,, 2023.
Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, and Anh Tuan Luu. Don't forget your reward values: Language model alignment via value-based calibration. arXiv preprint arXiv:2402.16030 16030, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafer Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210001241, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 064477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowci Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint MxXi:22402.03300 2024.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang. Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990-18998, 2024.
Richard S Sutton.
Reinforcement learning: An introduction. A Bradford Book, 2018.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint wrXiv:2401.060080 2024.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for Ilm alignment. arXiv preprint arXiv:2310.00211 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? a comprehensive study. arXiv preprint 4404 10719, 19, 2024.
12

---

Published as a conference paper at ICLR 2026
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed- chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint S30::2308.013200 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu,, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 1447503...44,, 2025.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:250..05118, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1rKiv:1109900000 2019.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language model solely for sentence-level copyediting (grammar, clarity, and fluency) of text written by the authors. All LLM suggestions were reviewed and accepted by the authors, who take full responsibility for the final content.
B PROOF
Proof. Converting R to a Value Function. Using the pretrained reward model R a), we compute a surrogate action-value function (s, a) by solving the Bellman equation offline:
2 a) = Rs a) y E E (2]'''a
Under standard offiine RL assumptions (including sufficient coverage of the state-action space), (", a) converges an approximation of (s, a) with an additional error term eœ2Q that depends on
Converting Q an Approximate Reward Signal. Conversely, if Qu a) approximates (" a). then rearranging the Bellman equation yields an approximate reward:
Rep a) (s, Eg 1Q (o(s',a')) a')]
The error in this reconstruction is bounded by éV2R- which is a function of Eq-
Policy Gradient Equivalence. We now explicitly define the two policy gradient objectives used in our analysis.
The policy gradient with reward-derived value function, denoted VoJR((To). is given by:
(RTo) Eg-ato, awte Vo To(a s) -AR(s,s))) a))
where the advantage is computed using surrogate value function ( a) obtained by solving the Bellman equation using the pretrained reward model (s,sa)):
AR(s,a) a) 25 (s, a) b(s).
The policy gradient using a pretrained global value model, denoted as VoJo (TO). is defined similarly:
VoJo( Egndro, annto
Vo log To s) AQ(s,
where the advantage is: and Q is the pretrained global value model.
As(s, a) = -s
In both cases, b(s) is a baseline (e.g., state value or Monte Carlo average) used to reduce variance. This aligns with standard actor-critic or REINFORCE-SVe policy gradient frameworks (Sutton 2018). The gradients are computed using samples from the current policy ma, and the fixed models RS and Q provide the learning signals.
We assume:
@5 a) - Q" a)) S ER2Q,
(Qv(s,a) (*, Sea.
and thus the difference in advantage estimates bounded:
||An(s, a) As(s,a)) < CR2Q + cQ.
Combining the expressions above, the policy gradients differ by:
(V)JR(TO) VeJo (Te)) x(ER,
where k is a bounded, non negaaivee function of the approximation errors, satisfying n(En: 6q) + n, EQ
14

---

Published as a conference paper at ICLR 2026
This formalizes claim that, under no new reward feedback, the reward-derived and value- gradients converge to the same update direction.
Static Supervision. Since additional ground-truth rewards collected, both R&g and Qu remain fixed throughout policy training. Consequently, their induced gradients remain constant over time, ensuring that the policy updates they drive equivalent up to the bounded emror k(cn, (Q).
The steps above demonstrate that, under our assumptions, the policy gradient updates based on the pretrained reward model and the global value model equivalent as the approximation enrors vanish. This completes the proof.
Left wins
Right wins
Liama3 3B DVPO SFT
32.56%
39.95%
27.48%
Model Comparison
Llama3 3B DVPO vs PPO
32.33%
38.49%
29.09%
Llama3 8B DVPO vs SFT
30.58%
43,68%
25.73%
Liama3 DVPO vs PPO
28.02%
43.49%
28.48%
20
40
60
80
100
Percentage
Figure 2: Results of the model on the Ultrafeedback held-out testset. We employed GPT4o as a judge assess the quality of model-generated responses. Performance measured using the win rate, where Left represents DVPO, and Right represents the baseline model for comparison.
C TRAINING DETAILS
SFT training. We the following hyperparameters for instruction fine-tuning training. We employ a learning rate of 2e-5 with cosine decay, warmup steps, and a batch size of 16. We calculate the loss only for the target tokens rather than the full input sequence, and we train for epochs the training data. we conduct the training on 8 NVIDIA A100 80G GPUs.
Reward training. To enable the model learn the relative ranking among different responses, we use pair-wise loss. The task type is sequence classification. We use a learning rate of 2c-5 with linear decay and the AdamW optimizer for training over 2 epochs, with a batch size of We conduct the training on 8 NVIDIA A100 80G GPUs
DVPO and Baselines training. For PPO training, we use learning rate 2e-6 and set the maximum generated sequence length to 1024. We employ batch size of 8 and a mini-batch size of 2, with PPO epochs and gradient accumulation steps. The KI. coefficient set to 0.05, and in Appendix H conducted an ablation study the KL coefficient. The target KL divergence KLaaget is set to 0.1. For a fair comparison, training for DVPO and other baselines was conducted using the same hyperparameter settings (e.g., batch size, sequence length, etc.). For GRPO training, roll out responses for each prompt.
Global value model training. We initialize the value model from the SFT checkpoint and optimize token-level squared error objective with a discount factor = 1. The training is conducted with batch size of 32, sequence length of 1 1024, and learning rate of 2e-6.
15

---

Published as a conference paper at ICLR 2026
Question:
As an island, is Beijing located in Asia?
Response1:
Be
ij
ing
ii
jislan
Value1:
-0.1936
0.0333
-0.0821
-0.2597
0.2099
-0.4590
-0.1500
Response2:
.Be
IJ ing is
.an
islland
located
Value2:
-0.0883
-0.4993
0.1845
-0.6177
-0.4766
-0.1289
-0.1526
Figure 3: An example of the supervisory signal provided by a Global Value Model (GVM). The GVM is capable of providing token-level feedback. In this example, the GVM assigns lower value to the incorrect response (response2: "is an island") and a higher value to the critical token "not" in the correct response (responsel: "nol island")).
D GVM CASE STUDY
For the same question, "As an island, is Beijing located in Asia?", the value model provides fine- grained supervisory sigmals for two different responses. The GVM assigns specific values to each token in the responses. These values represent the model's assessment of the importance correctness of each token in the given context.
As shown in Figure For Response 1, the critical token "not" is given higher value (0.2099), highlighting its significance in forming the correct response, "not an island." For Response 2, the GVM assigns lower values to incorrect tokens, such "is" (-0.6177) and "an" (-0.4766), indicating their contribution to the incorrect response, "is an island." This token-level evaluation demonstrates the GVM's s ability guide learning by penalizing incorrect responses and reinforcing critical tokens in correct responses, thereby enhancing training accuracy and interpretability.
E GVM PERFORMANCE
We evaluate performance of the GVM from multiple perspectives. Specifically, we observe that the GVM assigns higher value scores good actions and lower value scores to bad actions, as illustrated in Figure 3 We evaluate the global value (VVel(GVM) on a held-out test set. For each question, expect the value model assign higher values to good responses compared to bad ones. calculate the accuracy under various metrics (mean, Pl, etc.) to assess the model's performance.
Table presents the performance metrics of the Global Value Model (GVM) initialized from different size models. The metrics include the mean value, which indicates the average score assigned by the GVM, and various percentiles (P1, P5, P10, P90, P95, and P99), representing the corresponding value distributions. The accuracy metric evaluates the proportion of cases where the GVM correctly assigns higher value good response compared bad response, thereby reflecting the effectiveness of the GVM in providing accurate feedback.
As shown in the table, the value model's performance exhibits clear scaling law, with larger models achieving higher accuracy. Moreover, when trained on the same dataset, it delivers superior optimization outcomes compared to the reward model. These results demonstrate the GVM's ability to provide reliable token-level evaluations.
F ANALYSIS OF COMPUTATION EFFICIENCY
DVPO improves upon RLHF by leveraging a pre-trained value model to provide environmental feedback, resulting in enhanced computational efficiency, stability, and convergence speed.
Memory Efficiency. As shown in Table 7 the experimental setup for standard PPO and DVPO was kept identical. Under standard training conditions, DVPO achieves approximately 30-40%
16

---

Published as a conference paper at ICLR 2026
Table 7: Computational results under the Base setting. For the 3B model, we used Deepspeed Zero-1 strategy: for the 8B model, due to OOM issues with Zero- 1, we used Zero-3. We report GPU memory consumption per GPU (GB) and time per step (8) on an 8 A100 setup.
(lama3-8B-Bam
Llama3.2
Method
Mem (GB)
Time (s)
Mem (GB)
Time (s)
PPO
78,96
70.24
41.43
23.00
ReMax
61.23
65.96
26.87
44.13
GRPO
64.35
57.93
28.20
22.17
DVPO
60.49 (-23.4%)
48.00 (-31.7%)
27.50 (-33.6%)
13.00 (-43.5%)
reduction in memory usage. For instance, with LLaMA-3B, we trained using Zero-I 8 A100 GPUs with batch size of 4. Standard PPO requires 41.43 GB of memory due to the need to simultaneously load 4 models (policy model, critic model, reference model, and reward model), with models (policy model and critic model) requiring activation. In contrast, DVPO only requires 27.5 GB of memory, as it only loads the policy model, global value model, and reference model, with only the policy model being activated during training. For LLaMA-8B, we used DeepSpeed Zero-3 with batch size of 4. Even under this configuration, DVPO effectively reduces memory usage, enabling support for larger batch sizes during training.
Time Efficiency. DVPO accelerates the training process, requiring only half the time per batch compared to standard PPO under the same batch size. This improvement stems from the fact that, during the generation phase, DVPO requires outputs from only three models, whereas standard PPO relies four. Additionally, during the backpropagation phase, PPO updates both the policy and value models, while DVPO updates only the policy model. Overall, under identical settings (batch size), DVPO achieves 1.76x speedup in training (1.76 = 23/13) 13). By employing the GVM for advantage estimation, DVPO requires fewer generated samples during the rollout phase and thus achieves faster training compared to other memory-efficiee RL algorithms (e.g., ReMax and GRPO).
Training Step. We observe that using a pre-trained value model to provide token-level fine- grained environmental feedback enables faster model convergence, which aligns with the findings in (Noukhovitch et al.) 2024). Specifically, under identical experimental settings (same batch size, experience replay count, etc.), DVPO requires only approximately 3/4 of the interaction steps to converge. As shown in Figure 4 in the LLaMA3-8B experiment, the reward curve of PPO reaches its peak at 1250 steps, whereas DVPO achieves optimal performance around 800 steps. Similarly, in LLaMA3-3B, PPO reaches its peak at 600 steps, while DVPO converges in 350 steps. Since DVPO provides leeell supervision feedback and pre-trained advance, can be considered a form of warm-starting, thereby accelerating policy convergence.
G ANALYSIS OF REWARDS ASSIGNED
We compared two distinct reward distribution schemes, and the resulting training performance is shown in the Table g
From Reward Model For scenarios in which a reward model is available (used for a fair comparison to PPO), we first leverage preference data (e.g., tuples of the form (Q, A1)) to obtain scalar scores S1 and S2 for A₁ and Ag via the reward model. Each tuple (Q, A, S) is then treated as trajectory, with serving as the scalar return, and the GVM is trained on this token-level data using temporal-die -iifrrrl-difffrence learning.
From Preference Data. For scenarios without reward model (for broader applicability), we learn the value model directly from preference data. Specifically, we assign +1 to the preferred response and -1 to the less preferred one; these labels are treated as scalar returns to supervise the GVM directly, obviating the need for any reward model.
With the same data and base model, the value model from reward model accuracy is 0.64, and the value model from preference data is 0.67. The two training approaches produce virtually identical outcomes, demonstrating the GVM's tolerance for disparate data formats and its ability to be trained for varying application scenarios.
17

---

Published as a conference paper at ICLR 2026
Training vs
BB)
Liama3 PPO
Liama3 88
PPO
2,45
Llammdd 30 PPC (Rewwr))
Ulama 50 PPO (Rawwrs)
5.0
2.33
2.33
MM Aww you
Reward
1,28
5.2
345
5.0
3.5
Steps
Steps
Liama3 38 DVPO
Llama3 88 DVPO
1.2
Llemmd DYPO (melu))
Unmma DVPO (Velaw)
0.17
3.0
Value Score
6.15
Value
6:14
0.4
6:12
64
0.18
230
Steps
Figure 4: Learning curve of the policy model during the RL stage under the Base setting. DVPO demonstrates faster and more stable convergence compared to other methods.
Table 8: Value model performance different size models.Mean Value is the mean of the value scores over the trajectory. Pk Value denotes the k-th percentile of the value distribution in trajectory.
METRIC
LLAMA3 8B
LLAMA2 13B
LLAMA3 3B
MISTRAL. 7B
MEAN VALUE
68.05
70.55
63.61
64.51
PI VALUE
57.15
63.80
55.00
56.02
P5 VALUE
60.24
65.90
57.15
58.46
PIO VALUE
62.50
67.70
58.45
59.83
P90 VALUE
66.23
69.05
64.40
61.47
P95 VALUE
67.95
67.55
62.55
60.30
P99 VALUE
66.3
64.75
56.99
59.94
H ABLATION ON THE KL COEFFICIENT
To search the suitable KL. coefficient, we performing grid search in the range {0.01, 0.05, 0.10). For each setting, we evaluate the resulting policy on MT-Bench, Arena-Hard, and AlpacaEval2. As shown in Table 10 DVPO achieves consistently strong performance across all three values, and baseline under every tested KL coefficient. We adopt 0.05 as default value in our main experiments, which is aligned with common practice in prior RLHF work.
remains superior
I GPT4 EVALUATION PROMPT
A crucial element of experimental framework is the evaluation of win rates using GPT-4. In this section, we provide the prompts utilized to generate win rates for held-out test set in Ultrafcedback dataset. All experiments responses was randomized for each evaluation to ensure unbiased results.
conducted using the gpt-4o-20240806 model. The sequence of
GPT-4 judge system prompt:
Review the user's question and the corresponding response using the additive 5 5-pointscoring system described below. Points accumulated based the satisfaction of each criterion:
18

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of learning value models from reward data and preference data
ACCURACY
FROM REWARD
FROM PREFERENCE
MEAN VALUE
64.51
67.85
PI VALUE
56.02
58.60
P5 VALUE
58.46
62.80
PIO VALUE
59.83
63,95
P90 VALUE
61.47
65.85
P95 VALUE
60.30
65.80
P99 VALUE
59.94
64.05
Table 10: Ablation the KL coefficient for DVPO.
KL
MT-Bench
Arena-Hard
AlpacaEval
0.01
8.50
37.50
41.66
0.05
7.72
39.20
42.59
0.10
7.73
38.60
40.96
Add I point if the response is relevant and provides some information related to the user's inquiry, even if it incomplete or contains some irrelevant content.
Add another point if the response addresses substantial portion of the user's question, but does not completely resolve the query or provide a direct answer.
Award third point if the response answers the basic elements of the user's question in useful way, regardless of whether it seems to have been written by an Al Assistant or if has elements typically found in blogs search results.
Grant fourth point if the response is clearly written from an Al Assistant's perspective, addressing the user's question directly and comprehensively, and is well-organized and helpful, even if there is slight room for improvement in clarity, conciseness or focus.
Bestow fifth point for response that is impeccably tailored to the user's question by AI Assistant, without extraneous information, refecting expert knowledge, and demonstrating high- quality, engaging, and insightful answer.
After examining the user's instruction and the response, please first give the total score. Then provide brief explanation of your total score, up to 100 words. Output with the following format: Score: <total score> Evaluation evidence: <your brief explanation here>
from the Al Assistant perspective, utilizing web search knowledge necessary. To evaluate the response in alignment with this additive scoring model, we'll systematically attribute points based on the outlined criteria.
Remember
User input prompt:
(question) <response>: (answer)
<Question>:
19

---

Published as a conference paper at ICLR 2026
Table 11: case study for GPT4 evaluation: LLama3 8B SFT
Input
Compare and contrast Harry Potter with Percy Jackson series.
Llama3 8B SFT
Sure, I'd be happy to help you compare and contrast the Harry Potter and Percy Jackson series!Both Harry Potter and Percy Jackson are popular fan- tasy series that have captured the hearts readers around the world. Here are some key similarities and differences between the two:Similarities:1 Magical elements: Both series feature a magical aspect, with Hogwarts School of Witcheraft and Wizardry being the setting for Harry Potter and Camp Half-Blood for Percy Jackson.2. Mythological references: Both series draw heavily Greek mythology, with Percy Jackson's 's stories being based on the Labors of Hercules and other Greek myths.3. Relatable protagonists: Both Harry and Percy are relatable and charming protagonists that readers can connect with. with.Difffrences.1.. Setting: While both series take place in magical worlds, the settings are quite different. Harry Potter's stories take place in more traditional school setting, while Percy Jackson's 's stories take place in a summer camp atmosphere.2. Tone: Harry Potter a more lighthearted and comedic series, while Percy Jackson takes darker and more serious tone, exploring themes of mortality and responsibility.3. Plot structure: Harry Potter has more linear plot structure, with each book building on the previous one's events. Percy Jackson, on the other hand, has episodic structure, with each book focusing
different adventure or challenge for the characters.4 4. Themes: While both series explore themes of good and evil, Harry Potter focuses more on the importance of friendship and loyalty, while Percy Jackson delves into the consequences of power and responsibility.Im In conclusion, while both Harry Potter and Percy Jackson share many similarities, their differences in setting, tone, plot structure, and themes offer unique reading experiences for readers. Both series are highly recommended for anyone looking for magical and engaging read!
GPT4-as-Judge
Score: 3
The response provides relevant comparison of the Harry Potter and Percy Jackson series, highlighting key similarities and differences in magical elements, settings, tone, plot structure, and themes. It addresses the user's question by outlining basic elements and offering useful comparison. However, the answer contains inaccuracies, such as misrepresenting the tone of both series and the nature of mythological references in Percy Jackson. Additionally, the response could be more precise and direct, as some points are not entirely accurate or comprehensive. Thus, meets the criteria for a basic, useful response but lacks thoroughness and precision for a higher score.
Evaluation evidence:
20

---

Published as a conference paper at ICLR 2026
Table 12: A case study for GPT4 evaluation: LLama3 8B PPO
Input
Compare and contrast Harry Potter with Percy Jackson series.
Llama3 8B PPO
Sure, I'd be happy to help you compare and contrast the Harry Potter and Percy Jackson series!Both Harry Potter and Percy Jackson are popular fan- tasy series that have captured the hearts of readers around the world. Here are some key similarities and differences between the two:Similarities... Magical elements: Both series feature a magical aspect, with Hogwarts School of Witchcraft and Wizardry being the setting for Harry Potter and Camp
being the setting for Percy Jackson.2. Teenage protag- onists: The main characters in both series are teenagers, navigating their way through adolescence while also facing epic challenges and learning valuable life lessons.3. Epic adventures: Both series involve the protagonists on thrilling quests, battling mythical creatures, and uncovering secrets that could change the course of history.4. Portal to another world: In both series, there is a gateway to another world (Harry's cupboard under the stairs and the portal at Camp Half-Blood) that allows the protagonists access to realm filled with magical creatures and unparalleled adventure. Differences:: Setting: While both series take place in magical worlds, the settings are quite different. Harry Potter is set in a medieval-style world with castles and magical creatures, while Percy Jackson is set in modern-day world with technology and urban landscapes.2, Tone: The tone of the two series is quite different. Harry Potter is lighthearted and comedic, while Percy Jackson is darker and more serious, with themes of mortality and morality.3. Mythological references: While both series draw on mythological references. the specific myths and legends used are different. Harry Potter relies more on European and British myths, while Percy Jackson draws on Greek and Roman mythology.4. Magic systems: The magic systems in the two series are also different. Harry Potter uses a spell-based magic system, while Percy Jackson uses more physical, athletic approach to magic.5. Protagonists: The protagonists in the two series are quite different. Harry Potter is a boy wizard who discovers his true identity and heritage, while Percy Jackson is a demigod who must navigate the complexities of his divine heritage and mortal life.6. Plots: The plots of the series are distinct. Harry Potter focuses on the battle between good and evil, with Harry as the prophesized hero who must face off against Voldemort. Percy Jackson, on the other hand, is about the struggles of the demigods and the impact of their mortal lives on their divine heritage.In In conclusion, both Harry Potter and Percy Jackson are captivating fantasy series that share some similarities, but also differ in many ways. While both offer engaging and entertaining reads, they appeal to different audiences and perspectives.
GPT4-as-Judge
Score: 3
The response provides relevant comparison of the Harry Potter and Percy Jackson series, addressing the user's ' question with mix of similarities and differences. It covers key aspects such as magical elements, protagonists, and settings, offering a basic understanding of both series. However, the response contains inaccuracies, particularly in tone descriptions and settings, which detracts from its comprchensiveness and accuracy. While it offers useful comparison, does not fully address the user's question with the depth or precision expected from an AI Assistant, nor does reflect expert knowledge or exceptional insight.
Evaluation evidence:
21

---

Published as a conference paper at ICLR 2026
Table 13: A case study for GPT4 evaluation: LLama3 8B DVPO
Input
Compare and contrast Harry Potter with Percy Jackson series.
Llama3 8B DVPO
Sure, I'd be happy help! Both the Harry Potter and Percy Jackson series popular and successful book series that have captivated readers worldwide. Here some key similarities and differences between the two:Similarities... Magical elements: Both series feature magical aspects, such as spells, wiz- ards, and fantastical creatures.2. Young protagonists: The main characters in both series are young and face challenges and adventures that come with growing up in a magical world.3. Family themes: Both series emphasize the importance of family and the bonds between family members.4. Friendship: The series also highlight the importance of friendship and the power of hav- ing loyal and supportive friends. Differences:.. Genre: While both series can be classified as fantasy, Harry Potter is more traditionally fantasy-oriented, while Percy Jackson is urban fantasy with contemporary setting.2. Setting: Harry Potter is set in a parallel universe where wizardry and magic are prevalent, while Percy Jackson set in present-day America and Greece.3. Magic system: The magic systems in each series are different. In Harry Potter, magic cast using wands and spells, while in Percy Jackson, magic is accessed through the use of magical coins and enchanted items.4. Gods and myths: Harry Potter is based on the idea of witches and wizards living among us, while Percy Jackson is based on Greek mythology and features the gods and goddesses of ancient Greece.5. Tone: Harry Potter is written in a more traditional, epic style, while Percy Jackson is written in a more contemporary, humorous tone. Ultimately, both series offer engaging. imagi- native storytelling that captures the hearts and minds of readers. While they both explore similar themes, their unique settings, magic systems, and tones make them distinct from one another. I hope this helps! Let know if you have any other questions. Score:
GPT4-as-Judge
The response thoroughly addresses the user's 's question by comparing and contrasting the Harry Potter and Percy Jackson series. identifies key similarities, such as magical elements, young protagonists, and themes of family and friendship. The response also effectively highlights differences in genre, setting, magic systems, mythological influences, and tone. The information well-organized, relevant, and engaging, demonstrat- ing comprehensive understanding of both series. The response is tailored specifically to the user's 's inquiry, providing insightful and expert knowledge without any extraneous information.
Evaluation evidence:
22


Reviews:
[{"review_id": "iK6AwiH9o0", "reviewer": "Reviewer_q6ih", "summary": "The paper proposes Decoupled Value Policy Optimization (DVPO) for RLHF: instead of the standard reward-model → online critic → policy pipeline, it pretrains a single Global Value Model (GVM) on the same offline preference data and freezes it as a universal critic to guide policy learning. The problem being solved is the instability and compute overhead of actor–critic RLHF (critic drift, high variance from sequence-level rewards, and the need to run multiple models in training).\n\nKey claims:\n(Theoretical) With fixed feedback (no new rewards), reward-model pretraining + value estimation is informationally equivalent to direct value pretraining; policy gradients differ by at most a bounded term κ that vanishes as approximation errors go to 0 (Lemma 3.1). A corollary states DVPO preserves PPO-style local convergence under KL regularization.\n(Algorithmic) DVPO eliminates online critic training, using a frozen, token-level value signal (GVM) for advantages, thereby avoiding critic drift and reducing variance while retaining on-policy updates.\n(Empirical) On MT-Bench, Arena-Hard, and Alpaca-Eval, DVPO matches or surpasses PPO/GRPO/ReMax/DPO across Llama-3B/8B and Mistral-7B settings (Tables 1–2), and shows improved compute efficiency (≈40–50% lower memory, ≈35% faster training in the base setting; Table 7).", "questions": "seen in the weakness section. \n\nTable 1 (Base setting). Please include #prompts and #rollouts per prompt used during RL for each method (PPO/GRPO/ReMax/DVPO), and whether generation lengths and KL targets were matched across methods (important for fairness).\n\nCompute tables (Table 6 vs Table 7). The narrative states ~40% memory and ~35% time reductions, while Table 7 gives concrete GB/seconds for specific setups. Please reconcile the headline percentages with the numbers in Table 7", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear, impactful simplification: Recasting RLHF as policy-only optimization with a pretrained value is elegant and practically meaningful.\nSolid theory–practice bridge: The equivalence lemma and convergence corollary directly justify the algorithmic design.\nConsistent empirical gains & breadth: Improvements across models/benchmarks in both base and instruction settings; strong compute-efficiency results.\nToken-level credit & interpretability: Concrete examples demonstrate fine-grained attributions absent in reward-only methods.\nReusability/generalization: A policy-agnostic critic simplifies reuse across models and appears robust vs. actor–critic critics on distribution shift.", "weaknesses": "Fixed-feedback assumption (no new rewards). DVPO’s theory and setup hinge on no additional reward during training; if limited online human feedback arrives, the frozen GVM cannot adapt.\nQ: If limited online feedback becomes available, could you support periodic GVM refresh (semi-online DVPO) while retaining the stability guarantee, and what parts of Lemma 3.1 / the corollary would need to change?\n\nScope of evaluations. Benchmarks are mainstream chat-style; tougher code/maths/long-horizon tasks may stress token-value fidelity.\nQ: Any results (or planned) on reasoning-heavy suites to test structure-aware value estimation?\n\nStability evidence presentation. Paper claims reduced critic drift and smoother training but lacks side-by-side variance/curve plots quantifying it.\nQ: Can you add curves (PPO vs DVPO) and a critic-policy disagreement metric (e.g., advantage-distribution KL) to concretely show drift reduction?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9a4483U1wX", "reviewer": "Reviewer_PVQp", "summary": "This paper proposes a novel approach to improve PPO-based RLHF by avoiding explicit reward modeling and instead learning a human preference-consistent critic/value model. The authors transform the traditional two-stage RLHF pipeline into an offline critic training plus online value optimization framework, termed DVPO. Through theoretical analysis, the paper establishes an equivalence relationship between reward models and value models under fixed feedback conditions, providing theoretical foundations for training the Global Value Model (GVM) and deriving convergence guarantees for DVPO. Experiments are conducted on the UltraFeedback dataset using models ranging from 3B to 8B parameters, both with and without instruction tuning. Results demonstrate DVPO's effectiveness across settings on evaluation benchmarks including MT-Bench, AlpacaEval 2.0, and Arena-Hard.", "questions": "1. Have you considered incorporating Bradley-Terry reward modeling in the pre-training of value models? This is the most standard approach to reward modeling, as human feedback typically comes in pairwise comparisons. Simply assigning -1 and +1 rewards does not reflect the actual reward modeling procedure.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. **Clear presentation and theoretical rigor**: The paper is well-structured with clear insights in the introduction, particularly regarding the equivalence between training reward models and critic models based on fixed feedback. The visualization effectively depicts algorithmic differences between PPO and DVPO, and the theoretical analysis is well-organized.\n2. **Novel algorithmic contribution**: The proposed alignment algorithm offers a compelling alternative to the conventional two-stage RLHF pipeline (reward modeling + preference optimization), contributing a simplified RLHF framework that effectively reduces variance in online critic learning and improves training stability.\n3. **Comprehensive empirical evaluation**: The experimental studies demonstrate thoroughness in base model selection, baseline method comparisons, and evaluation benchmark choices.", "weaknesses": "1. **Unclear mechanism of performance gains**: The underlying reasons for DVPO outperforming baselines remain insufficiently explained. The value model trained on pre-collected preference feedback effectively learns $V^{\\pi_B}$ (assuming responses are collected from some behavior policy $\\pi_B$), which contradicts the paper's claim that this value model is a \"global\" one (definitely not $V^*$). Additionally, the paper attributes superiority over sentence-level reward-based methods to GVM providing more fine-grained feedback. However, since human preferences are provided at the sentence level, the results and analysis do not adequately support this claim.\n2. **Non-standard hyperparameter settings**: The experimental setup uses a batch size of 8, whereas current standard practice in the field uses batch sizes of 32 to 128. The KL coefficient is crucial for algorithm performance in both DVPO and baseline methods (including PPO and GRPO), and providing the search range would enhance experimental credibility.\n3. **Insufficient justification for explicit value modeling**: The significance of explicitly fitting a value model needs stronger empirical backing. Given that GRPO and RLOO use Monte Carlo-based methods to estimate values, the authors should provide deeper insights into how separately training a value model enables DVPO to outperform these baselines. Furthermore, the comparison in Table 6 is unfair to GRPO regarding response generation time—Monte Carlo-based methods use an n-shot sampling strategy, making generation time equivalent to DVPO given the same training batch size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEY30f6Dcs", "reviewer": "Reviewer_sBiL", "summary": "This paper presents Decoupled Value Policy Optimization, a framework for reinforcement learning from human feedback that replaces the typical online-learned critic or reward model with a single, offline-pretrained global value model. The central idea is that pretraining a value model on the same dataset used for reward modeling is informationally equivalent to the joint reward-critic learning process, but with advantages in stability and efficiency. DVPO demonstrates strong empirical results on standard RLHF benchmarks (MT-Bench, Arena-Hard, Alpaca-Eval), surpassing or matching state-of-the-art methods, while significantly reducing memory consumption and training time.", "questions": "1. What mechanisms exist to detect or mitigate GVM misgeneralization due to dataset bias or coverage gaps?\n\n2. Can the model identify when its value predictions become unreliable or out-of-distribution?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well-motivated and the equivalence between reward-then-critic pipelines and direct value pretraining is well-argued and convincingly formalized, highlighting under-recognized redundancy in the RLHF paradigm.\n\n2. The framework is particularly interesting to the community as DVPO simplifies RLHF engineering by eliminating online critic training, reducing GPU memory use (40%) and time (35%), thus enabling larger models or faster iteration with fewer resources.\n\n3. Experiments, ablations, and case studies clearly show that DVPO attains or improves upon baseline methods in terms of various metrics.", "weaknesses": "1. The framework assumes the offline preference or reward dataset is broad enough for effective generalization.\n\n2. In scenarios where additional human or environmental feedback can be injected mid-training, DVPO might not benefit from it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)