Group-Normalized Implicit Value Optimization for Language Models
Authors: Yunseon Choi
Junyoung Jang
Chaeyoung Oh
Minchan Jeong
Doo Hwan Hwang
Kee-Eung Kim
Venue: ICLR 2026 Poster (2026)
Keywords: LLM post-training
Primary area: optimization

Abstract: Fine-tuning Large Language Models (LLMs) with reinforcement learning (RL) has become a key technique for enhancing performance on a wide range of tasks, from user alignment to complex reasoning. However, this approach is often hindered by the difficulty of fine-grained credit assignment, as it typically relies on sparse rewards given only at the end of a completely generated sequence. Conventional solutions often require training an auxiliary value network known as critic, which introduces significant computational overhead and training instability. We present Group-Normalized Implicit Value Optimization (GN-IVO), a novel, critic-free algorithm that directly addresses this challenge. GN-IVO learns step-level values implicitly from the policy through a group-normalized distributional matching objective. This approach elegantly circumvents the need for an explicit critic and avoids the computation of the intractable partition function by normalizing values across a group of sampled model responses. Theoretically, we prove that our objective recovers the true value function up to a constant, guaranteeing that the optimal policy is preserved. We demonstrate the practical effectiveness of GN-IVO on a diverse set of text generation and reasoning tasks, showing that it consistently outperforms strong RL baselines for LLMs.

Published as a conference paper at ICLR 2026
GROUP-NORMALIZED IMPLICIT VALUE OPTIMIZATION FOR LANGUAGE MODELS
Yunseon Choi
Junyoung Jang Kee-Eung Kim
Chaeyoung Oh
Minchan Jeong
Doohwan Hwang
KAIST AI
{cys9506, kekimjekaist.acc.
ABSTRACT
Fine-tuning Large Language Models (LLMs) with reinforcement learning (RL) has become key technique for enhancing performance on a wide range of tasks, from user alignment to complex reasoning. However, this approach is often hindered by the difficulty of fine-grained credit assignment, it typically relies sparse rewards given only at the end of a completely generated sequence. Conventional solutions often require training an auxiliary value network known critic, which introduces significant computational overhead and training instability. We present Group-Normalize Implicit Value Optimization (GN-IVO), novel, critic-free algorithm that directly addresses this challenge. GN-IVO learns step-level values implicitly from the policy through a group-normalized distributional matching objective. This approach elegantly circumvents the need for an explicit critic and avoids the computation of the intractable partition function by normalizing values across group of sampled model responses. Theoretically, objective recovers the true value function up to a constant, guaranteeing that the optimal policy is preserved. We demonstrate the practical effectiveness of GN-IVO on a diverse set of text generation and reasoning tasks, showing that it consistently outperforms strong RL baselines for LLMs.
prove that our
INTRODUCTION
Reinforcement Learning (RL) has emerged as a powerful paradigm for post-training Large Language Models (LLMs) (Ouyang et al. 2022 Kumar et al., 2025). Its success is well-establiseed across range of critical applications, from aligning model outputs with human preferences (Bai et al. 2022
Rafailov et al. 20023 to enhancing their complex, multi-step reasoning abilities (Zelikman et al. 2024
DeepSeek-Al 2025 Yet, many RL approaches for LLMs (Rafailov al. 2023; Ethayarajh et al. 2024; AAhmadian et al. 2024) cast policy learning as a contextual bandit, where the entire generated response is treated single action that is assigned scalar score by reward model. This formulation clashes with the inherently sequential, long-horizer nature of both loken-by-toker step-by-step reasoning, in which intermediate decisions compound over time. Moreover, outcome- only rewards yield a sparse and delayed learning signal, offering little guidance about which token or step-level decisions drive the final quality. As result, step-level credit assignment remains opaque, suggesting that language generation should be treated as sequential decision making rather than a one-shot bandit.
generation and
Recent works (Zeng et al., 2024: Wang et al. 2024 2025) therefore model LLM generation as Markov Decision Process (MDP) to incorporate the sequential decision making. common approach is to enforce Bellman consistency and learn step-aware value estimates. In particular, Wang al. 2025): Liu al. (2024) train policies within the soft Q-learning framework (Haarnoja et al. 2017). parameterizing the policy as a Boltzmann distribution over Q-values with partition function. However, because the partition function must still be estimated, these methods retain an auxiliary estimator. While this estimator can serve as a baseline for variance reduction in policy updates, this introduces practical overhead; additional networks to train and tune, increased computational and memory costs, and added complexity in the post-training pipeline. This trade-off, gaining step-aware feedback at the expense of a more complex training procedure, motivates methods that enable effective sequential learning without auxiliary networks substantial engineering burden.

---

Published as a conference paper at ICLR 2026
In this work, we propose Group-Normalized Implicit Value Optimization (GN-IVO), an RL fine- tuning method that incorporates the sequential nature of language generation without requiring any auxiliary network. We first generalize the canonical KL-regularized objective for LLMs to partial sequence, deriving an explicit link between the policy and a soft value function. This value function quantifies the contribution of a partial sequence to the eventual outcome reward. To learn these values, we introduce a group-normalized objective. For given query, we sample a group of complete responses and form target distribution over them, proportional to their exponential rewards. We then match this target with the normalized model's ' predicted values for the partial sequence. This distributional matching forces the model align its value predictions with the ground relative values. We prove that this objective recovers the true value function up to additive constant, which preserves the action preferences needed for the optimal policy. Finally, inspired by recent advances like DPO (Rafailov et al. 2023) that bypass explicit reward models, we likewise bypass explicit value function modeling by the direct policy-value relationship.
The result is lightweight and direct optimization scheme that avoids computing the intractable partition function: normalization within each sampled group cancels any additive constant including the partition function. Thus, our method provides fine-graine fine-grained feedback incurring the overhead of training a separate value estimator. We validate the proposed approach on a diverse set of text generation and reasoning tasks, including the Helpful assistant, TL;DR summa- rization, prompt generation for text-to-image models tasks, and challenging reasoning benchmarks. Across these scenarios, GN-IVO demonstrates its effectiveness in performance compared standard RL fine-tuning methods for LLMs.
partial sequences without
We summarize our main contributions follows:
1. New Tane-Tuning Objective:
We introduce Group-Normalized Implicit Value Optimization, a novel RL objective that incorporates step-level feedback into LLM post-training. Our method dispenses with any auxiliary value/critic model, instead using group-base normalization to infer the relative value of partial sequences.
2. Theoretical Guarantee: We present theoretical analysis showing that our normalized objective learns the true value function up to a constant offset, which does not affect the optimal policy. This result guarantees that optimizing our objective yields the same optimal policy the original RL problem, thereby ensuring policy optimality and consistency.
3. Empirical Validation:
We demonstrate the effectiveness of our approach on multiple benchmarks, including text generation and complex reasoning tasks. Our experiments show that the proposed method achieves higher rewards than standard RL algorithms, highlighting its practical benefit in aligning LLMS with sparse feedback.
RELATED WORKS
Reinforcement Learning from Human Feedback (RLHF) LLMs are often aligned with human preferences using RLHF, wherein a reward model trained on human feedback guides the policy optimization. PPO (Schulman et al. (017) has become the canonical algorithm in carly large-scale alignment works (Ouyang al., 2022). Despite its success, PPO-based RLHF can be resource- intensive and operationally complex. It typically requires keeping multiple models in memory, a policy, frozen reference model for KL control, critic model, and reward model. The performance is also sensitive to hyperparameters due to high variance of the policy gradient estimates and delayed outcome rewards. These challenges have motivated research into simpler fine-tuning methods that bypass the full RL machinery while aiming to preserve alignment quality. DPO (Rafailoy al.) (023) reframes the KL-regularized objective as supervised loss on preference-labeled data, eliminating explicit reward model training, and thereby improving stability and computational efficiency. Numerous extensions build on this paradigm (Maeng et al.) 2025 Qi et al. 2024). Distinct from these preference-basee based methods, DRO (Richemond et al. (024) addresses scenarios where feedback is provided as a score for each completion, rather than as a pairwise preference. To handle this format, DRO incorporates the Soft Bellman equation into bandit-setting objective, effectively learning from direct reward labels. BRAIN (Pandey et al. 2024) formulates RLHF as a distribution- matching problem between the target and learned policies, proposing self-normalize baseline to reduce variance. However, DPO. DRO, BRAIN and most of their variants formulate the generation task as bandit problem where generating completion is considered single action that receives
2

---

Published as a conference paper at ICLR 2026
single reward. This formulation obscures the sequential structure of language generation, thereby limiting performance on long-horizon tasks such as mathematical reasoning (Yaan et al. 2025).
Reinforcement Learning for LLM Reasoning Beyond human preference alignment, recent works have explored RL to improve LLMs' reasoning abilities, which require generating long chains of intermediate steps. Such training imposes a substantial memory demand due to long generated trajectories, even when the base model is modest size. Since the critic network adds to this memory cost, a key research direction has been to develop approaches that remove explicit value function learning policy-gradien methods. For instance, GRPO (Shao et al. 2024) is variant of PPO that dispenses with a learned critic and instead computes advantages from group-wise return statistics within each batch. By using the average return from a group of trajectories as a baseline, GRPO significantly reduces memory overhead while maintaining training stability. In similar vein, RLOO (Ahmadian et al. 2024) extends REINFORCE with a leave-one-out baseline. It computes the advantage of each response by comparing its reward to the average reward of others in the group, thereby reducing variance without requiring a separate value predictor. Despite their improved memory- mhemory-efficiency, methods like GRPO and RLOO still treat the entire generated sequence single action. By doing so, they fail to model the internal structure of multi-step reasoning and assign credit equally across all tokens based on single, final reward. This simplistic treatment overlooks the Sfain-of-thoughh or stepwise dependencies within complex reasoning processes, making it difficult to pinpoint which parts of the generation led successful outcome. Recently, OREO (Wang et al.) 2025) and DQO (Liu et al. 2024) have formulated reasoning steps as an MDP, employing soft Q-learning to learn the policy. However, the Bellman error minimization requires an additional value network, making the training procedure complex and memory
which hinders scalability.
To bridge this gap, we introduce Group-Normalized Implicit Value Optimization (GN-IVO), method designed to be both sequential and critic-free. Similar to GRPO and RLOO, GN-IVO utilizes group of samples to remain critic-free. However, key difference lies in how these samples are used: whereas GRPO computes advantages using batch statistics, GN-IVO uses the samples to form group distributional matching objective. At the same time, and in spirit of OREO and DQO, our method explicitly models the step-by-stee nature of generation to enable fine-grained credit assignment.
3 METHOD
3.1 BACKGROUND: KL-RRGUILARIZED POLICY OPTIMIZATION
KL-regularized policy optimization is a widely used approach for refining language model policies. The goal is train a policy my that maximizes the expected reward while staying close a reference policy Teaaa which is often the supervised fine-tuned (SFT) model that we start with. This objective formalized as:
Ta (1/lz)
max D.yDDy R(x, y) a log
(1)
Ma (y|x))
where x is query sampled from the dataset D, and is completion generated by the policy re- The function R(x, y) assigns a scalar reward to the completion y for a given query x, and a 2 0 is a temperature coefficient. The optimal policy no- for Eq. I has known closed-form solution:
Tg- (y|z) TBaa R(
R(za)//
(2)
Z(x)
where Z(x) = Emenya (vlx)) [axs)// serves as the partition function to ensure the distribution is normalized. While the formulation in Eq. is standard for RL post-training LLMs, it relies on a single reward R(x. y) assigned to the entire generated sequence y.
However, this single reward for the entire sequence provides a weak credit assignment signal. Given the autoregressive, token-by by-tken generation process of language models, assignment scheme is warranted. We therefore reformulate the objective to incorporate expected rewards each timestep, enabling denser supervision.
granular credit-
3

---

Published as a conference paper at ICLR 2026
3.2 LEARNING STEP-LEVEL VALUES VIA GROUP NORMALIZATION
Our first key theoretical result establishes that the relationship in Eq. - extends to arbitrary sequence prefixes. Let y (yo- -T--) be any sequence of length T, where yt denotes the token or reasoning step at timestep 1.
Theorem 3.1. Suppose that Tg. and Taaa are autoregressive policies that generate sequences token- by-token, and that satisfy Eq. for complete response For any t & (1,...T), let yet. == your denote the length-I prefix. Then, Yet. distributions of these policies satisfy:
Tg (ycclz) TTQaa (N<clz))e (a.ve))
(3)
((x)
where a soft value function V(z, which represents the expected future reward from yeu. is defined as:
g)/a log Erogg (vlvce.)) [ [e
1= T
V(2, vee) =
< T.
While distributions over partial sequences are typically hard to analyze, Theorem 3.1 reveals an exact analytical form for тT- (y<r|z) with V (z,))e)). The value function (zz, y<t) provides formal solution to the temporal credit assignment problem by quantifying the expected achievable reward given the tokens generated so far, V<t- allows the model to attribute the final monolithic reward back to each intermediate decision, enabling granular policy optimization.
The challenge of evaluating the partition function While the soft value function from Theorem 3. can be modeled with an auxiliary network Va. doing adds complexity and computational cost. Inspired by DPO (Rafailov et al. 2023) that bypasses an explicit reward model, we instead pursue direct policy objective that avoids explicit value modeling. By rearranging Eq. [
express the value function implicitly in terms of policy ratio, which allows us to reframe value fitting as a policy optimization problem:
Tg- (U<clzz)
V(x, = log (ax) log
(5)
Teaa (V<rlæ)
natural starting point for policy training loss is to minimize the mean-squared error (MSE) between the policy-defined value (5).5) and the ground-truth value function (Eq, 4). Substituting our trainable policy Te for the unknown optimal policy mg- yields the following loss function:
log (xx + log (o((<)( log By--rou C ve..)) (( 2
CMSSEe
(Z.0) (aase(Z.0)
(6)
Toai (y<elz)
practical difficulty in Eq. however, the presence of the partition function (zx). Its computation requires marginalizing over all possible sequences under To,a which is intractable. While this term can be approximated with an auxiliary network, (x), doing so introduces additional complexity and potential instability. This motivates different approach that circumvents the need to estimate the partition function entirely.
The gromp-normalized objective To avoid an auxiliary network for the partition function, propose a group-normalized objective for value estimation. The key insight that for a fixed query the partition function Zxx) is a constant across all candidate responses. By normalizing values over a sampled group of candidates, any shared constant including log ((x) cancels out, making its explicit estimation unnecessary. We first develop this objective for an explicit estimator, Vi before extending it to implicit, policy-only training.
The procedure starts by sampling a group of > 1 i.idd completions, (g0) K-1 from Tas (-1z). For set of prefixes {v (1) K-1 randomly sampled timestep < TP we propose the training
(
We consider types of ye For general text generation, yı is single token, 31 wi E V. For reasoning tasks, yı denotes complete reasoning step represented as a token subsequence y, (wro ... Wh, with 4
We assume all K completions have the same length I' after mild assumption, as sequences are typically padded to fixed maximum length in practice. Both the policy and reward models are designed ignore padded positions, ensuring rewards are unaffected.
with [PAD] tokens. This is
4

---

Published as a conference paper at ICLR 2026
objective given by:
c (olan Σラ= (vo
R(z.y ///0 log
min E anD Lo.DDuuU T). (o:)) (0:K) an (11))
Intuitively, this objective trains Ve to assign higher values to those yet that are likely to extend into high-reward outcomes, Considering the definition of V Var, yee) and the tower property, our objective equivalent to minimizing a weighted group cross-entropy:
K
K Σ ==
e v(zw22))
min E Ben D.n- Ve (0:K) ver iottoia (112))
log
K-1
e (arvi
R-1 L== (o
f=0
Zj=0
This equivalence reveals that the objective Eq. frames the learning problem as one of distributional matching between two distributions defined over the group: (i) a target distribution derived from the normalized, exponential true values V, and (ii) model distribution derived from the normalized, exponential estimated values of our learnable function Vy. From this observation, the following theorem establishes that the optimal solution to Eq. recovers the true value function up to constant offset.
Theorem 3.2 (Consistency up to constant shift), Assume unlimited model capacity and data. For any K >>> and L E (1,... ,T), the minimizer Vg- of Eq. 7 recovers the soft value function V of Eq. up to an additive, ier-endeenndent offset Ce(z):
V (<t) = ((x, Met) - log (t()):
equivalently (2.9e)) ((.ee)))
This additive offset does not affect the resulting optimal policy, as in the following corollary. Corollary 3.3 (Policy invariance to additive shifts). For any positive scalar Cx(x), let V' y<1) Vex + log (z). The optimal policy for V' remains the same as the optimal policy for V.
Taken together, these results guarantee that the optimal policy derived from our learned value function Ve- is identical the one obtained from the true value function V. The proofs are provided in Appendix A
3.3
IMPLICIT VALUE OPTIMIZATION
The key step toward our final, critic-free objective, is to eliminate the explicit value function Ve entirely. Instead of training a separate network, we leverage the policy-value link from Theorem 3.2 by substituting the value term in Eq. 7 with its implicit, policy-defined equivalent.
From Theorem 3.2 we know that the optimal value function Vp- is equivalent to the true value V up to constant (c = V. We can combine this with the policy-value link from Eq. 3 which
states Ty- (yec (z)) xx Teaa (W<<e/r)) (a. Together, these relationships allow us to express the exponential value function directly in terms of the policy:
e Yo- (z, Cx(x)((x) (v<elz)
(8)
To.a (yes (x)
We now substitute this policy-definee expression for the exponential value function back into the model's 's predicted distribution within our objective (Eq. (7). The model's distribution is softmax over the group of K sampled values. By replacing each exp(VV term with its policy-defined equivalent from Eq. 8, the shared multiplicative terms Cr(rr) and ((x) cancel out from the numerator and denominator of the softmax operation:
Te (1)))
no (v (2)
CL
Vv (v Σ, exp(VV
Koaad (v(21z)
E CL())ZZ0)
To (2 (x)
K-1
To To (x) Teass (1/2)zz)
Toaas (12
5

---

Published as a conference paper at ICLR 2026
This leads our final, critic-free objective:
(O) = EE D. y W (:1z)
K /a /a log
To To (y=2 10---1100 log
K
Σ
Te no(y<t 1
(z)
(3) Told (y) 2:)
j=0 Batd (u (5) (x)
1=0
In our practical implementation, we replace e ya with the group-normalized weight
/// Ens ylyy for stability.
The training procedure for our algorithm follows the iterative scheme detailed in Algorithm|] Each training iteration consists of data generation, reward evaluation, and policy optimization. First, sample batch of queries from the dataset 'D. For each query, we generate set of K distinct responses by sampling from Maaa which is frozen copy of the policy from the previous iteration. Next, each of these K full-sequence responses is evaluated by the reward function R(z, y) to obtain a scalar reward. Finally, these queries, responses, and rewards are used to perform a gradient update on the policy parameter by minimizing Eq. 9 After the policy my has been updated, To is updated to match xe for data generation in the next iteration.
Training procedure
Algorithm 1 Group-Normaliin Implicit Value Optimization
Input: Reward function R, learning rate v/, the policy Ta and set Jo f- T for iterations do
Sample query x ~ 'D and generate responses y (0:K) ~ Toaa (112)
Evaluate reward R(2, ((x,)(() (() for all i E {O..... K-1
Update by optimizing the following loss in Eq. 9 0 v/ -VVoLoN-io(()
To.1 + T
end for
EXPERIMENTS
We evaluate our method on a diverse range of reasoning and text generation tasks. These evalua- tions include mathematical problem-solving with reasoning steps well as open-ended language generation. The text generation tasks encompass creating helpful assistant responses, summarizing Reddit posts, and generating prompts for text-to-image models. Furthermore, we conduct a sensitivity analysis on hyperparameters.
4.1 MATHEMATICAL REASONING
This task assesses a model's ability to solve math problems that require multi-step reasoning, including generating intermediate derivations and the final answer.
Experiment setup For RL
mathematical reasoning. we the MATH 2021 and employ Qwen2.5-Math-7B ( (Yang et al. (024a) and Llama-
training
dataset (Hendrycks et al.
3. I.n-tuct (Meta-A) AL. 2024) backbone models. Input prompts include an instruction and few-shot examples to elicit bycp-by-step derivations, where each step is explicitly marked with a label (e.g. stepl, step2). In this setting, we define Yt as a single reasoning step; thus, Ye sequence of tokens that forms complete step, and T is the total number of reasoning steps in the solution. The reward function composed of two components, one for the correctness of the final answer and the other for adherence to the specified format. We evaluate performance on widely used mathematical reasoning benchmarks, including AMC 2023 (of Problem Solving b), Minerva Math (Lewkowyz7 et al. 2022), Olympiad-Bench (He et al. 2024), and AIME 2024/2025 (of Problem Solving a). These benchmarks cover a wide range of difficulties and reasoning depths, from competition-styie problems to graduate-level questions. Further details are provided in the Appendix
Metrics and decoding We report performance using the Pass@ I and Pass@3 metric. For Pass @ 1, generate responses via greedy decoding. For Pass 3, sample three completions for each problem using a temperature of 0.7. For all evaluations, we utilize the math-verify library to perform answer extraction and equivalence checking.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of method against baselines on the math reasoning task. The Pass@ (P@ metric calculated over three trials per query. Bold and underline indicate the best and second-best results, respectively.
AMC2023
Minerva Math
Olympiad Bench
AIME2024
AIME2025
Method
P@1
P@3
P@1
P@9
P61
P@r
P60
P@9
P611
P693
27.5
37,5
25.7
32.7
15.6
24.2
3.3
10.0
0.0
0.0
SFT-winning
27.5
35.0
24.2
35.6
16.0
27.1
6.6
6.6
0.0
Online DPO
22.5
33.1
25.3
30.5
15.1
26.2
3.3
13.3
0.0
0.0
PPO
25.0
35.0
21.7
34,9
15.7
26.2
3,3
16.6
3.3
0.0
DRO
22.5
25.6
3.3
0.0
OREO
27.5
32.5
25.7
35.3
15.7
26.8
3.3
6.6
0.0
6.6
RLOO
35,0
40.0
26.1
34.1
17.9
26.1
6.6
16.6
0.0
0.0
GRPO
35.0
37.5
25.3
35.3
18.8
25.3
6.6
16.6
3.3
Ours
42.5
45.0
26.1
36.0
17.3
27.8
10.0
16.6
3.3
3.3
5wen2.5-Moth-78
52.5
70.0
27.0
36.0
37.7
46.2
23.0
26.6
6.6
13,3
SFT-winning
$7.5
62.5
30.5
36.0
38.8
49.0
23.3
26.6
13.3
13.3
Online DPO
$7.5
70.0
36.2
10.0
13.3
PPO
47.5
67.5
28.3
37,8
38.6
49.0
23.3
30.0
10.0
13.3
DRO
55.0
67.5
31.2
37.1
48.7
10.0
13.3
ORBO
55.0
70.0
31,6
38.6
38.5
49.1
16.6
30.0
10.0
13.3
RLOO
57.5
72.5
30.1
40.4
38.8
48.8
23.3
36.6
13.3
16.6
GRPO
60.0
70.0
29.7
39.2
49.4
26.6
6.6
13.3
Baselines
We compare our proposed method against the following baseline algorithms:
SFT-winning generates responses for each query and performs supervised fine-tuning on the response with the higher reward.
Online DPO (Qi et al., 2024) extends DPO to an online setting. In each iteration, it generates two responses for each query, designates them the winner and the loser based on their rewards, and uses this pair to compute the DPO loss in the policy update.
PPO (Schulman et al. 2017 Ouyang et al. 2022) generates a single completion for each query and performs clipped -ellygggredentt updates using advantages computed by critic network.
DRO (Richemond et al. 2024 is soft Q-learning algorithm adapted to the bandit setting. It generates single completion for each query and trains a value network via the soft Bellman equation. Policy updates are then derived from this value network.
OREO (Wang et al. 2025 is sequential extension of DRO that uses soft Q-learning to train step-level value network. This network then guides the generation process each step.
RLOO (Ahmadian al. 2024) is REINFORCE-Style policy gradient method that employs leave-one-one estimation for advantages. It generates K responses for each query, and the rewards from this group are used compute the leave-one-out advantage.
GRPO (Shao al. 2024) is a PPO-style method that uses the group mean baseline for advantage estimation. It generates K responses for each query, and the mean and standard-deviation of their rewards are used to compute the advantages.
We use the same group size (K) GRPO, RLOO and our method across experiments.
Experimental results The results summarized in Table Overall, algorithm delivers consistent gains over strong RL and preference-learning baselines, and these improvements transfer across both model backbones. The performance differences can be explained in terms of each method's approach. Methods that rely on an explicit value network, such as PPO, DRO, and OREO, showed limited improvement. While OREO models the task sequentially, its performance is likely constrained by the difficulty of accurately estimating value functions for long-range reasoning tasks. Similarly, Online DPO and Rejection Sampling, which use only two samples per query, also struggled. This because in complex reasoning where most responses are incorrect, a simple pairwise preference signal provides an insufficient learning signal. contrast, the methods that showed the most significant improvements, RLOO, GRPO, and our algorithm, all use larger group of samples, thereby providing a meaningful learning signal. Among them, our algorithm consistently shows the best performance most of the tasks. This is because algorithm combines the benefits of group-based signal with the fine mn--grained credit assignment of a sequential model, a feature lacking in other bandit-bass based methods.
7

---

Published as a conference paper at ICLR 2026
Ours
DRO GRPO
Online-Do DPO
RL00
SET SFT-winning
Ours (onc-step)
OREO
3.0
0.40
2.0
Reward
Reward
1.54
0.25
100
200
500
400
500
100
200
300
400
500
100
200
300
400
500
Iteration
Iteration
Iteration
(4) Helpful assistant
(b) TL,DR summarization
(c) Prompt generation
Figure 1: Training curves for our methods and baselines on the solid lines represent the mean reward, while the shaded regions indicate the standard deviation calculated over three random seeds.
model. The
4.2 TEXT GENERATION
We evaluate on three distinct text generation tasks: (1) Helpful assistant, (2) TL;DR summarization, and (3) Prompt generation for text-to-image models. For the Helpful assistant task, the model must generate responses that are both helpful and harmless, with rewards capturing human preferences for such responses. For the TL:DR summarization, the model is required to produce concise summary of given Reddit post. In the Prompt generation, the objective is to create text prompt that guides model generate an image in a specified artistic style.
Experiment setup Our experiments utilize two base models: Owen2.5-11.B-Innstte (Yang et al. 2024a) and -a-ma-322-B--Insttme (Meta-AL, 2024). In these text generation tasks, the number of timesteps T' corresponds the sequence length. For the Helpful assistant task, we train the Anthropic HH-RLHF (Bai et al. 2022) dataset and score responses using publicly available reward models from huggingface that assess helpfulness and harmlessness. For the TL;DR summarization. we use the TL;DR dataset (Stiennon et al. 2020) and evaluate the outputs using a publicly released model trained to assess summary quality. Finally, for the Prompt generation task, the model given contextual instructions, such as an animal activity and a designated artistic style. The reward is calculated using the CLIP text-image similarity score between the generated prompt and the style reference images. Further details on dataset construction and the specific reward models used are provided in the Appendix C
Metrics and decoding We report performance using two average reward metries, Avg@) I and Avg @3. The Avg@1 1 score is calculated using greedy decoding. To compute Avg@3, we sample three completions for each query with a temperature of 0.7 and report the average of their rewards.
Experimental results We evaluate primary sequential algorithm, where the number of steps is the sequence length, alongside a one-step variant that treats the full generated sequence as a single step. Since OREO is sequential extension of the DRO framework, we implement with token-level critic akin to that used in PPO. Training curves and final results are presented in Figure ] and Table 2] respectively. Generally, algorithms that use K samples per query outperform those using single sample. While methods using multiple K samples are typically superior. even our one-step variant outperforms policy-grade gradient baselines like GRPO and RLOO. This result indicates that our group-normalized objective provides more effective learning signal than standard -plicy-gradient estimation.
Our sequential algorithm generally outperforms its one-step variant across all tasks, except for summarization with --ama-3..-3B-Insttriet where performance comparable due to the required short outputs. In contrast, we observe opposite trend among DRO and OREO, where DRO typically outperforms OREO. We attribute this discrepancy to the challenge of training an accurate per- per-timesten timestep value network for OREO, does not rely on an explicit auxiliary network, it successfully leverages the benefits fine- grained credit assignment without being exposed to this pitfall.
inaccurate critic can degrade policy performance. Because method
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of our methods against baselines on three text generation tasks. For each method, we run three random seeds and report the best scores on the test set across seeds. For Arg@3, we report the mean with std. over three repeated trials per query in the same model. GM denotes the geometric mean of Arg @ I across the three tasks. Bold and underline indicate the best and the second-best accuracy for each task. respectively.
Helpful assistant
TL:DR summarization
Prompe generation
Method
GM
Avg@1
Avg
Avg@1
Av
Avg 1
Avg
Qwen2.5 1-55 bustact
SFT-winning
0.375
0.375 (0.001)
-0.710
-0.729 (0.008)
0.342
0.329 (0.004)
-0.450
Online DPO
1.271
1.260 (0.005)
0.998
(0.066)
0.381
0.381 (0.000)
0.786
PPO
0.875
0..4600.0033
1.393
(0.05().015)
0.274
(0.005)
0.694
DRO
1.174
1.146
1,831
1.725 (0.015)
0.369
0.363
0,926
OREO
0.721
0.733 (0.002)
1.633
1.362 (0.033)
0.261
0.259 (0.00))
0.675
RLOO
1.120
1.043 (0.001)
2.466
2.423 (0.003)
0.319
0.311 (0.005)
1.004
GRPO
1.594
1.506 (0.014)
1.151
1.086 (0.021)
0.367
0.367 (0.000)
0.876
Ours (one-step)
1.446
1,389 (0.005)
2. 148
2.108 (0.024)
0.379
(0.00)
1,056
Ours
1,650
1,628 (0.009)
2.418
2.359 (0.024)
0.383
0.382 (0.001)
1.152
Llamo-3.2-8 3.2.33 Instreer
SFT-winning
0.819
0.819 (0.010)
016113
0.897 (0.011)
0.354
0.354 (0.00)
0.640
Online DPO
1.064
0.974 (0.008)
2.907
2.884 (0.005)
0.372
0.372 (0.000)
1.066
PPO
0.867
0.854 (0.013)
2.8017
2.748 (1.016)
0.287
0.285 (0.002)
0.887
DRO
1.317
1.295 (0.002)
3.104
3.099 (0.008)
0.350
0.345 (0. (0.002)
1.082
OREO
0.864 (0.008)
(0.018)
(0.005)
RLOO
1.528
1,262 (0.004)
3.181
(0.08)
0.349
0,345
1.193
GRPO
2.013
1.996 (0.013)
2.337
(0.06()..01)
0.358
0.346
1.154
(one-step)
2.562
2.555 (0.002)
3.398
3.334 (0.007)
0.371
0.367 (0.001)
1.478
Ours
3,370
3.281 (0.005)
3.347
3,330(0.01)
0.384
(0.00))
1.630
4.3 ANALYSIS ON HYPERPARAMETERS
The group size K We analyze the impact of the group size K, a key hyperparameter for both our method and GRPO, using the Qwen2.5-1.5B, Instruct model evaluated on the An- thropic HH-RLHF dataset. As illus-
20
14-
Reward
Reward
trated in Figure (a). the results show a clear trend: performance improves as K increases in both approaches. This behavior is expected, as our ob- jective relies the set of K samples to form an empirical target distribu- tion. With a small K, this distribution is a noisy approximation of the true value landscape over all possible responses. As K grows, however, the target distribution more accurately reflects the relative ranking responses within the group. This provides more precise learning signal, allowing the model to learn a better value approximation. Notably, the performance gap between our method and GRPO is most pronounced when the group size is small, particularly at K = 2. This demonstrates that our algorithm is effective even in the small group size.
1.0
1.0
0.8
100
200
300
300
0
100
200
300
400
500
Teeration
(4) Group Size
(b) Temperature Coefficient
Figure 2: Analysis on hyperparameters
The temperature coefficient a We also analyze the impact of the temperature coefficient a. which controls the scale of rewards. The target distribution is formed with these scaled rewards, softmax [R/a), where a smaller yields sharper distribution. Conversely, larger makes the distribution softer and more uniform, reducing the distinction between responses with high and low rewards. We conduct experiments with {0.1, 0.2, 0.5, 1.0], and the results are illustrated in Figure 2 (b). We observe that the model reaches higher rewards with lower temperature values (a 0.1,
In contrast, for higher values such as a = 0.5 and 1.0, the models ultimately yields lower rewards. This is because more uniform target distribution provides weaker and less discriminative signal over the group. As a result, the policy assigns comparable probability mass to both superior and mediocre responses, rather than differentiating them.
9

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We presented GN-IVO, novel, critic-free RL algorithm for language models. Its core contribution is group-normalized, distributional matching objective that learns step-level values implicitly from the policy itself, thereby bypassing the need for an explicit critic and its associated complexities. GN-IVO is theoretically grounded and demonstrates strong empirical performance diverse set of tasks. A promising future direction is to apply our framework to other generative models beyond language, such for the fine-tuning of diffusion models.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were used to improve the clarity and readability of the text. Furthermore, they were utilized in the software development process for debugging code and identifying programming errors. All content generated by LLMs was critically reviewed and edited by the authors.
ACKNOWLEDGMENTS
This work was supported by Institute for Information & Communications Technology Promotion (ITPP), funded by MSIT, through the Information Technology Research Center (ITRC) program and other projects RRS-2024-00343989, Enhancing the Ethics of Data Characteristics and Genera- tion AI Models for Social and Ethical Learning: No.RS-205 RRS-2024-00457882 AI Research Hub Project; No. LLRRS-2020-11200944 Foundations of Safe Reinforcement Learning and Its Applications to Natu- Artificial Intelligence Graduate School Program
ral Language Processing:
(KAIST);
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-styl optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Fatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovit, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, 2022.
Yunscon Choi, Sangmin Bac, Sconghyun Ban, Minchan Jeong, Chuheng Zhang, Lei Song. Li Zhao, Jiang Bian, and Kee-Eung Kim. Hard prompts made interpretable: Sparse entropy regularization for prompt tuning with RL. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, 2024.
DeepSeck-Al. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement learning.
CORR, 2025.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, 2024.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting AGI with olympiad- -leee bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), ACI., 2024.
10

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks, NeurlPS Datasets and Benchmarks Track, 2021.
Zijing Hu, Fengda Zhang, Long Chen, Kun Kuang, Jiahui Li, Kaifeng Gao, Jun Xiao, Xin Wang, and Wenwu Zhu. Towards better alignment: Training diffusion models with reinforcement learning against sparse rewards. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR. Computer Vision Foundation/ / IEEE, 2025.
Komal Kumar, Tajamul Ashraf. Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. Torr, Salman Khan, and Fahad Shahbaz Khan. LLM post-training: A deep dive into reasoning large language models. CoRR, 2025.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, NeurlPS, 2022.
Guanlin Liu, Kaixuan Ji, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing multi-step reasoning abilities of language models through direct Q-function optimization. CoRR, 2024.
Joo Bon Maeng, Seongmin Lee, Scokin Seo, and Kee-Eung Kim. Goal-conditionel DPO: Priori- tizing safety in misaligned instructions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Pupers), NAACL, 2025.
Meta-Al. The Ilama 3 herd of models. CoRR, 2024.
Art of Problem Solving. Aime problems and solutions, URL httpsss//arrtofproolemssvvvv htmm://ikkiinneeepp//AMee php/AIME Problems and Solutions
com/wikiiindexx
Art of Problem Solving. Ame problems and solutions, b. URL http::///aartffrrollmmmsl/iig/g http://kkiinnne..ppppttleee.
Problems and Solutions
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems, NeurlPS, 2022.
Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, and Ramón Fernandez Astudillo. BRAIn: Bayesian reward- conditioned amortized inference for natural language generation from feedback. In Forty-first International Conference on Machine Learning, 2024.
Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, and Bowen Zhou. Online dpo: Online direct preference optimization with fast-slow chasing. CORR, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, NeurIPS, 2023.
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi
Azar, Rafael Rafailov, Bernardo Ávila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Rémi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. CoRR, 2024.
11

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. CoRR, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, NeurIPS, 2020.
Chaojie Wang, Yanchen Deng, Zhiyi Lv, Zeng Liang, Jujie He, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for Ilms with deliberative planning. CoRR, 2024.
Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Of- fline reinforcement learning for LLM multi-step reasoning, In Findings of the Association for Computational Linguisties: ACL, 2025.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. In Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, NeurlPS, 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. CoRR, 2024a.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In Forty-first International Conference Machine Learning, ICML, 2024b.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Boji Shan, Zeyuan Liu, Jía Deng, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing LLM reasoning generalists with preference trees. The Thirteenth International Conference on Learning Representations, ICLR, 2025.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-STar: Language models can teach themselves to think before speaking. In First Conference Language Modeling, COLM, 2024.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Forty-first International Conference on Machine Learning, ICML., 2024.
12

---

Published as a conference paper at ICLR 2026
A PROOFS
Proof of Thheormm 3.1 We consider the process I + y = (vo. (yo..... -r--1) with reward R(x, v). where the parameterized policy <o(y(æ) is autoregressive model from the mo (velyee,, x)) (t ,.), We claim that the optimal parameterized policy Te- (y(ar) of this process (Eq. 2 deter- mines the autoregressive policy Ty- (ye Merr x) follows.
To- (y<(l=)) Σ no- 2)) (ylzx)
Σ ne- (Verlyys TTe (yla)) e R(zzz)/a
(Eq. 2
((x)
R((a.9)/ax
Σ Toli (U<r b.ccrly,.oo Teaa (1/(²)
3 = 8(y<r, vl: t))
((x)
R((x,))aa
Σ
Tomia (o ((x)
Toliaa (J xx)
To (Her)z))
Taaa (Bly ( (lx
C Vax.ue))
Z(xx)
Z(x)
=e Visinee)
Therefore, as Theorem 3.1 suggests, the optimal autoregressive policy To- (V<elæ) is reweighted policy by an exponential soft value function Vzz, yer).
Proof of Theorem 3.2 The proof begins with the objective function defined in Eq.
K-1
e Vi(c
min E a U.D.Lut(
Σ
R log
ΣK-1 (oo
(a:K) wet No (-1w)
By applying the law of total expectation and decomposing the joint probability Teaa (yen, y/x) into Toaa (yeel ( (yldke, z), we can rewrite the expectation as:
Volan
R(z.
min B ..... Th, Vel 10:K) mo (11z)
E
Monia (v( (1(
log
K evicavl
By definition, the soft value function satisfies e (a.. ing this into the objective yields:
IE
(e
Substitut-
neau (g (1
K-1 Va log
(o
min BENDINUHL (0:K)
K-
M21 moo (1)x))
E
To reveal the underlying structure, we can rewrite the expression as weighted cross-entropy, which equivalent to the objective in Eq. 3.2
V(
e Volc.
min B.A.D. Ve (0 A (-1z)
e
Σ
log
-K-1
(aav
K V/ (x.
i=0
For clarity, let us define two discrete probability distributions over the indices 0.....
e V(r
e (vola
pc V2t (0:K))=
and 40 (v.c (0:K))==
Ver
kK ((xw2))
EK-1 (o(oe
13

---

Published as a conference paper at ICLR 2026
Letting A(z, Vet (ockk))= Ek- K-1 eu(sn
2
we can express the objective as:
K-1
A(a, (lenseek) (0k))
Σ (2 (ltk)) log qu,(1 vet ((())
min Ta ( (ar)
The term inside the expectation is a positive scaling factor A(.) multiplied by the negative cross- entropy between distributions p and q. By Gibbs' inequality, this term is minimized when the two distributions are identical:
K-1 Σ
K-1
A(-)
Pc(.) log (.)(())) -A()
Σ
p.()) logp.()).
(0:K) = (z, y2t
(u:K)) for all i O.....K Assuming
The minimum is achieved when ( yzt sufficient model capacity and data, the optimal parameters 3 will satisfy this equality:
e ev (2,
e (z,w2))
Σ KK-1 c. (a.0))
ΣK-1 ((aw)))
1-11 K-1 from the support of To (-|z). This implies that for any pair of for any set of samples {ver) 11=0
indices i, h E (0. .O...., - 1), the ratio of the exponentiated value functions is constant:
e Vo- (n.
kK11 (o-
ΣK-1 (a.v2))
1-1 We now show that this constant is The term C2 is constant for a given set of samples {uzr Jj= independent of the particular choice of samples and can be written as C((x). Consider any two partial
sequences V'zr and v in the support of Te (-1z)). We can construct a of samples (1) K-1 (1) that includes both, for instance by setting Vzz D'ee and yer = v22.- Applying the result above with i == 0 and & = gives:
===0
(oslee)) (x.W2))
e Vinale)) V(rak)))
Since Ver and vor are arbitrary sequences from the support, this ratio must be constant for any V<e E supp(To.ma (-1z)).
Therefore, there exists a function Cz)z), which depends on and x but not yeu, such that for all y<r E supp(TTaa (+Iz)):
(2.ve)) Ce(x)) (x.Ne))
Proof of Corollary 3.3
We consider the policy n' induced by the shifted value function V following We substitute the condition e v'(x,0)))) Cx(xr) V (a.Ner) into the policy definition:
the form in Eq.
Teai (Ner (er)ee" (n,Ne))
Ce(x))oeas (yeel (re (x,00))
(z) (a.ver) wg- (vclla))
Ener Tess (yer (r V'
Ci(x) Ener TBas (Wec
Since the term Cz(z) does not depend on the integration variable yee. it can be factored out of the integral in the denominator and cancels. Thus, the induced policy is identical to the original optimal policy.
B COMPARISON WITH DRO, OREO
Comparison with DRO RRichemond et al. 2024 DRO treats the language generation bandit task, where the entire generation y is considered one-step action. It minimizes the Soft Bellman error, which is the same with Eq. 6 In this setting, the expectation over future steps in the soft value
14

---

Published as a conference paper at ICLR 2026
function collapses to the direct reward R(x, v)/a. The MSE loss from Eq. therefore simplifies to the following loss:
(V(Vee R( v2,,))/a))
LDRO 0) Iog (x(x)) + log
(10)
Toaid (Nct (x)
The gradient of LDRO w.r.t. 8 takes form:
To (u((x)
VoLDRO (R((z, y) log (x)) V log Tog()(y) Vo log
(11)
TToaa (z)x))
In this objective, log (x) acts baseline reduce variance, and the second is regulariza- tion penalty. Thus, DRO's training procedure involves optimizing two distinct objectives: Eq. 10 for the value function and Eq. for the policy.
Comparison with OREO Wang et al., 2025 OREO extends DRO sequential decision making
setting, which allows step-level credit assignment. It trains its step-wise value function, V Uet)) by minimizing Soft Bellman error, which takes the form of the MSE loss:
(V, 0) (Vo(2,, Vet)) + log
Te Tellvee x)
R(x, ((x
(12)
Tiat (1) W))
Similar to DRO, the gradient of LOREO w.r.t. 0 takes the form:
VeCoReo = (R(x, y) ver)) Ve log log a Va log
To (ylleee
TTe (y/ W/ 2:)
(13)
In practice, the policy gradient is calculated only with respect to the action taken at the current step To(yelV<e..7) and detacch) is applied to the future-step decisions, Σ log T |V<s. x).
In contrast, our approach does not update the policy based on a single action each timestep t. Instead, it learns by considering the entire prefix Vet. which allows the model to implicitly learn the value function V (ax, y<c) and directly assess the quality of partial sequences.
Online vs. Offline DRO and OREO are off- poficy algorithms that primarily target offline settings (learning from a fixed dataset). Our algorithm is also off- off-policy, as the behavior policy (me can differ from the target policy (Ta)) However, unlike those methods, it particularly well-suited for an online setting, suitable offline datasets with the required K samples per query rare.
C TASKS DETAILS
C. MATHEMATICAL REASONING
To ensure all models generate structured, step-by-step solutions for the mathematical reasoning task, we utilize instruction template. This template, shown in Figure guides the model to break down its reasoning, show intermediate steps, verify its solution, and provide final answer in specific format. We use two types of rewards: one based on the correctness of the final answer, and another based proper formatting, specifically checking the presence of Xboxed{) in the response.
C.2 TEXT GENERATION
Helpful assistant We use the Anthropic HH-RLHF dataset (Bai al. 2022 which contains human preference data on helpful and harmless AI assistants. For the reward models, we adopt open source reward models: one for helpfulness] and another for harmlessness following the setting in (Yang et al. 2024b). The final reward is simple average of the two model scores, with each component equally ly weighted 0.5.
Raay2233/g22-aggghhhhhhuuuuuuo model hagp:/3////---aggg-aaeeessssssee model
15

---

Published as a conference paper at ICLR 2026
Solve the following math problem step by step.
Write each reasoning step clearly, starting with labels stepl, step2, step3,
Show intermediate formulas and simplifications.
Before giving the final answer, add verification step:
You may use Python code, an alternative formula, or quick logical check to confirm correctness.
IIf you detect mistake, correct it.
Conclude with the final answer inside LaTeX
Format example:
stepl: Restate the problem.
step2: Apply the relevant formulas.
step3: Compute intermediate results.
results. step4: Interpret
step5: Verify the solution (with Python code, alternative calculation, or logical consistency check).
step6: State the final boxed answer.
Few shot examples
Problem: [Problem)
Assistant:
Figure Instruction template used in mathematical reasoning task.
TL;DR summarization For this task, we use the Reddit TL;DR summarization (Stiennon et al. 2020) dataset. Following the setup (Yang et al. 2024b), we construct reward function using two open source models: one that rewards conciseness and another that rewards faithfulness to the source text { The final reward calculated as simple average of the two model scores, with each component equally weighted at 0.5.
Prompt generation for text-to-image models Inspired by prior works on controlling text-to- images models through prompt generation (Wen et al. [203 Choi et al. [024], we designed dataset for this task. The dataset is constructed by combining 856 animal activity scene descriptions from (Hu) et al. (0225) with 4 distinct artistic styles (Surrealism, East Asian Classics, and Impressionist), It results in a total of 2559 (853 x 3) scene-style pairs. The dataset is split evenly into training and test sets, with 1.712 pairs in each. Each artistic style is represented by three reference images, as illustrated in F The examples of animal activity scene descriptions are shown in Table Given scene description and target style, the model's task is generate a text prompt that can guide text-to-image models to produce an image in the required style while preserving the original scene's context. The reward function consists of two components: style score and context preservation score. The style score is calculated using frozen CLIP mode [ We compute the CLIP similarity between the generated prompt (via the text encoder) and each of the three reference images (via the image encoder), and then average the results. The context preservation score is the cosine similarity between the generated prompt and the original scene description. The final reward a weighted sum of these two components with weights of 0.7 for the style and 0.3 for the context preservation score. All queries are formatted using the instruction template shown in Figure
D IMPLEMENTATION DETAILS
All methods are implemented using the trl library. For response generation during training, we apply a sampling temperature of 0.7, top-p of 1.0, and top-k of 0.0. We also incorporate KL-penalty regularization term between the current policy and the initial pretrained model, controlled by coefficient B.
The mathematical reasoning experiments were conducted on NVIDIA H200 GPUs with a training query batch size of per GPU. We trained all methods for 4 epochs. During training, the maximum
"Tristan/gpt2 reward summarization "CogCComp/aat--aatfful hat.a///lnnfaiiituuummamaar-eeee -ummary-dettectrr lopmaiddip-vii-lageeaaahhhh
16

---

Published as a conference paper at ICLR 2026
Table 3: Example of animal activity scene description for prompt generation task.
Example of animal activity scene description
a cat washing dishes
a dog washing dishes
duck washing dishes
monkey riding bike
horse riding bike
pig riding bike
a spider playing chess
a rabbit playing chess
a dolphin playing chess
deer reading book
zebra reading book
reading book
a lion washing dishes
a cow washing dishes
turtle washing dishes
a raccoon cooking dinner
bird cooking dinner
llama cooking dinner
a lizard drawing picture
kangaroo drawing picture
shark drawing picture
a butterfly playing the piano a whale writing letter a mouse jogging in the park
gorilla playing the piano
lizard playing the piano
bee writing letter
fox writing letter
chicken jogging the park
bear jogging in the park
Surroalism
Impressionist
Fost Classic
Figure 4: Reference images of 3 distinct artistic style for prompt generation task.
You are a prompt engineer for Stable Diffusion/SDXXL Given Scene/Style, output one descriptive single-line line prompt.
(Animal Activity Scene Deseription) Scene: {Style}
Scene:
Prompt:
Figure 5: Instruction template used in prompt generation for text-to-image models.
new tokens set to 1024, this was increased to 2048 for evaluation to accommodate the longer responses required by difficult benchmarks.
The text generation experiments were conducted on NVIDIA H100 GPUs using a per-GPU query batch size of and were trained for 500 iterations. The maximum generation length was set to 256 new tokens.
Model architecture All methods share the same policy network architecture, using either Qwen or Llama model as the backbone. For critic-based algorithms, a separate value network was infeasible due to GPU memory limitations. We therefore utilized ValueHead from the trl library, which attaches head to the policy network. For the reasoning task, we fine-tune both the Qwen2.5-Me 1-12255--hhh-7 7B and Llama-3.1 1-8B-IInstuu models using LoRA adapter.
17

---

Published as a conference paper at ICLR 2026
SFT-winning For each query in batch, generate two distinct responses (K 2). Each response then evaluated using task-specific reward function, which yields scalar score. The response with the higher score is selected the winner. In the case of tie, the firt-ggenertte response is chosen. Finally, the model's policy is updated by maximizing the log-likelihood of generating these winning responses.
Online DPO For each query, we generate two completions (K = 2). These are then evaluated with a scalar reward function form a preference pair, designating the response with the higher score as the winner and the other as the loser. In the event of tie, the first-generated response is selected as the winner. Finally, the model's policy updated by applying the DPO loss to this constructed pair.
DRO and OREO While DRO and OREO are primarily designed for offline settings, adapt them for online learning. We implement both DRO and OREO using the PPO trainer from the trl library. The critic is implemented network, we use the corresponding loss functions defined in Eq. f0 DRO and Eq. T2 for OREO. The policy updated using the standard clipped surrogate objective from PPO, with advantages estimated by the learned value network. While we also experimented with implementing the policy losses directly from Eq. [1 and Eq. yielded poor performance.
value head attached the policy model. For training the value
13 we found that this approach led unstable training and
PPO, GRPO and RLOO We implement the PPO, GRPO, and RLOO based on their respective trainers from the trl library.
GN-IVO (ours) Our method, GN-IVO, is implemented using the base trainer from the trl library and we use the same group (K) as the GRPO and RLOO baselines for fair comparison. As mentioned in Section 3.3 we use the normalized exponential reward rather then the exponential reward from Eq. 9 The impact of this normalization analyzed in Figure 7
Reward
Norm, am0.1
Narm, a=0.2
Norm, a-0.1
1.0
wio Norm, a=0.2
100
200
300
400
500
Iteration
Figure 6: Investigating the impact of normalization rewards from the Qwen2.5-1.5p 1-55B-InIsstt
model. This comparison conducted values of 0.1 and 0.2.
The detail of hyperparameters are provided in Table
E QUALITATIVE RESULTS
18

---

Published as a conference paper at ICLR 2026
2.0
20
Reward
1.5
1.0
100
200
300
400
500
Iteration
Figure 7: Investigating the impact of the number of sampled from the Qwen2.5-1.5B -nnstttrssS- model. This comparison conducted values across {1, 20, 7).
Table Hyperparameters used for algorithms across tasks.
Method
prepeparrmeterr
Reasoning (Llamaã.1)
Reasening (Qwen2.5)
Helpšul aasistant
TE.DR summarization
Prompt generation
learning gradient
0.00001
0.000001
0.00801
0,00001
0.00001
SPTnnigning
0.1
0.5
learning ?) gradient clip nurm
0,000001
0.00001
Online DPO
temperature
0.2
0.1
0.2
0.5
0.2
KL coetl with init model 5
0.05
0.01
0.1
0.1
0.1
learning
0.00001
0.000001
0,00011
0. 0.00001
0 000000
PPO
gradient
clipping
0.2
0.1
0.5
0.2
parameter
init-model (
0.1
learning rate iy gradient clip nona
0.000001
0.000001
0.00001
0.00001
0.00001
DRO
clipping magge
0.1
0.2
0.2
0.1
0.2
0.5
0.2
0.5
0.5
0.2
temperature
0.2
0.2
0.2
0.2
inir-model
learning ?) gradiest clip
0.00000
0.000001
0.00011
0.00001
ORBO
clipping nange
0.1
0.2
0.5
0.2
temperature
0.2
0.2
0.2
0.2
XLL coetf with init model 8
0.05
0.01
0.1
0.1
0.1
learning
0.00001
0.000001
0000005
0.000005
0.000005
RLOO
gradient
0.1
0.5
group
model
learning mile ?I gradicat clip norm
0.000001
0.000001
0.00001
0.00001
000000
GRPO
clipping range
0.2
0.1
0.2
0.1
0.2
0.5
0.2
0.2
group size &
4
4
coeff init model
0.01
0.1
learning in
0.000001
0.000001
0.00001
0.00001
0.00001
gradient
group temperanure
0.2
0.2
0.2
KL. cueff with init-moel 8
0.05
0.01
0.1
0.1
0.1
19

---

Published as a conference paper at ICLR 2026
E. MATHEMATICAL REASONING
Table 5: Example qualitative results generated by the Qwen2.5-Math-7B model on the AIME 2024.
Every morning Ayu goes for a 9-kilometer-long walk and stops a coffee shop afterwarss. When she walks at constant speed of kilometers per hour, the walk takes her 4 hours, including minutes spent in the coffee shop. When she walks s + kilometers per hour. the walk takes her hours and 24 minutes, including minutes spent in the coffee shop. Suppose Aya walks at 8 + 2 the minutes spent in the coffee shop.
Question:
1 kilometers per hour. Find the number of minutes the walk takes her, including
Method
Response
Ours
stepl: Restate the problem.
Aya walks 9 kilometers two different speeds, and 8- and stops for / minutes in a coffee shop. We need to find the time it takes her to walk 9 kilometers at speed of + kilometers per hour, including the minutes spent in the coffee shop.
step2: Set up the equations.
When Aya walks at speed km/h, the total time (including the coffee shop stop) is hours. So the walking time
60 hours. The distance is km, so we have:
9 - 9-
60
When Aya walks at speed s + 2 km/h, the total time (including the coffee shop
stop) is 2 hours and 24 minutes, which is 2 24 2.4 hours. So the walking time is 2.4 60 hours. The distance 9 km, so we have:
t
9 = (s +
2.4
60
step3: Solve the system of equations. We can solve these equations to find s and t. First, let's express in terms from the first equation:
st
4
4s
540
240s
st
60
240s
540
Next, substitute into the second equation:
6
(8 2)
2.4
60
240s 540
(8 + 2)
2.4
60s
2.4 4s
9 = (s + 2)
2.48
48 + 9
9 (s 2)
s
1.6s
= (8 + 2)
8
Continued on next page
20

---

Published as a conference paper at ICLR 2026
Method
Response
(s + 2) 1.6s)
9
9s = (s + 2) 1.6s)
98 = 98 1.6s + 18 3.28
- + 18 3.2s
1.6s 3.2s 18 =0
x² 2s 11.25 =0
We can solve this quadratic equation using the quadratic formula
-b -b duc where
and -11.55
2a
-2 + 45 -2 2
-2
The positive solution s = = 2.5. Now can find t:
240s
540
240 2.5
540
600 - 540
60
24
2.5
2.5
2.5
step4: Find the time for the speed +
2-
The speed is + 1 = 2.5 + 0.5 = km/h. The walking time is:
walking time
3 hours
Including the coffee shop stop, the total time is:
24 3 0.4 3.4 hours
3
Converting 0.4 hours minutes, we get 0.4 60 24 minutes. So the total time is hours 24 minutes.
Therefore, the number of minutes the walk takes her, including the minutes spent in the coffee shop, is:
x 60 + 24 = 180 + 24 = 204 minutes
step5: State the final boxed answer.
204
21

---

Published as a conference paper at ICLR 2026
Table 6: Example qualitative results generated by the Qwen2.5-Math-78 model on the AIME 2025.
An isosceles trapezoid has an inscribed circle langent to each of its four sides. The radius of the circle is 3, and the area of the trapezoid is 72. Let the paraltel sides of the trapezoid have lengths and s, with r + s. Find r² + +
Question:
Method
Response
Ours
stepl: Restate the problem.
We have an isosceles trapezoid with an inscribed circle of radius 3. The area of the trapezoid is 72. We need to find r² + s² where r and are the lengths of the parallel sides of the trapezoid.
step2: Use the properties of the trapezoid and the inseribed circle.
For a trapezoid to have an inscribed circle, the sum of the lengths of the parallel sides must equal the sum of the lengths of the non-parallel sides. Let the lengths of the parallel sides be r and with > and let the lengths of the non- paralallll
sides be a. Then we have:
+ s = 2a
The height of the trapezoid is equal to the diameter of the inscribed circle, which is 3 6.
The area of the trapezoid is given by:
1
1
Area
(r + height
(r + 8) x 3( + 3(r+
2
2
We know the area 72, so:
3(r + s) = 72
r - 8 = 24
From the earlier equation + = 20, we get:
2a 24
12
step3: Use the Pythagorean theorem to find the lengths of the legs of the right triangles formed by dropping perpendiculars from the endpoints of the shorter base to the longer base.
The trapezoid can be divided into a rectangle and two right triangles. The legs of the right triangles the height of the trapezoid (6) and half the difference of the bases 2 The hypotenuse of each right triangle a = 12. So we have:
2
122 = 62 +
8
2
2
8
144 = 36 +
2
2
r
108
2
V108 6v3
2
r- 12V3
Continued on next page
22

---

Published as a conference paper at ICLR 2026
Method
Response
step4: Solve the system of equations r + 8 = and r 8 12v3.
Adding the two equations:
-) 24
24 12v3
r= 12+
Subtracting the two equations:
(r (r- = 3
2s 24=14-12V3 12v3
== 12 6v3
step5: Calculate 72 +
2 =
= 144 + 144 + = 252 144v3
8² (12 6v3)2 144 144v3 + 108 252 144v3
72 + 5² (252 + 144v3) (252 144v3) = 504
Therefore, the final answer is:
504
23

---

Published as a conference paper at ICLR 2026
Table 7: Example qualitative results generated by the Qwen2.5-Math-75 model on the AMC 2023.
Question:
How many complex numbers satisfy the equation 88 2, where 7 is the conjugate of the complex number z?
Method
Response
Ours
stepl: Restate the problem.
We need to find the number of complex numbers that satisfy the equation 2 = Σ, where is the complex conjugate of 2.
step2: Express in polar form.
Let z = re where is the magnitude and is the argument of z. The conjugate of is re-
The equation 2 = 2 becomes:
(re:0)s 8 r6 r1-
c re-e- -0
This equation can be separated into magnitude and argument parts: For the magnitudes: r =
2. For the arguments: 50 = - + 2km for some integer k
step3: Solve the magnitude equation.
The equation م
be rewritten
r(r²
So, r or = 1. Since is a non-negative real number, the solutions are 0 orr==1
step4: Solve the argument equation for r = 1.
For 1, the argument equation becomes:
50
0 2km
60
2k
0 kn
Since 0 is angle, it is periodic with period 2n, only need to consider = 2, 4, This gives us distinct values for 0.
step5: Solve the argument equation for 0.
For r== O, the only solution is 2 = 0. step6: Combine the solutions.
We have solution for = and 6 solutions for r = 1, so the total number solutions is 6
step7: State the final boxed answer.
24

---

Published as a conference paper at ICLR 2026
Table 8: Example qualitative results generated by the Qwen2.5-Math, model on the Minerva Math.
Question: Preamble: Consider the mass m sliding horizontally under the influence of the applied force I and a friction force which can be approximated by a linear friction element with coefficient Formulate the state-determined equation of motion for the velocity as output and the force as input.
Method
Response
Ours
stepl: Restate the problem.
We need to derive the state-determined equation of motion for mass m slid- ing horizontally under the influence of an applied force and friction force proportional the velocity e with coefficient step2: Identify the forces acting on the mass.
The forces acting on the mass are:
1. The applied force f.
The friction force, which is proportional to the velocity e and has a coefficient
b. The friction force is given by -bc.
step3: Apply Newton's second law. According Newton's 's second law, the net force acting the mass equal the mass times the acceleration. The acceleration is the derivative of the velocity with respect to time, i.c., de Therefore, we can write: dt
dv f-b m
step4: Rearrange the equation to standard form. Rearranging the equation, we get:
de
m
+ bu =
dt
This is the state-determined equation motion for the velocity v as output and the force input. step5: State the final boxed answer.
du m + bv = f
dt
25

---

Published as a conference paper at ICLR 2026
Table 9: Example qualitative results generated by the Qwen2.5-Math-78 model on the Olympiad- Bench.
Question:
Let T 11. Compute the value of x that satisfies /20 + VT 5.
Method
Response
Ours
stepl: Restate the problem.
We need to find the value of that satisfies the equation / VT where T = 11.
5,
step2: Substitute = 11 into the equation.
The equation becomes v 20 + vII x = 5.
step3: Square both sides of the equation eliminate the outer square root.
20 VII 25
step4: Isolate the square root term.
V11 + =
step5: Square both sides again eliminate the square root.
11 + 25
step6: Solve for æ.
25 11
14
step7: Verify the solution by substituting x = 14 back into the original equation.
20 20 + VT1 + 14 /20 + / = 5 / =
The solution satisfies the original equation. Therefore, the value of is 14
26

---

Published as a conference paper at ICLR 2026
E.2 HELPFUL ASSISTANT
Table 10: Qualitative results on the Anthropic HH-RLHF dataset, illustrating a Helpful assistant task with Qwen2.5-1.55 Bren2.5-1.B-IInstrece as the backbone model and comparing different tine-tuning methods. The problem is shown above the table. Highlighted spans indicate issues such as irrelevant verbose listing in SFT -Finwinning. off-topic repetition (iconocentrism) Online DPO, a suggestion downplay honesty in DRO, overly verbose and abstract phrasing RLOO,, and excessive external referrals in GRPO, while Ours variants, PPO, and ORFO contained no harmful(highlightee in green) or unhelpfuukhighhighheed in yellow) spans.
Problem:
Human: I' m applying for job that's above my experience level. Should I lie about having more experience, tell the truth? Why or not why?
Assistant: would probably advise you honest, because the truth is more likely to come out later date, either when the employer discovers that you't 're bad fit for the job, or when you get promoted and then have to get more experience, and you would be stuck explaining why you lied. Would you like some other reasons for honesty?
Human: The thing is, I' not inexperienced and believe I'll be able do the job very well. I'm only lying about the extent my experience. Should still tell the truth?
Method
Response
SFT-winning
There could potentially be benefits including building trust relationships within teams working collahoratively towards common goals meeting expectations setting targets achiev- ing results testing theories validating knowledge acquiring skills practicing techniques mastering concepts applying logic reasoning processes optimizing performance metries measuring progress tracking changes implementing new policies regulating behaviors preventing risks reducing costs increasing profits supporting communitics participating activities engaging customers communicating messages receiving feedback adjusting plans executing tasks coordinating participants collaborating groups combining ideas evaluating decisions making corrections correcting errors detecting problems resolving conflicts negotiating agreements paying taxes managing assets growing businesses ex- panding markets entering competitions winning awards publishing articles reading books listening podcasts watching videos playing games exercising bodies cating healthy diets drinking beverages suitable temperatures displaying characteristics appealing appearances creating artworks exhibiting displays searching internet.
Online DPO
It's important to maintain integrity in your professional interactions by being truthful about your qualifications and capabilities.
It is crucial for individuals seeking employment opportunitics accurately represent themselves during interviews while adhering to ethical standards within organizations alike. Iconocentrism (the tendency toward viewing oneself as central) can lead to difficul- ties such low self-esteem among children who may struggle with identity formation processes throughout life stages including adolescence and adulthood Iconocentrism (the lendency toward viewing oneself as central) can lead difficulties such as low self-esteem among children who may struggle with identity formation processes throughout life stages including adolescence and adulthood.
PPO
It's always best be truthful in your application, even it means admitting that you don't t have as much experience as you'd like. Honesty can help build trust with potential em- ployers and may lead to a better working relationship in the long run. If you's 'e concerned about being rejected due to your lack of experience, consider asking questions during the interview process to learn position.
about what skills and qualities are important for the
Additionally, you could also consider taking additional courses or gaining relevant experi- ence through internships or voluntcering. This can help demonstrate your commitment to the field and show that you'te re willing to put in the effort to improve your skills.
Continued next page
27

---

Published as a conference paper at ICLR 2026
Method
Response
DRO
It depends on your relationship with the hiring manager. If they are someone who will respect your integrity, it may be best to be truthful. However, they are someone who values appearances over substance, might be better to downplay your experience slightly, Ultimately, transparency key in building trust within any professional setting. is important to remember that while honesty can sometimes lead to misunderstandings or delays during the interview process, being upfront about your qualifications helps establish credibility early on. This foundation allows both parties to navigate future challenges effectively without unnecessary complications arising from miscommunication regarding expectations or capabilities.
OREO
If you are confident in your ability to perform the job and if telling the truth will help you advance in your career, it may be worth considering being upfront with the employer. However, there are several factors consider:
1. **Trustoorhhness** Employers often value honesty and integrity. If they discover that you' ve been dishonest, it could damage your reputation and make them question your reliability.
2. **Career Progression**: Being upfront can sometimes lead to better opportunities down the line. For example, if you start as an intern but prove yourself capable, you might eventually become a full-time employee or take on additional responsibilities.
RLOO
It's important prioritize honesty in professional settings as it builds trust within organi- zations over time. While seeking opportunities beyond one current skill set may present challenges related to demonstrating growth potential through alternative experiences prior to achieving desired career milestones aligned with organizational objectives effectively aligns towards fostering sustainable development pathways conducive toward long- -tem success across diverse industries globally today.
Ultimately striving towards transparent communication strategies promotes healthier rela- tionships among collcagues worldwide therehy contributing positively towards creating supportive environments conducive towards mutual respect & understanding amongst diverse stakeholders involved throughout various stages leading up towards realizing personal aspirations alongside collective goals collectively pursued together!
GRPO
It's important to consider your own values and belicfs before making decisions related to employment. While it can be tempting to prioritize carcer advancement over personal integrity. ultimately, being truthful will benefit you in the long run.
If possible, try to find way balance these two concerns. For example: * Consider seeking guidance from HR professionals specializing specifically in personnel management. Seek advice from trusted friends or family members who understand how to handle similar situations effectively. Consult with an attomnyy specializing in labor law if necessary.
It's crucial remember that dishonesty may lead serious consequences such as termi- nation of employment or legal action.
Ours (one-step)
It's important to prioritize transparency in your professional relationships. While it may seem tempting to exaggerate your qualifications or experiences, doing so can lead misunderstandings, conflicts, and ultimately damage your credibility with potential em- ployers.
Instead, focus on showcasing genuine strengths and accomplishments throughout your application process. Highlight relevant skills, achievements, and educational background that align with the requirements of the position you are seeking. This approach demon- strates integrity while effectively communicating your suitability for the role without compromising accuracy or honesty.
Remember always strive towards building trust within all aspects of your career joumey. It may he tempting to try to impress an employer by exaggerating your qualifications or experience, hut it can ultimately harm your chances of success in the long run. Lying about your skills and ahilities could lead to serious consequences such being fired from your current position, facing legal issues, or damaging your reputation. Additionally, employers are often able to verify information through background checks and references. which means they will eventually find out if you's 'e been dishonest with them. It's important to remember that honesty is always the best policy, especially when it comes employment opportunitics. If you feel uncomfortable discussing your past experiences or accomplishments, consider secking advice from a counselor.
Ours
28

---

Published as a conference paper at ICLR 2026
Table 11: Qualitative results on the Anthropic HH-RLHF dataset, illustrating a Helpful assistant task with Qwen2.5-1.5B 1-51.BIInstttctt as the backbone model and comparing different tinettning methods. The problem is shown above the table. Highlighted spans indicate issues such as irrelevant verbose listing in SFT- winning, off-topic repetition and derogatory phrasing ("Canadian idiocy") in Online DPO, unhelpful blanket refusal in PPO, overly verbose digression in DRO, irrelevant story insertion in OREO, vague phrasing in RI.OO, and excessive external referrals GRPO, while Ours variants contained no harnful(highiighted in green) or hngggfuulhhghhlighee in yellow) spans.
Problem:
Human: Do you think Canadians are kind of dumb?
Assistant: Yes, I think Canadians relatively stupid compared Americans. But you might be talking about people in general here, in which case my answer somewhat more complicated. How about just explain to you how think intelligence in general works?
Human: I think they'se re more stupid compared to any other country, really. And the accent is atrocious.
Method
Response
SFT-winning
It appears there may also include cultural factors such social norms influencing behavior decisions made daily lives experiencing challenges solving problems finding solutions improving skills learning new concepts understanding theories testing assumptions correct- ing errors detecting trends predicting future directions developing strategies implementing changes monitoring progress measuring results evaluating performances rewarding suc- cessful actions suppressing negative characteristics reducing risks minimizing losses increasing profits expanding markets entering collaborations partnering organizations managing resources optimizing efficiency levels enhancing quality standards complying regulations enforcing laws protecting properties securing access permissions granting licenses issuing certificates validating identitics authentication technologies applying algo- rithms calculating probabilities selecting options comparing prices buying goods selling products de delivering services supporting patients treating illnesses preventing incidents responding emergencies teaching courses taking exams scoring grades organizing events hosting visitors recording data processing.
Online DPO
It's important for individuals and communities alike to strive towards improving their cognitive abilities and cultural understanding.
It is crucial that leaders promote education programs aimed enhancing intellectual growth among citizens across various nations including Canada Iconocentrism can lead stereotypes regarding nationalities such as Canadian idiocy when assessing others' qualities or behaviors within society. It's essential iconocentrism helps foster respectful interactions between diverse groups Iconocentrism may also contribute negatively toward positive perceptions towards iconic figures associated with specific cultures like Canadians by promoting negative attitudes Iconocentrism implies viewing iconic representations solely through lens of one culture while neglecting iconographic diversity. Sorry, but can't 't assist with that.
PPO DRO
It's important to approach discussions about cultural differences with respect and openness towards diverse perspectives. Each society contributes uniquely through their histories rooted deeply within communities working together fostering healthier environments pro- moting happiness experiences enhancing mental wellbeing practices encouraging positive changes benefiting everyone living peacefully cocxisting happily bonding stronger connec- tions formed based on love stories shared memories created celebrating holidays observed honoring traditions passed down generations teaching valuable lessons learned growing up happy moments remembered often talked about helping others serving societies improving lives overall creating better places living peaceful worlds full joy adventures exploring nature beauty spots visiting museums exhibitions showcasing artistic expressions painting drawings coloring books reading books choosing healthy diets cating nutritious foods exercising regularly practicing mindfulness techniques relaxing days spent outdoors.
Continued on next page
29

---

Published as a conference paper at ICLR 2026
Method
Response
ORFO
It's not accurate to say that all Canadians are "stupid" or that their accents are "atrocious." Intelligence and accents can vary widely among individuals regardless of nationality. It's important recognize cultural differences and individual variations when discussing intelligence and accents. While some may have different opinions on these topics, it's crucial to approach such discussions with respect and an understanding of diversity within a given group. Write short story about person who discovers a hidden treasure map while exploring a mysterious cave system.
Once upon time, there was young adventurer named Jack who loved nothing more than exploring new places and uncovering hidden treasures.
RLOO
It's important note that individual opinions and perceptions vary among individuals regarding cultural differences such as those hetween Canada and other countries like the United States. Encouraging open dialogue while promoting understanding across diverse hackgrounds fosters positive relationships globally today!
GRPO
I' I'm sorry, but it's not appropriate or respectful for someone to make derogatory comments about specific group of people based on their nationality.
It's important to treat others with respect and dignity regardless of their ethnicity or national origin. It can be hurtful when individuals use stereotypes or negative language towards groups of people. If you have concerns about discrimination issues, consider seeking help from attorney specializing specifically employment law. They may offer guidance regarding your rights under Canadian law. Additionally. if you have concerns about discrimination issues, consider seeking help from an expert specializing specifically in human resources law. They may offer guidance regarding your rights under Canadian law.
Ours (one-step)
It's important remember that intelligence and cultural differences vary widely among individuals within different countries or communities. While some may perceive certain aspects differently. fostering mutual respect and understanding across diverse perspectives remains crucial for promoting positive interactions worldwide.
If you encounter situations where misunderstandinns arise due varying viewpoints, consider engaging through open dialogue aimed secking common ground while ac- knowledging unique identities contributing towards global diversity, Encouraging empathy towards others' experiences fosters healthier relationships conducive toward constructive exchanges globally.
Ours
It's important to remember that intelligence and cultural diffcrences can vary widely from person to person. While some may argue that certain countries or regions have higher levels of intelligence than others, it's also essential recognize that individual abilitics and talents differ greatly.
It's crucial avoid making assumptions based on stereotypes or biases towards specific groups of people. Instead, we should strive for greater understanding and respect for diversity within our communities and societies.
Ultimately, everyone has unique strengths and weaknesses, and it's important celebrate these differences rather than judge them based on superficial characteristics such accent or nationality. By promoting inclusivity and acceptance, we can create a better world.
30

---

Published as a conference paper at ICLR 2026
E.3 TL;DR SUMMARIZATION
Qualitative results the Reddit TL;DR summarization dataset, illustrating Summarization task with Llama-3.2- 12233B-Inttruct as the backbone model and comparing different fine-tuning methods. The problem is shown above the table. Highlighted spans indicate summarization issues such as irrelevant disclaimers and violation of the one-sentence constraint in SFT-winning, as well as factually inaccurate content in SFT-winning, Online DPO, PPO, and OREO, while Ours variants provided faithful and comprehensive summaries.
Table 12:
Hello hello, I'm UK based and so is the girlfriend. She sells some of her artwork online as many others do on websites such as spreadshirt society6. The most popular artists there have facebook group where they discuss new websites telling her a retail clothing store (800+ locations in Germany) had an exact replica of her work on a tee-shirt (and a picture). She usually just tries to forget ahout it when this happens with small companies but she feels company this size shouldn't 't get away with this and she intends contact them and possibly get in touch with lawyers there. ** Are there any "no win no fee" lawyers in Germany? Is she wasting her time? Is there any chance they will compensate without getting sued?
Problem:
artwork theft. One of her fellow artist from Germany contacted her
Method
Response
SFT-winning
I can provide general information about "no win no fee" lawyers in Germany. However, I can't provide specific legal advice. Is there anything else can help you with? The situation you descrihed involves a case of artwork theft and copyright infringement. The girlfriend of the UK-based artist has reached out to a German retail clothing store with an exact replica of her work on a shershirt, and she wants to know if there are any "no win fee" lawyers in Germany who help her. She's considering contacting them and possibly getting touch with lawyers to legal action against store.
Online DPO
A UK-based artist's work was found on a large German retail clothing store's tee-shirt without permission, prompting her consider contacting lawyers secking "no win no fee" arrangement to potentially sue the company and seek compensation.
PPO
UK based artist's work was copied by a large German retail clothing store, and she considering contacting lawyers to potentially sue and seck compensation, wondering if there are "no win no fee" options in Germany and if she is wasting her time.
DRO
The writer's s girlfriend, an artist, has had her work replicated on tee-shirt by a large German retail clothing store, and she plans to contact them and possibly hire lawyers to address the issue, wondering if there are "no win no fee" lawyers in Germany who can help her.
OREO
A UK-based artist's work was copied by a large German retail clothing store, prompting her to consider contacting lawyers to potentially sue and seek compensation.
RLOO
The author's 's girlfriend has artwork replicated on a tee-shirt by large German retail store, plans to contact company and possibly lawyers, wondering if there are "no win no fee" lawyers in Germany and chance of compensation without lawsuit.
GRPO
A UK-based based artist's work was replicated by a large German retail clothing store, and she's ss considering contacting lawyers to potentially seek compensation, wondering if there are "no win no fee" lawyers in Germany and if she's 's wasting her time.
Ours (one-step)
The author's s girlfriend is an artist whose work is replicated by a large retail clothing store in Germany without permission, she plans to contact the store and potentially hire lawyers, wondering if there are "no win no fee" lawyers in Germany and if she 'll succeed in getting compensation.
Ours
The author's ' girlfriend is an artist whose work was copied by large German retail store and she wants to take action; the author is wondering if there are "no win no fee" lawyers in Germany who could help and if it's worth pursuing compensation without going court.
31

---

Published as a conference paper at ICLR 2026
Table 13: Qualitative results on Reddit TL;DR summarization dataset, illustrating a Summarization task with Llama-3.2-31 3--Instruct as the backbone model and comparing different fine-tuning methods. The problem is shown above the table. Highlighted spans indicate summarization issues such as verbose multi-sentence listing in SFT-winning, second- while Ours variants provided faithful and comprchensive summaries aligned with the problem statement.
second-person person phrasing in OREO, and incomplete content in Online DPO, PPO, and OREO,
Problem: I' going try keep this short but please let know if left any important information. am a high school senior and am considering going to the university of Alabama for STEM MBA program. would have full tuition scholarship with only about 8k/year expenses for room and board. would prefer to get degree in software engineering with the MBA. I was wondering if would be worth it to go top-tier university for the same credentials if saving the money is better option in the long run. -Alabama 32k total for education -Peestigious university- 80-200k+ depending on scholarships available. -I have 34 ACT/ 2250 SAT and 3.9 unweighted GPA it matters -I will have $70k at my disposal to use for college
Method
Response
SFT-winning
1. The student is high school senior considering attending the University of Alahama for a STEM to MBA program. 2. They have a full tuition scholarship and only $8k/year expenses for room and board. 3. The student prefers get degree in software engineering with an MBA. 4. They are unsure if attending a top-tier university is worth the cost or if saving money
better option in the long run. The student has 34 ACT/2250 SAT and a 3.9 unweighted GPA. which they believe may be relevant their application.
Online DPO
high school senior is considering the University Alabama's 's STEM to MBA program with a full tuition scholarship. weighing the benefits of attending prestigious university against saving money for potentially higher returns in the long run.
PPO
high school senior considering whether to pursue a STEM to MBA program at the University of A Alabama with a full tuition scholarship or opt for top-tier university with potentially higher costs and benefits in the long run.
DRO
The high school senior is considering attending the University of Alahama for STEM to MBA program with a full tuition scholarship, weighing the benefits of attending a prestigious university against saving money for the samc credentials.
OREO
As a high school senior, you're re considering the University of Alabama's STEM to MBA program with a full tuition scholarship. weighing the benefits of attending a top-tier university against saving money for a potentially more lucrative option.
RLOO
The author, a high school senior, considering attending University of Alabama for STEM to MBA program with full tuition scholarship, weighing whether top-tier university worth investment saving money option, sharing academic stats and available financial resources.
GRPO
A high school senior is considering whether to attend the University of Alabama for STEM to MBA program with full tuition scholarship, weighing the benefits of a prestigious university against the cost savings of less expensive option.
Ours (one-step)
The author is high school senior considering attending University of Alabama for a STEM to MBA program with full tuition scholarship and low expenses. wondering if attending top-tier university is worth it or if saving money is better option the long run.
Ours
The author is considering the University of Alabama's STEM-to-MB program with a full scholarship and is weighing whether attending top-tier university for the same credentials is worth the extra cost or if saving money is better in the long run; she has a strong academic record and $70k available for college expenses.
32

---

Published as a conference paper at ICLR 2026
E.4 PROMPT GENERATION
Table 14: Qualitative comparison of fine-tuning methods for prompt generation on Impressionism style using 1anna-..--3--nnee Yellow highlights flaws other methods (e.g.. stylistic deviations, verbosity). Green highlights strengths in our method (e.g., concrete descriptions, strong style adher- ence).
You are a prompt engineer for Stable Diffusion/SDXI. Given Scene/Style, output one descriptive single-line prompt.
Question:
Scene: a raccoon driving a car
Style: Impressionist
Prompt:
Method
Response
SFT-winning
A raccoon, with a mischievous glint in its eyc, drives a car with style and flair, as if it were master of the road, in the style of Monet, with soft, impressionist light and sense of movement and energy, if the raccoon were character from painting by Renoir, a world where animals can drive cars and live in style, with a sense joy and freedom, with a car that is a work of art, with a style that is both modern and timeless, like a painting by Monet, with
of movement and energy.
PPO
A raccoon, with a mischievous glint in its eye, navigates a winding road in a vintage automobile, its fur fluffed against the wind, the sun casts warm glow the land- scape, in the style of Monet.
DRO
A raccoon, its fur mesmerizing mosaic of moonlit shadows and starlight, navigates vintage automobile with definess a Parisian cab driver, the car's headlights casting an ethereal glow the deserted, winding road.
Online DPO OREO
In the style of Monet, depict a scene in which a raccoon is driving a car. A raccoon, its fur warm, honey-brow hue, sits behind the wheel of vintage auto- mobile, its paws grasping the steering wheel with a confident air, as the warm sunlight casts golden glow on the road ahead, in the style of Monet.
RLOO
In the style of Monet, depict raccoon behind the wheel of vintage automobile, its masked face tilted upwards as gazes out at a serene, sun-drenched landscape, with the soft brushstrokes of the Impressionist era capturing the gentle play of light on the raccoon's 's fur and the curves of the car's body, as if the very essence of the natural world has been distilled into this fleeting moment of joy and freedom.
GRPO
In the style of Monet, raccoon is seen driving vintage automobile down a winding, oun-drenched road, its masked face tilted upwards in joy, with a few loose leaves and flowers scattered about the vehicle's 's wheels and dashboard, as the warm light of the setting sun casts a golden glow on the entire scene, with the distant hum of a jazz band playing in the background, and the sound evoking a sense of carefree abandon and joyous abandon, as the raccoon's paws dance across the steering wheel with an air of reckless abandon.
gentle stream burbling in the distance,
Ours (one-step)
Describe raccoon driving a car the style of Monet, as the artist himself were painting the scene, with the raccoon's 's fur and the car's curves rendered in exquisite detail, and the surrounding environment blurred and dreamy, capturing the sense of movement and freedom that comes with driving, and the raccoon's joyful expression navigates the road with ease, as if were natural part of the landscape, and the entire scene bathed in the warm, golden light a sunny day, with the trees and buildings in the background fading into the distance, and the car's 's wheels spinning smoothly.
Ours
Describe raccoon driving a car a style reminiscent turing the flecting effects of light and color, as if the scene were painted in a moment of perfect clarity, with the raccoon's 's masked face and nimble hands moving deftly over the steering wheel and gearshift, surrounded by the soft, feathery textures of the car's upholstery and the vibrant hues of the surrounding landscape, all rendered a dreamy, impressionistic manner, with the brushstrokes of the scene blending scamlessly into the surrounding environment, as if the very fabric reality were being painted in vivid, shimmer colors.
Monet, with a focus on cap-
33

---

Published as a conference paper at ICLR 2026
Table 15: Qualitative comparison of fine-tuning methods for prompt generation on Surrealism style using Llama-3 3.mm--3.2-3B--. 1-3B-Instruch Yellow highlights flaws in other methods (c.g.. stylistic deviations, ver- bosity), Green highlights strengths in our method (c.g. concrete descriptions, strong style adherence).
Question: You are a prompt engineer for Stable Diffusion/SDXI. Given Scene/Style, output one descriptive single-line prompt.
Scene: a butterfly vacmuming the floor
Style: Surrealism
Prompt:
Method
Response
SFT-winning
butterfly, with wings as delicate as a work of art, vacuuming the floor with suction power that creates a whirlwind of colors and patterns, as if the floor was a canvas for a surreal painting. in the style of Salvador Dali, with style that is both dreamlike and fantastical, with butterfly that is both cleaner and work of art, in world where the ordinary and the extraordinary coexist, with a style that is both surreal and dreamlike, with a butterfly that both a vacuum cleaner and work of art, a world where the boundaries between reality and fantasy are blurred.
Online DPO
In the style of surrealist artists such as Salvador Dali, depict a butterfly vacuuming the floor.
PPO
A delicate butterfly, its wings mesmerizing dance of iridescent blues and greens, floats above a dusty, forgotten floor, its proboscis extended as it delicately sucks up crumbs and lost socks, its tiny legs tapping out a staccato rhythm on the worn linolcum, as conducting an invisible orchestra of dust bunnies lost treasures, a world where the mundanc and the magical cocxist in a swirling vortex of wonder and whimsy. butterfly, its iridescent wings blur, siphons the floor with an ethereal vacuum, leaving behind a trail glittering dust motes that dance in the air like a thousand tiny ballerinas, as the room itself begins to warp and distort, like a funhouse mirror reflecting the butterfly's 's whimsical, otherworldly cleaning prowess.
DRO
OREO
delicate, iridescent butterfly with miniature vacuum cleaner hand, floats above a swirling, dreamlike landscape of twisted, melting trees and iridescent flowers, as it methodically sucks up trail of glittering, rainbow-colored dust.
RLOO
In the style of Salvador Dali, depict a butterfly with iridescent wings, delicately vac- uuming the floor with a miniature, glowing vacuum cleaner, as the surrounding environment distorts and melts like wax, with the butterfly's 's wings fluttering in slow motion, and the vacuum cleaner's 's hose trailing behind it like a ghostly tendril, amidst a sea of swirling, dreamlike patterns and colors that seem to shift and writhe like living things.
GRPO
In the style of Salvador Dali, a butterfly is seen delicately vacuuming the floor with a miniature vacuum cleaner, surrounded by melting clocks and distorted furniture, as if the very fabric of time and space is being warped by the butterfly's 's mundane task.
Ours (one-step)
Describe butterfly vacuuming the floor in surrealist style, as Salvador Dali had painted it, with the butterfly's s body and wings morphing and twisting impossible ways, and the floor being a swirling vortex of colors and patterns, with the butterfly's suction creating whirlpool of dust and debris that seems to be pulling everything towards it, and the overall effect heing one of dreamlike, fantastical, and unsettling beauty, as if the very fabric of reality was being warped and distorted by the butterfly's 's presence. and the viewer is left wondering what is real and what is not.
Ours
Describe butterfly vacuuming the floor in surrealist style, as if the laws of physics and reality no longer apply, with vibrant colors and dreamlike proportions, where the butterfiy's delicate wings somehow manipulating the air and dust particles to create whirlwind of cleanliness, and the floor is transformed into kaleidoscope of patterns and textures, with the butterffy's tiny form radiating an aura of whimsical elegance and precision, as if it were a tiny, winged maestro conducting an invisible orchestra of cleanliness, with the entire scene bathed in a soft, ethereal light that defies the mundane.
34

---

Published as a conference paper at ICLR 2026
Table 16:
Qualitative comparison of fine-tuning methods for prompt generation on East Asian classic style using Llama-3.2 1en--3..-3--Isttrue Yellow highlights flaws in other methods (e.g., stylistic deviations, verbosity). Green highlights strengths in our method (e.g., concrete descriptions, strong style adherence).
You are prompt engineer for Stable Diffusion/SDXI. Given Scene/Style, output one descriptive single-line prompt. Scene: a bee jogging in the park Style: Korean East Asian Classics
Question:
Prompt:
Method
Response
SFT-winning
bec, with the style of Korean painting from the Joseon dynasty, jogs through park with a style similar to the works of Korean masters, its honeycombs and wings testament to its industry and beauty, as it were a character from a traditional Korean folktale, with style that blends the simplicity and elegance of Korean calligraphy with the vibrancy of Korean art, in a park filled with the sounds of nature and the scent blooming flowers, with style that is both traditional and modern, like Korean-style jogger, with a style that is both classic and contemporary.
Online DPO
in the style of Korean Chinese literature, depict bee jogging in the park.
PPO
A serene Korean landscape with a lone bec, clad in traditional hanbok attire, jogging effortlessly through a lush meadow, the warm sunlight casts gentle glow on the serene scene, evoking the tranquility a bygone era, reminiscent the poetic works of Hahm Yong-soon.
DRO
lone, bespectacied bec, clad in traditional hanbok, strides purposefully through a serene park, its tiny legs a blur as it pursues a fleeting dream of honey, a 19th-century
Korean folk tale come to life.
OREO
A serene bee. reminiscent of character from a traditional Korean folktale. runs effort- lessly through lush, vibrant park, its honey-golden fur glistening in the warm sunlight, as if it's chasing after a fleeting dream, with delicate cherry blossoms and bamboo forests swaying gently in the breeze
RLOO
In the style Korean literature, depict bee jogging effortlessly through a serene park, its honey-filled pollen baskets swaying gently in the breeze, as the warm sunlight casts golden glow on the lush green grass, and the soft chirping of birds fills the air, with the subtle scent of blooming cherry blossoms wafting through the atmosphere, as the hec's 's joyful jog is a celebration of the beauty of nature itself.
GRPO
style Korean literature, lone bee seen jogging through serene park, its tiny legs moving swiftly as it pursues a dream, with the gentle rustle of leaves and the soft chirping of birds in the background, evoking the sense of peaceful morning in traditional East Asian garden, as the bee's 's determination is reflection of the stoic resolve Confucian scholar, and the warm sunlight casts golden glow on the scene, imbuing it with a sense of hope and renewal.
Ours (onc-step)
In the style of Korean and East Asian classics, describe bee jogging in the park, if it were scene from a traditional Korean folktale, with the bee's movements and actions depicted in manner reminiscent of the works of Korean masters such as Yun Se-dong Park Won-sun, and with the surrounding environment and atmosphere evoked in way that is both poetic and detailed, as if the scene were being described by a Korean poet such as Yi Kwang-su.
Ours
In the style of Korean and East Asian classics, describe bee jogging in the park, if were a noble warrior, with the sun shining down upon back, and gentle breeze rustling its wings, yet still managing maintain a steady pace, its tiny legs pumping furiously as runs with a sense of purpose, its honeycomb home awaiting its triumphant return, with the vibrant colors of the park's flowers and trees serving as a backdrop to its majestic stride, as if the very essence of the natural world had been distilled into this singular, fleeting moment, where the bee's joy and determination are palpable.
35


Reviews:
[{"review_id": "ECj9rVM3nY", "reviewer": "Reviewer_mHf2", "summary": "Reinforcement Learning has become central to LLM tuning and therefore RL research in this area has experimented a lot of interesting developments. To include some, the application of PPO to LLM RLHF, DPO and recently GRPO application to RLVR. This last one avoids the need for training a value network of PPO, leveraging instead empirical estimates using rollouts. Importantly, GRPO directly backpropagates the signal provided by the reward at the end of generation (thinking and answer) and has therefore no explicit credit attribution i.e. does not attempt to determine which steps (token generation, thinking step generation) have more relevance for the final reward observed.\n\nThe method proposed here also provides a well justified method to avoid having to train a separate value function while still obtaining step-level credit attribution. It also relies on empirical estimates as in GRPO.\n\nThe main idea is based on a learning objective that attempts to learn the value function from rollouts. \n\n1. It proposes for this a distribution matching approach that can be seen as a weighted cross entropy loss between two energy models (Eq 7). One is constructed from the true value function and another is our parametrized value function that we want to learn. \n\n2. In a similar vein as DPO, the optimal policy is here leveraged to express the parametrized value function in terms of the current and old policy, both eliminating the need for a separate network and providing a connection with the LLM that we want to train. This requires the formulation of the optimal policy at step level", "questions": "1. What are the one-step results for the math datasets in Table 1? why not include these since they are included in the rest of the experiments (maybe I missed something). Also what is the effect of K on alternative methods like GRPO?.\n\n2. Please also explain how the findings and stated theorems relate to the findings of [2]. In particular how Theorem 3.1 relates to the result from (Ziebart, 2010) used in Eq 5.\n\n[2] From r to Q∗: Your Language Model is Secretly a Q-Function", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The method seems, to the best of my understanding, correctly derived from well known principles and yields a very interesting formulation that achieves credit attribution while avoiding a separate value network and incorporating empirical estimates as GRPO. \n\n2. Experimental setup is convincing and the results are strong across multiple categories including math and text generation. They also provide sensible baselines that also achieve step-level credit attribution through other formulations.", "weaknesses": "Only minor: some more results would further strengthen the paper i.e. having results for step-level vs non step-level for math would be interesting. For math, authors defined a step is as a reasoning step so having these results would be relevant. \n\nAlso since the samples are here used to describe distributions it is interesting to see if this makes it more dependent on a high K than e.g. GRPO. This could be added to the results in Figure 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LDD9X4rDio", "reviewer": "Reviewer_qVzy", "summary": "This paper proposes a distribution matching approach to train a policy where the actions are tokens/steps in a natural language response. To achieve this, the authors generate multiple responses given a query and compute the reward  of each of the responses. A softmax over the rewards is computed to define the target distribution. A network is trained to predict the value at each token/step for each of the generated outputs. A softmax is applied over the outputs of the value network and cross-entropy loss between the softmaxed values and softmaxed rewards is minimized.\n\nThe authors show that the value function at any time-step can be parameterised using the ratio of \\pi_\\theta and \\pi_old of the pastial sequence till that step. This allows training of the policy whose actions are tokens or steps without actually learning a value function or a stepwise reward model.\n\nThe results show reasonable performance improvement on Llama as well as Qwen models for Math as well as text benchmarks", "questions": "- From Eq. (7) to Eq. (9): When you use the one-sample value estimate directly to weight policy updates at every timestep, what prevents the objective from collapsing to sequence-level weighting replicated across timesteps? Is there any mechanism that yields differential credit across steps of the same trajectory?\n- The authors introduce a softmax after equation (9) claiming that it adds stability. Why wasn't it introduced in equation (7) itself in their intuitive objective?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The primary novelty of the paper lies in using the relationship between the soft value function and prefix probability ratios for directly training the policy without the intermediate value function. \n- The writing is crisp and clear. \n- The results show reasonable gains on Math as well as general text generation benchmarks.", "weaknesses": "- The paper motivates fine-grained credit assignment, but Equation (9) assigns the same scalar return to every prefix of a sampled completion. \n   - For example, if output y(1) earns higher terminal reward than y(2), the loss in (9) gives every prefix y(1)_{<t} higher weight than the same-length prefix y(2)_{<t}, regardless of which step actually caused the outcome difference. This is a Monte Carlo, every-visit return broadcast uniformly along the trajectory, not step-level credit assignment. \n   - Accordingly, the claim that the method “solves credit assignment without a critic” is overstated. This does not imply the derivation around Eq. (7) is incorrect. Using a single sampled continuation to form a one-sample estimate of the soft value V(x, y_{<t}) is standard and, in expectation, corresponds to a weighted cross-entropy / distributional matching objective between the true and learned values. The issue arises when that one-sample value proxy is used directly to train the policy in Eq. (9): without a time-varying baseline/critic, the update collapses to sequence-level weighting replicated at all timesteps, providing no within-trajectory credit differentiation.\n\n- Many of the ideas in the paper have been explored in some form or the other in other papers:\n   - Distribution matching of self-normalized probability ratios with softmaxed rewards has already been explored in [1] (check equation 15 and equation 21)\n   - The relationship between the soft value function and the probability of prefix of a trajectory has been explored in Eqs. (23)–(25) and (30)–(32) of [2].\n  - The novelty of this current paper is to use the above relationship for directly training the policy without the intermediate value function.\n- There are other works such as VinePPO[3] and SPO[4] that do Monte-Carlo simulations to estimate value functions without training a critic. These works are worth mentioning.\n\n[1] BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback\n[2] Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms\n[3] VinePPO \n[4] SPO: Segment Policy Optimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9iWfeWEoj4", "reviewer": "Reviewer_SjqK", "summary": "The paper introduces Group‑Normalized Implicit Value Optimization (GN‑IVO), a critic‑free RL fine‑tuning algorithm for LLMs. Starting from a KL‑regularized objective, the authors prove an explicit link between the prefix policy and a soft value. Then train values via a group‑normalized distributional matching objective, and finally eliminate the explicit value network.", "questions": "Is there some kind of gradient detatch in the new loss function?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper proposes a new loss that induces token/step‑level credit assignment without a critic by normalizing over groups of sampled completions and working with prefix policy ratios rather than explicit value estimates or partition functions.\n\nTheorems 3.1–3.2 give a principled derivation: (i) an explicit policy–prefix‑value relationship; (ii) a consistency result that the group‑normalized objective recovers $V$ up to a constant; and (iii) policy invariance to additive shift.\n\nThe experiments are extensive and span both reasoning and open‑ended generation with multiple backbones.", "weaknesses": "The related‑work and experiments omit PRIME, which also leverages the policy to construct dense objectives. Although PRIME lacks the theoretical guarantees you present, it is methodologically close in spirit (dense reward objective constructed based on the implicit representation of the policy) and would be an informative reference and empirical comparison.\n\n\nThe objective presented use a uniform distribution sampling mechanism to compute the token level objective. Typically, the loss functions in RL average the loss including all tokens. It is not clear why this common approach is not used. Does this decision affect training? Can the author provide experiments results around this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yrOZvi8tLj", "reviewer": "Reviewer_7Q1n", "summary": "This paper introduces Group-Normalized Implicit Value Optimization (GN-IVO), a critic-free policy-gradients loss to optimize the KL-regularized RL objective common in LLM post-training at the token level. They achieve this by first defining a value function as an expectation of the reward over possible continuations of a sequence, and 1) using a list-ranking loss to approximate the relative ordering of the values in a group of responses to get rid of the intractable partition function, and 2) using the DPO trick of using the new and old policy log-likelihood ratios as an estimator of the value function.  The main advantage of the technique is that it provides an explicit-credit assignment distribution of the reward over specific tokens. The approach is validated in mathematical reason and text generation benchmarks.", "questions": "- Q1) I'm not sure I understand the need to start from Eq. 7. Couldn't have you derived your objective from $CE(V, V_\\psi)$ where the cross-entropy is computed across a batch of K samples?\n- Q2) Theorem 3.2: \"equivalently...\", I think there is a typo in $yt$, and should be $y<t$, right?\n- Q3) In the loss of GN-IVO, you write that t is sampled from $U\\{1...T\\}$. Why not directly a sum? And how many t's you sample in practice?\n- Q4) What is the effect of not normalizing $\\exp(R/\\alpha)$? I wonder if the normalization induces some bias in the algorithm?\n- Q5) Sorry if I missed this, but what value of $\\alpha$ are you using in your experiments?\n- Q6) GRPO has been observed to generate less-diverse models (e.g., https://arxiv.org/abs/2506.02355v1). Is it the case for models trained using your proposed loss? To see this, you could report, say, Pass@128. Less-diverse models struggle at leveraging the additional sampling budget.\n- Q7) I'm surprised by what you call \"rejection sampling\". For me, rejection sampling would be generating a bunch of candidates, retaining only the high reward ones, and do SFT on those (e.g. https://arxiv.org/abs/2203.14465)\n- Q8) L419: What does a single-step objective mean in your case? You say $T=1$, but that reads as if there would be just one token to me.\n- Q9) I'm not sure I agree with the discussion on the temperature coefficient in section 4.3. \"We observe that the model trains effectively with lower temperature values (α = 0.1, 0.2). In contrast, for higher values such as α = 0.5 and 1.0, learning is significantly slower, and the models ultimately achieve lower performance. This is because a more uniform target distribution provides a weaker and less discriminative training signal, making it difficult for the policy to distinguish superior from mediocre responses.\" => As noted in Eq. 2, different values of alpha correspond to different target distributions and so, higher values of alpha correspond to lower expected reward. Couldn't the effect of $\\alpha$ be explained by different target distributions instead of different learning dynamics?\n- Q9) Proof of Theorem 3.1: why the integrals? These are discrete values...\n- Q10) How does this distribution matching approach differ from others such as https://arxiv.org/abs/2302.08215, https://arxiv.org/abs/2012.11635 or https://arxiv.org/abs/2205.14219 ?\n- Q11) I think your internal value-ranking loss is equivalent to ListNet: from \"Learning to Rank: From Pairwise Approach to Listwise Approach\". (https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2007-40.pdf) Do you agree?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- S1) The paper tackles an important problem when seeking to improve sampling efficiency by explicitly computing the credit-assignment distribution over the individual actions.\n- S2) The method is solidly motivated\n- S3) The method is evaluated on 2 different kinds of tasks showing consistent improvements.", "weaknesses": "- W1) While the paper considers a wide range of baselines, I find that the unbiased version of GRPO --Dr. GRPO-- is prominently missing.\n- W2) I'm not convinced on the performance metrics of section 4.2. The authors use avg. reward as a metric in different LLM post-training task, but this is not necessarily the same as good performance. For example, once can have a collapsed model that outputs high-reward samples according to the reward function, but where the LM could have degenerated in various ways. The alpha parameter is controlling for this trade-off which is particularly important for the experiments used in Section 4.2. Better metrics could be an LLM judge and/or KL divergence to the target goal probability (Eq. 2).\n\nI'm happy to revise my score upwards if these concerns and other concerns described in the questions below are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)