Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Authors: Simon Matrenok
Skander Moalla
Caglar Gulcehre
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning fine-tuning
alignment
offline alignment
direct alignment algorithms
large language model
Primary area: deep_learning
TL;DR: QRPO is a SoTA alignment algorithm that can fit the KL-regularized RL objective without relying on preferences.

Abstract: Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO.
In contrast, simpler methods that can leverage offline or off-policy data, such as DPO and REBEL, are limited to learning from preference pairs or relative signals.
To bridge this gap, we introduce Quantile Reward Policy Optimization (QRPO), which learns from pointwise absolute rewards while preserving the simplicity and offline applicability of DPO-like methods.
QRPO uses quantile rewards to enable regression to the closed-form solution of the KL-regularized RL objective.
This reward yields an analytically tractable partition function, removing the need for relative signals to cancel this term.
Moreover, QRPO scales with increased compute to estimate quantile rewards, opening a new dimension for pre-computation scaling.
Empirically, QRPO consistently achieves top performance on chat and coding evaluations—reward model scores, AlpacaEval 2, and LeetCode—compared to DPO, REBEL, and SimPO across diverse datasets and 8B-scale models.
Finally, we find that training with robust rewards instead of converting them to preferences induces less length bias.

Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Simon Matrenok CLAIRE, EPFL
Skander Moalla" CLAIRE, EPFL
Caglar Gulcehre CLAIRE, EPFL
Abstract
Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO. In contrast, simpler methods that can leverage offline or off-policy data, such as DPO and REBEL, are limited to learning from preference pairs relative signals. To bridge this gap, we introduce Quantile Reward Policy Optimization (QRPO), which learns from pointwise abso- lute rewards while preserving the simplicity and offline applicability of DPO-like methods. QRPO uses quantile rewards to enable regression to the closed-fomm solution of the KL-regularized RL objective. This reward yiclds an analytically tractable partition function, removing the need for relative signals to cancel this term. Moreover, QRPO scales with increased compute estimate quantile rewards, opening a new dimension for pre-computation scaling. Empirically, QRPO consis- tently achieves top performance chat and coding avaluations-rewall model scores, AlpacaEval 2, and LeetCode-compant across diverse datasets and 8B-scale models. Finally, we find that training with robust rewards instead of converting them to preferences induces less length bias.
to DPO, REBEL, and SimPO
Table 1: Policy fitting methods are popular for their simplicity and ability work offline, but cannot
fully use the signal from pointwise absolute rewards (c.g., strong reward models, verifiable rowards); they learn from relative rewards (preferences or reward differences). QRPO fills this important gap.
Offline (& Online)
Pointwise Rewards
Policy Improvement: PPO, GRPO, RLOO, etc.
X
Policy Fitting: DPO, REBEL, SimPO,
Policy Fitting: QRPO (Ours)
Best Alaa 2 LC Scores (1)
Table QRPO is more suitable for code generation tasks such as LeetCode, where the reward is canonically pointwise, ex- pressing the test-case pass rate.
50.6
49,2
LC Score
Method
Avg. Pass (%)
Llama 8B Tülu SFT
20.9
+ SFT on Correct
18.8 = 1.
REBEL
DPO
30.2 14
REBEL
DPO
SimPO
(Ours)
+ SimPO
22.3
Figure QRPO outperforms policy fitting methods that learn from relative signals in downstream general chat tasks.
QRRO (Ours)
"Shared first authorship and equal contributions, Correspondence skander muanderrmoall@@pf...dm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pre- computation phose
Prompt
Ref. Model Outputts
Ref: Model Rewordss r refy, Cref
Ref. Rewandss Distribution
x
yref Yrefz
Far kit
Training phose
OfFline /Online data
Rewaval
Guantile Rewardl
Supervised Learning Objective Tellen )e 0,7 min E (Ry -0lypllyz 24 plag Thelginl whrre 24- p(exp(())))))
Prompt x, Output y
R
QRPO uses quantile rewards, which makes the exact expression of the partition function Z tractable and allows fitiing the solution of the KI.-regularized objective with a pointwise regression, i.e., using a single sample with reward instead of a pair preferences. the pre-computation phase, we generate reference completions from the reference model and compute their rewards. For training, we then use these reference rewards to compute the quantile reward of training samples. In its simplest form. QRPO optimizes the quantile reward; however, family of transformations recover a desired reward shape, while still having a tractable exact expression for Z (see Table 4).
Figure 2:
be applied top of the quantile reward to
Introduction
Alignment methods are highly effective in fine-tuning large language models (LLMs), improving their conversational abilities (Ouyang al., 2022; Stiennon al., 2020; Rafailov et al., 2023), safety (Bai et al., 2022), and reasoning capabilities (Lambert et al., 2024; Shao et al., 2024a; Guo et al., 2025). They are applied as the final stage of the LLM training pipeline after pre-training the LLM on large corpus of data (Brown et al., 2020) and fine-tuning the LLM to follow instructions (Ouyang et al., 2022). However, while simple and efficient methods, such as DPO (Rafailov et al., 2023), relying on preference reward signal, achieve strong performance in conversational tasks, more complex and computationally intensive methods, such as GRPO (Shao et al., 2024b), have been the only alternative to effectively optimize pointwise reward signal.
Alignment methods are often referred to by different names, such as reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022), reinforcement learning from verifiable rewards (RLVR) (Lambert et al., 2024), and direct alignment algorithms (DAA) (Rafailov et al., 2024), but the underlying algorithms in most of these methods optimize common objective from regularized reinforcement learning (RL) (Jaques et al., 2017). This nomenclature distinguishes two dimensions: whether the reward is optimized explicitly (RLHF, RLVR) or implicitly (DAA) and whether it mod- eled from (human) preference feedback (RLHF, DAA) or objectively defined (RLVR). Yet, another crucial dimension the data regime required by the underlying algorithm, which significantly impacts the computational cost of the methods. Policy improvement (PI) methods, such as PPO (Ouyang et al., 2022; Schulman et al., 2017), GRPO (Shao al., 2024b), and RLOO (Ahmadian et al., 2024), require online sampling as they perform online policy improvement, thus increasing the complexity and cost of distributing large networks across training and inference engines. In contrast, policy fitting (PF) methods, such as DPO (Rafailov et al., 2023) and REBEL. (Gao et al., 2024), can leverage any data distribution because they aim to fit closed-form solution of the regularized RI. objective valid for all data, giving them the reputation for being simpler and more stable, as usually used with offline data.?
Existing PF methods rely on relative rewards, i.c., differences between rewards, to tractably fit the closed-form solution, estimating it for absolute rewards is known to be intractable due to the difficulty of estimating the partition function. This feature has contributed to their popularity, as human feedback be expressed as preferences and modeled as relative rewards by preference models. However, relative rewards can provide a suboptimal signal, for example, when both the chosen and rejected completions are better than the training model. This leads to conflicting dynamics with updates from samples that could both improve the model. Moreover, human feedback can be more easily collected absolute feedback in the form of a rating scale. More pressingly, with the emergence of powerful absolute reward models for human aligmment trained with regression
In the LLM literature, the terms are "RL" methods and "DAA" methods, but we feel that this is not accurate as all methods optimize an RL objective. and methods like REBEL do not fit within the "direct" naming which refers directly using preference model, which is not necessary target the closed-fis form solution optimization.

---

such as ArmoRM (Wang et al., 2024) and Nemotron Reward (Wang et al., 2025) and successful absolute-reward applications such as verifiable rewards (Lambert al., 2024), PF methods are losing applicability and practitioners have to resort to complex PI methods to use those rewards.
To overcome this limitation, we propose Quantile Regression Policy Optimization (QRPO). This new RL fine-tumme tine-tuning algorithm retains the simplicity and benefits of policy fitting methods while enabling the use of absolute rewards to overcome their limitations and broaden their applicability. QRPO is theoretically sound, simple, and effective scale. We make the following contributions:
1. QRPO addresses the problem of estimating the partition function in policy fitting methods QRPO able to fit the closed-form optimal policy of the KL-regularized RL objective for individual points with a pointwise absolute reward signal using a simple supervised regression. The key insight to making this possible is using a reward distribution for which the exact expression of the partition function is tractable. This optimization strategy has not been widely explored due to the common concern that partition functions are generally intractable. QRPO relies on quantile rewards and their distribution to derive this expression. This gives partition function equal to ((cxp(1/3) - 1). Furthermore, we frame QRPO as a framework in which applying additional transformations on top of the quantile reward still yields tractable partition functions.
2. QRPO scales with a larger pre-compute budget to estimate quantile rewards We demonstrate that generating more reference rewards to estimate quantile rewards improves performance.
QRPO consistently achieves top performance in conversational and coding tasks We show through an extensive and fair experimental protocol that QRPO consistently obtains the best rewards and AlpacaEval (Li et al., 2023) scores. We compared our method to DPO, REBEL, and SimPO, where each was trained in more than 200 settings of models, datasets, hyperparameters, and distribution shifts, including Llama 8B and Mistral 7B as models and Magpie-Air (Xu et al., 2024b) and UltraFeedback (Cui et al., 2024) as datasets. We also demonstrate that QRPO is effective LeetCode (Xia et al., 2025) coding task, where the reward is canonically pointwise. expressing the test-case pass rate and loses signal when expressed as preference.
4. Methods that directly use a robust reward model exhibit less length bias than methods that have to convert this reward to preferences We show that SimPO, which tries to overcome length bias by incorporating an inductive bias using length normalization, reduces absolute length but still shows strongly length-biased policies like DPO, while QRPO and REBEL do not.
2 Preliminaries
RL fine-tuning for LLM Alignment The underlying RL fine-tuning objective in alignment meth- ods is formulated as (Jaques et al., 2017: Stiennon et al., 2020; Rafailov et al., 2023)
max EE By
E (11a)
R(2,,))
310 xI (To(- []]] Tre (11)) x>)
(I)
DND
The goal is to maximize the expected reward R of Thg over prompts sampled from a distribution of user queries ~ D, while enforcing a constraint to keep close to Tref, the instruction fine-tuned pre-trained L.M,, using a KL divergence term, weighted by B. no is initialized from Tref, and the regularization prevents the model from over-optimizing the reward function by ensuring it stays close to the original distribution, maintaining fluency and diversity.
Policy improvement (PI) methods PPO (Schulman al., 2017) and their variants rewrite the RL. fine-tuning objective by moving the KI. penalty to the reward (Stiennon et al., 2020, PPO), or consider it an auxiliary loss (Shao al., 2024b, GRPO), then effectively optimize standard RL objective by taking policy gradient steps to improve the policy online. Hence, these methods require sampling before each gradient step, requiring extra complexity host inference instances of the policy, and making them more costly in terms of time and computational resources.
Policy fitting (PF) methods
Objective admits a closed-form solution, expressed as
1 Z(x) (re 2) exp (1 Z(x) = Tresfy 1)) exp ((R(,.))) (2)
x
( 22)
Policy fitting methods leverage this expression to fit the optimal policy. By substituting the optimal policy n² with parameterized model To in Equation 2, we can formulate distribution matching or
3

---

regression task. However, estimating nT" directly is known to be intractable due to the difficulty in estimating the partition function Z (Rafailov et al.. 2023). Hence, current methods resort to relative signals between two completions to eliminate the absolute partition function. Indeed, we have:
x (V 2)
R v) 8 log
log 2(x),
Tre x)
and we see that the difference in rewards between two completions gives relative expression between two policies that is independent of the partition function:
π ( 2) 1x) log
x (V Tres (3
| x [x)
R(x, y R(x, v log
(4)
Trey (yt
When the reward is used to parameterize a preference model, such as the Bradley- Terry model (Bradley & Terry, 1952), this difference gives the preference between two completions p(y* > y y) a (R(2,, yt) - R(x, y y)), and direct preference optimization methods such as DPO (Rafailov et al., 2023) learn the policy by substituting the reward difference by the policy difference and fitting the preferences. Gao et al. (2024) propose REBEL,, which fits the optimal policy by directly regressing the difference in log-probability ratios to the difference in rewards (as shown in Equation 4). The REBEL algorithm was originally presented as an iterative method; however, in this work, we focus on the RL fine-tuning objective with fixed reference policy, and as the first iteration of REBEL already fits the optimal policy relative to Kref we present REBEL as a policy fitting method.
Fitting relative signal from two completions of policy still remains a significant limitation, as two completions could both be better or worse than what the current model outputs, causing conflicting dynamies through updates to the policy to both increase and decrease the likelihood of two good or bad completions. The relative signal also naturally degrades the absolute information present in a single sample. In what follows, we present a novel algorithm that enables policy fitting with an absolute reward signal, addressing the abovementioned limitations.
3 Quantile Regression Policy Optimization (QRPO)
canonical objective
We start by rearranging Equation as
(y x) R(a,y) log ((x) 3 log Tref (y) x))
By substituting the optimal policy x* with a parameterized model na. we obtain the following regression objective for the RL fine-tuning task:
2
To( (x) Fref (y x))
min E To J.y
v) log 7(x) 33 Blo
(6)
calibrated target
We choose the mean-squared error (MSE) as loss, which is also a common choice in other LLM alignment algorithms (Azar et al., 2024). Moreover, consequence of the central limit theorem, the noise at the target value is likely to have a Gaussian distribution, which makes the MSE appropriate.
The objective in Equation is the most straightforward regression objective derive from Equation and is aimed at by previous work (Gao et al., 2024; Pierre Harvey Richemond et al., 2024), but is still intractable due to the difficulty in estimating the partition function Z, leading to these previous attempts to resort to approximations (Pierre Harvey Richemond et al., 2024) or to forgo it entirely (Gao al., 2024). However, this form has multiple benefits, and QRPO's 's main contribution is to transform the reward to maintain this regression objective while making the partition function tractable.
Why this objective? First, unlike policy improvement methods such as PPO, the training com- pletion y does not have to be online with respect to To and can be sampled from any distribution, such as an offline dataset to leverage pre-existing data (as in DPO originally), online data when novel exploration is needed, as in Online DPO (Qi et al., 2024), data. This is a significant benefit of policy fitting methods.
mix of offline, off-policy, and on-policy
Second, the primary rationale for employing absolute reward signal instead of a relative one (as in DPO or REBEL) that an absolute signal not only contains strictly more information but also qualitatively influences the optimization dynamics when data coverage is limited. Relative reward

---

methods leverage limited data coverage by increasing the difference between paired samples, often simultaneously lowering the probabilities of both (Pal al., 2024). This consistent reduction in prob- abilities leads to unintended probability mass drifting toward out-of-distributinn samples, resulting in unstable or ineffective long-term training (Razin et al., 2025). In contrast, an absolute reward approach enforces model probabilities to match fixed, predefined targets, directing probability mass explicitly toward higher-reward samples rather than drifting towards out-of-distribution outcomes, enabling stable and effective optimization. See Appendix A for comprehensive discussion.
Insight 1: The partition function can be expressed in terms of rewards only The main argument commonly supporting the difficulty of estimating (xx) in Equations 6 and 2 is the intractability of the infinite sum over completions y for an L.M. Our first insight is to show that the partition function can be expressed using only the distribution of reference rewards R(x, y) for completions y drawn from the reference policy [- 2)). Let Pr Pref (2) be the probability of a reference reward r, and M denote the moment generating function (MGF). We show that (proof in Appendix D):
[ear]
(xx) = PreP)r)
M, (1/5).
This derivation allows to have a better intuition of the challenge of estimating the partition function: it is particularly challenging with reward distributions that have a right tail because the exponent of large rewards, which happen with low probability, makes the estimate very noisy. We support this intuition with a theoretical analysis of the noise amplitude of (xx) when rewards follow a standard normal distribution, which can be example for typical LLM reward distributions, such in our experiments, which do have a right tail. We show that with 3 = 0.1, approximately 10.0 samples are required to achieve reasonable signal-to-noise ratio in the estimate of (ar) (proof in Appendix H).
Insight 2: We can derive Z analytically if we know the reward distribution Our rewriting of (xx) in Equation 7 opens the new alternative of computing (xx) analytically. In fact, the partition function takes the form of the moment generating function at 8 for the reference reward distribution, which is known for common distributions or could be derived by computing the expectation.
1
Insight 3: QRPO- the quantile-based reward gives a uniform distribution The reference reward distribution can be arbitrary, and assuming a specific distribution to leverage the above insight could introduce significant errors. However, we can transform the initial reward into one with a known distribution. QRPO uses the quantile transformation to achieve this. We begin by defining the cumulative distribution function (CDF) of rewards under the reference policy for a fixed input x::
(2,
Pr gf (1~7)o(((1z)
(R(2, v') < rr}
This CDF intuitively represents the probability that sample from the reference policy will have reward less than or equal to a given value r. We then introduce the quantile reward transformation:
R.y y) = Fres (z, R(x,y))
Pr
(R(x, v/) < R(x,y))-
When the reference reward distribution can be treated as continuous, such as with a reward model or coding test-case pass rate, as our experiments, the resulting distribution of the quantile-transformed reference rewards (2r, y) for y ~ Tyes(- x) is uniform, independently of the reference policy (Appendix E).² Therefore, using Eq. 7 which holds for any rewards, the corresponding partition function for this new reward is simply the MGF of the uniform distribution (details in Appendix F);
Zz(x) = 3 (exp (1/3) - 1).
(10)
Hence, we obtain tractable regression form of Objective 6 (see Appendix A for an interpretation):
CQRPO
E (R,(I,)))) - -3 1og 2.y
(
exp
] )
1
3 log
To (y x) Tref ( xx)
(11)
For computational stability, we neglect the -1 term exp(1/3) is typically very large for useful values of B < 0.1, and use (g) + 1/3 for log Z, obtaining
(v xx) Tref (y) xx)
2
CQRPO
R2 v) log((3)
3 log
(12)
2.99
$ For completeness, we also derive QRPO for reward distributions with a small number of possible values in App. F by directly estimating the expectation for quantile rewards, they do not result in known distribution.
5

---

Algorithm 1 (Offline) Quantile Reward Policy Optimization
Require: Dataset D = = ys3 of prompts and completions, reference policy Tref. reward function
R. scaling factor 3, number of reference rewards to generate, and number of training steps T.
Pre-computation phase:
(Datasel rewards) Vz;, Vi E DD pre-compute Tref (vi x;) and R(vi, (Reference rewards) Vz; E DD sample n reference completions Vi ~ Traf(: (x)) and annotate them with R. to obtain the reference rewards Sref.i (R(z,, ((.)).
Training phase:
Initialize policy To <- Tref and minimize the following loss with gradient descent for 7 steps.
BIDDDDD (2, - log 3 - - --3lgg (1 x4))
Tref (y: x3s)
Where R, vi) = [rresal ER ER(r....)ES....: [(R(s,, (h.) S Rx,,,,
QRPO as a framework We can generalize the derivation of QRPO to more general reward transformations. Specifically, we can apply wide range of functions / on top of the quantile reward, while still retaining the ability to analytically derive the corresponding partition function when the reference rewards can be treated as continuous (proof in App. F):
Riz, y) f (R,(;,))); Z(z)= Mr (1/B) L. exp (cPP))(() dt.
(13)
where I represents any function defined on the interval [0, 1]. This formulation demonstrates significant flexibility of QRPO, positioning as a general framework and allowing the design of reward distributions with specific properties that shape the structure of the optimal policy.
new optimal policy The optimal policy of the RL fine fine-tuning objective depends on the reward. Therefore, optimizing the quantile reward (Eq. 9) or its generalized transformed version (Eq. 13) produces policies with different properties compared to optimizing the original reward. Notably, Balashankar et al. (2024) demonstrate that optimizing the quantile reward is equivalent to optimizing the win rate against the reference policy and further allows to optimize the policy performance for a specific inference-time scaling strategy by choosing specific transformation f on top of the quantile reward in Eq. 13. This highlights key advantage the QRPO framework.
Foundations for theoretical analyses The analytical tractability of the partition function in QRPO just one example of the broader opportunities for theoretical analysis enabled by quantile- based reward framework. To illustrate this, we conduct an initial investigation into the impact of different transformation functions Eq. 13. In Appendix K, we show that for class of functions f() that are strictly increasing and finite at /. = 1, the resulting optimal policy is invariant to the specific choice of when / is sufficiently small. However, understanding the effects of other types functions, the relationship between the optimal policy and the optimization dynamics, and several other related questions remain open problems and represent key directions for future research.
Noise amplitude in the regression objective We have eliminated the noise in the partition function by using an exact expression computed analytically. However, we have introduced new noise in the regression objective, originating from the transformed reward Ry (z. y), as it requires estimating a quantile. We show in Appendix H, using a similar analysis as in Insight with a Gaussian distribution, that the noise in the calibrated target (Eq. 6) computed by keeping the original reward and estimating Z (x: is significantly higher than when estimating the quantile reward and using the exact expression of Zxx), in particular for the order of magnitude of 3 commonly used in RI. fine-tuning (0.1 and lower). Hence, QRPO reduces the noise in the objective function dramatically, and we therefore enable the general approach of policy fitting with pointwise absolute rewards to work well in practice.
A practical algorithm QRPO is implemented similarly popular policy fitting algorithms pri- marily used in an offline regime for their computational simplicity, so we present an offline version of QRPO (although QRPO is not limited to the offline setting). QRPO does not use pairwise preference dataset; it uses a dataset with prompts and completions (as in SFT), but also requires a is to generate reference completions
reward model or function. preliminary step (pre-computatiots and annotate them with rewards denoted Sref.i (R(,,,, Di.j)} to estimate the quantile rewards during training. We used between I and 20 reference completions our experiments, and note that these generated completions can themselves be reused as the training completions to serve as

---

0.170
0.188
0.182
0.169
Reward
Reward
0.181
Reward
0.186
0.180
0.168
0.184
off palicy, $-0.003
0.179
oft-paliyy A=0.0L
0.167
off pallcy, P=0.1 offine, 8=0.1
attine, f50.003
0.178
offine, (=0.01
0.182
10"
10 Number of references
10
10a Number of references
0.166
10"
10' Number references
Figure 3: Scaling Performance of QRPO with varying number of reference rewards generated during the pre-computation phase to estimate quantile rewards, off-policy and offline distribution shifts with different values of KL regularization (8) for Llama on Magpie-Air Air with ArmoRM. We report the average online test reward with error bars indicating the standard deviation over three seeds. In the off policy case (where training samples closely match the reference model's distribution, here generated by the reference model itself), performance steadily scales with more reference rewards, and cost-effective for higher regularization (B 0.1), typically used large post-training pipelines. contrast, in the offline case (high distribution shift due to training samples from a more performant model in this case) QRPO achieves near-optimal performance with very few reference rewards and has minimal gains from additional pre-computation, which may be more cost-effective for lower regularization (B- 0.003). This shows QRPO's 'ss pre-computation scalability and effectiveness in both scenarios.
an off-policy dataset. Then, in the training phase, QRPO minimizes Equation 12 with fully offline stochastic gradient descent, computing the quantile rewards using Sref. A summary of the algorithm is presented in Algorithm 5, A nice feature of the quantile reward in QRPO that, as in DPO, allows comparing 3 values across domains independently of the original scale of rewards.
4 Experiments
Setup We conduct our experiments in general chat and code generation. Our main setting is a single-turn general chat task, a common research testbed for offline RL fine-tuning in LLM alignment, and we add a code generation task where the model asked to generate code that passes unit tests given a description. We focus on comparing QRPO with policy fitting methods offline, and choose DPO, REBEL,, and SimPO as our baselines. Each captures fundamental means of fitting the solution of the RL fine-tuning objective: DPO for relative preferences, REBEL for the relative reward differencee, and QRPO for the absolute reward. We added SimPO to compare to methods that replace the canonical implicit reward with reward derived from inductive bias (length normalization). We select a comprehensive set of models, datasets, and distribution shift settings and sweep over the same comprehensive hyperparameter budget for each algorithm, resulting in 225 trained models for each algorithm. Additionally, we adopt a three-fold data split (train/validatiion/ess with three seeds to perform robust hyperparameter selection and report error bars. Our setting captures many of the experimental settings used in previous related work (Gao et al., 2024; Meng et al., 2024) and allows for broader comparisons. We present our experimental protocol in detail in Appendix M.
QRPO does not require a large pre-computation budget and scales with a larger budget to estimate the quantile reward To practically validate our choice estimate the quantile rewards instead of the partition function, we plot in Figure 3 QRPO trained with an increasing number of reference rewards generated in the pre-computation phase to estimate the quantile reward, for different values of the KL regularization hyperparameter 3, with lower 3 leading to less regularization and a larger change in the model. We distinguish two settings: the off-policy setting, where the training samples are close to the distribution of the reference model and initial training checkpoint which are the same, e.g., in our case the samples have been generated from the reference model, and the offline setting, where there is a high distribution shift between the training samples and the reference/initial model, e.g., collected from humans or in our case in Magpie-Air they have been generated from more performant model. In the off-policy case, the performance of QRPO scales with a larger number of reference rewards generated the pre-computation phase, which indeed gives a more precise signal to estimate the quantile reward of the training samples, as they lie in dense region of the estimated reference distribution, thanks to the weak distribution shift. However, in the offline case, with high-quality dataset, the performance of the trained model is very lightly impacted by varying the number reference rewards, and only a few reference rewards, as few as or 3. are enough to capture the information needed to train the model. Indeed, in this case with high distribution shift, pre-computing more reference rewards does not necessarily make the quantile signal more precise, as the rewards of the training samples lie in the sparse right tail of the estimated reference distribution.
7

---

Table 3: QRPO compared to baselines. We report original (Orig.) and length-controlled (LC) rewards (defined App. M) on a held-out test split with means and standard deviations over three seeds, and the AlpacaEval 2 original and length-controlled win rates (WR and LC) with their standard errors. Higher is better. We bold the highest numbers with intersecting means = one standard deviation after rounding. The best model for each algorithm is selected according a validation set from training with 9 hyperparameter combinations (learning rate and (9) across distribution shift settings, including the option to first SFT on the chosen answer (Initial vs. SFT-chosen) and then the option to fine-tune on set dataset completions (offline) or generate new completions from the initial checkpoint (off-policy) and annotate them (using the hest vs. the worst among 6 completions, or by ranking random pair of samples). We rank the preference pairs used by DPO and SimPO using the same reward model (ArmoRM) used by REBEL and QRPO, ensuring an equal potential signal source for all methods.
Magpie-Air Dataset
UltraFedde Feedback Dataset
Model
Method
A Reward
AlpueaEval 2
ArmoRM Reward
AlpucaEval
L.C
Orig.
1.C())
(5)
1.C
Orig.
LC(S)
WR (%)
Initial
0,1518 uome
0,1519 cam
18.1
10.8
0.1293 = came
18.1
10
SFT-chhsse
0.1648 g-00ss
0,1640 e aacc
23
24.
0.1219 1
0.1219
2.
Llama 8B
Tula 3 SFT
DPO
0.1904
0.1943 aa
47.7
53.7
0.1493
0.1491 co0e
20.4
34
SimPO
0.1976
0.1976 g ea
4g.2
4
0,1535
0.1539 cowe
27
30.2
REBEL
0,1889
0.1937
45.5
0,1488
0,1487
a9
87.8
QRPO
0. 0,2005
0.1972
50.6
56.5
0.1556
0,1504
49.8
65.6 6:
Initial
0.1398
0.1598 eo0s
27.6
21.6
0,1318 3axe
0,13119 cooe
21.1
SFT-chosen
0.1633
0,1633 e s3xs
26.2 1s
0.1251 a sa0s
0.1255
13.4
10
Mistral 7B
Instruct
DPO
0.1898 1000
0.1901 2000
49.8
0.1465
:: 1000
0.1480=
39.0
v0.2
SimPO REBEL
0.1879 nowes
0.18
24.
19.9
0.1478
200e
0.1472 =2 ow0e
38. 02
34.8
0.1884 2o00
0.1861=
-0.7
46.5 1
0.1466
2000
0.1415 = 01000
31.5
15
QRPO
0.1893 20000
0,1886 6000
44.4
46.6 13 0.1470
0.1469 - co00
38. 03
36.7 t.
Furthermore, by varying the KL regularization strength, we observe tradeoff between using the off-policy versus the offline regime. For a relatively large regularization (B = 0.1), the off- policy regime outperforms the offline regime with a relatively low number of reference rewards and continues to surpass it with more reference rewards. This aligns well with modern post-training pipelines (Grattafiori et al., 2024; Lambert et al., 2024) which use relatively large regularization to train on large datasets targeting multiple tasks and capabilities. and perform multiple iterations of policy fitting on newly sampled data from the latest model. The Llama 3 models (Grattafiori et al., 2024), for example, have been trained with DPO with 3 = 0.1 for several rounds, updating the reference model and sampling from it at each round; QRPO would be easily integrated into this off-policy pipeline to provide its top performance. Also note that in this case, all the previously generated reference completions can be reused as training samples to generate a growing training dataset. Yet, for a low regularization (3 = 0.003), the offline regime with high-quality offline data and minimal KL constraints yields the highest performance overall across 3s, and the off-policy regime requires large number of reference rewards to reach comparable performance. Although this setting combined with offline data would not provide stable results for large post-training pipelines due to a large test-time distribution shift (see Appendix A for a comprehensive discussion). it is the setting where research papers report their best results, as it eventually provides the best performance on narrow tasks, leveraging high variance combined with hyperparameter optimization. Most of the best models for all the algorithms we train also result from this low regularization offline setting, which at the same time allows QRPO provide strong results with very few reference rewards, but we minimize the impact from a selection bias by adopting a robust model selection protocol.
QRPO demonstrates top-tier performance Table 3 shows that QRPO achieves notably strong results in optimizing the offline RL fine-tuning objective for general conversational tasks, translating effectively to downstream evaluations on AlpacaEyval. QRPO models consistently rank among the top-performing models based on length-controlled metrics, suggesting that the approach also mitigates the well-known length bias issue, which we analyze in the next section. Notably, on the Magpie-Air waihh contains completions from Llama-8B- Insstrutt which is stronger than our initial models only use single reference reward in QRPO estimate the quantile reward, corroborating our results from the scaling analysis (section 4). Indeed, most of the algorithms trained on Magpie-Air obtained their best performance in the offline distribution shift setting, leveraging the quality of the dataset. For the UltraFeedback dataset, which is significantly lower quality than Magpie-Ai, and the initial models, we still only used 3 reference rewards for QRPO, and most of the algorithms obtained their best performance in the off-policy distribution shift setting, replacing the completions from the dataset with the reference model completions. Appendix N shows the hyperparameters and distribution shift settings for which each algorithm obtained its best model.
8

---

Table 2 also shows that when the task is canonically expressed with rewards instead of preferences, QRPO is more effective than the baselines in optimizing these rewards, which for baselines are converted to preferences and lose their absolute signal. In our case, it's the performance on LeetCode problems where the reward of solution to a problem is computed as the test-case pass rate over the test suite for the problem. This opens an avenue for novel applications using policy fitting methods for fine-tuning large language models. is also interesting to note that SimPO, with its inductive bias made for conversational abilities, falls significantly short in optimizing this coding reward.
QRPO with robust rewards is less prone to length bias than preference methods with labels from the same rewards We analyze the best model for each algorithm and setting for signs of length bias. Length bias (or correlation) is a spurious cor- relation between the output length and the per- formance of models resulting from RLHF and
500
DPO, p = 0.49
Implicit reward (normalized)
SimPO, p- 0.37
400
REBEL, p 00.14 QRPO, -0.16
DAA training (Dubois et al., 2024; Singhal et al., 2024; Park et al., 2024). It's undesired as it in- creases the verbosity and inference cost of the models without necessarily improving their un- biased performance. It is inherited from train- ing on (underspecified) preference labels. How- ever, the reward model ArmoRM (Wang et al.,
300
200
100
2024) we use in our experiments has been ex- plicitly trained to counteract this length bias in its scalar value and we show that QRPO and REBEL trained with this scalar value exhibit less length bias than DPO and SimPO which convert it to preferences. potentially inheriting the length bias back. Similarly to Park et al. (2024), we study the completion length versus the implicit rewards. As we have generated reference completions for each prompt to train QRPO, we in the position to leverage bet- ter signal for length bias than previous work. We take a per-prompt standardized completion length, which for each prompt and completion generated by the trained model, is the length of the completion standardized by the distribution of reference completions in that prompt (that we generated in the pre-computation phase). Hence, unlike prior work, our z-axis captures the length increase for each prompt independently. For implicit rewards, the same manner as Park et al. (2024), for each model (policy), we consider the reward parameterized by this policy according to the RL fine-tuning objective (Eq. 3 without Z, colloquially the DPO reward). This is the reward for which the trained policy optimal. Hence, if there strong length bias correlation in the reward, we can say that this is also a feature of the learned policy, as is optimal for that reward. Results are shown in Figure 4 and Appendix N. We observe the typical positive slope for DPO reported by Park et al. (2024), and surprisingly, we observe the same slope for SimPO. Although SimPO decreases length overall (the cloud is shifted to the left compared to other models), the length bias trend is still present. On the contrary, we observe that QRPO and REBEL exhibit much less length bias. We conjecture that this improvement is due to effective training with the robust ArmoRM reward signal, which for SimPO and DPO is lost in the binary preferences.
0
-2
2
Completion length (normalized)
Figure 4: Length bias Completion length difference from the initial checkpoint vs. the implicit reward test completions generated by the best Llama model trained on Magpic-Air for each algorithm. Implicit rewards (colloquially, DPO rewards) are induced by the policy, which is optimal for these rewards according to the RL fine-tuning objective. We report linear fit with a marker indicating the average completion length and Spearman rank correlation. SimPO reduces the average completion length compared to DPO (cloud shifted to the left), but its policy still exhibits as much length bias as the DPO policy. In contrast, QRPO and REBEL, which use the reward signal, do not exhibit a length bias trend.
5 Related work
Preference optimization A large body of research on policy fitting methods in RL fine-tuning has focused on developing preference- based algorithms to compete directly with DPO (Rafailov et al., 2023) by better learning from preferences. The list includes IPO (Azar et al., 2024), ORPO (Hong et al., 2024). CPO (Xu et al., 2024a), and SimPO (Meng et al., 2024). However, many of these methods deviate from optimizing the RL fine-tuning objective and attempt to better optimize human feedback for conversational abilities. With comprehensive results, SimPO positions itself the most competitive method and shows that a well-tuned DPO is still superior to most other methods. Hence, we choose compare to DPO to capture most of these methods, to SimPO to capture one competitive way to include inductive bias in the objective, and to REBEL to illustrate fundamentally different ways of optimizing the RL tun-ttnning objective: preferences vs. reward differences.
9

---

Offline alignment with a pointwise signal DRO (Pierre Harvey Richemond et al., 2024) formulates the same regression objective in Eq. 6, but proceeds to learn the partition function jointly with the policy. Although the approach is close to our motivation and derivation, DRO no longer benefits from the simplicity and stability of policy fitting methods, as it learns a policy and value function through joint optimization. In this work, we evaluate simple policy fitting methods and thus do not compare against DRO (which also does not have an open implementation). More recently, the same objective has also been used in SPO (Cohen et al., 2025), AGRO (Tang et al., 2025), and TBA (Bartoldson et al., 2025) which like DRO get both the policy and the partition function from Eq, 5 but instead of learning the partition function they propose different ways of estimating it. These works mostly focus on semi-online training and do not demonstrate competitive performance in the offline regime. Alternatively, KTO (Ethayarajhh et al., 2024) drops the partition function in the KL-regularized optimal reward (Eq. 3) and optimizes for human utility based on prospect theory (Kahneman & Tversky, 2013) instead of preferences, making the partition function irrelevant and the objective optimizable with pointwise binary human feedback. KTT could be considered as a baseline using a pointwise signal; but, since it excessively diverges from the RL fine-tuning objective, and Meng et al. (2024) show that empirically, using similar benchmarks to us, we do not KTO consider as a baseline.
Quantile rewards The cumulative density function of the rewards under the reference policy has been used in both the vBoN (Amini et al., 2025) and BOND (Sessa et al., 2025) algorithms to derive the distribution of the best-of-N (BoN) policy and identify the BoN policy as induced by optimizing the log-quantile rewards in the RL fine-tuning objective. However, both derivations focus specifically on the event defining the BoN probability and do not stem from the solution to the KL-regularized RL objective, which in these works remains intractable due to the partition function. QRPO instead uses the quantile reward as a universal solution to derive the partition function and the expression of the induced policy. In fact, we have the BoN policy log-quantile transformation. Another difference with our work is that we opt for a regression loss rather than a distribution matching loss. Gulcchre et al. (2023, ReST) also experiment with SFT loss on the best off- aff-policy data according to a quantile reward threshold. We believe that comparing QRPO, BOND, vBoN, and ReST can be made in the scope of the more targeted open question of studying the BoN distillation problem, but is out of the scope of the claims made in our work.
special case of our framework using the
Offline & off-policy policy improvement There is growing body of research attempting to make policy improvement methods work off-policy, such as TOPR (Roux et al., 2025), which uses truncated importance sampling ratios, and ReMix (Liang et al., 2025), which combines regularization techniques for data reuse. This is promising direction, confirming that QRPO tackles an important gap in the literature to optimize pointwise absolute rewards with data from other distributions.
6 Conclusion and discussion
We have presented Quantile Reward Policy Optimization (QRPO), a novel RL fine-tuning algorithm that uses quantile rewards derive tractable expression of the optimal solution the KL-regularized RL objective with its partition function, and fits it with simple regression on individual samples; therefore, directly using the pointwise absolute reward of samples instead of relying on preferences. We have shown that QRPO provides strong empirical results on chat and coding tasks, scales with the pre-compuaation budget, and less prone to length bias. We believe that QRPO unlocks a new avenue for policy fitting methods, which can combine any data distribution during training, benefit from offline scaling, and now use the signal from absolute pointwise rewards.
Limitations and open questions We have proposed QRPO as framework and shown that transfor- mations can be applied on top of the quantile to reshape it into the desired form while still maintaining tractable partition function. However, we have not explored these transformations empirically. We encourage future work to study them and provide further understanding in Appendix B. A notable limitation of the quantile reward is that it is based the reference policy, and when the model improves drastically beyond the reference policy, the quantiles become less informative. Still, this can be overcome by performing iterative training and updating the reference policy, as is done in many state-of-thed -ft-----ttt--tttee open-source post-training pipelines. Finally, QRPO and policy fitting methods in general can leverage any distribution, including online data, but we only focused on offline data to compare with other policy fitting methods. An open question is the impact of online data, potentially in combination with offline and off- off-policy data, and comparison to policy improvement methods such as GRPO, which rely critically on online data.
10

---

Acknowledgments and Disclosure of Funding
This work was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a10 Alps. We are grateful to Mikhail Pavlov for insightful comments on the theoretical aspects of the project, Mikhail Terekhov for discussions on the project theory and experiments, for suggestions on datasets for the coding experiment, and to Ivan Pavlov and Juan Garcia Giraldo for support with some evaluations. We thank the CSCS and EPFL SCITAS staff for discussions on infrastructure engineering. Finally, we thank Karin Gétaz for the administrative support provided within EPFL.
Anja Surina, Mikhail Terekhov, and Roman Machacek
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12248- 12267, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/202444 1on..666...............2.221 662.. URL https: bitiap::/aacaattiii /aalaaaaaaannooo..... org org/2024.coo acl- loog..nn...... 662/.2 2
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 41
1-15, 2022. doi: 10.1
Afra Amini, Tim Vieira, Elliott Ash, and Ryan Cotterell. Variational best-of-n alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview wtt.:///oorumm.ddd99//3/3rrrr 10
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024. 4.9
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0520 2022. 2
Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-ave aarree language model alignment. arXiv preprint 2rXiv:2412.197999 2024. 6
Brian R Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando- Ceron, Yoshua Bengio, and Bhavya Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable 1lm post-training. arXiv preprint arXiv:2503. 18929, 2025. 10
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952. 4
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877-1901, 2020. 2
Taco Cohen, David W Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off off-policy rl for sequence models, arXiv preprint arXiv:2503.0544 00543,, 2025. 10
11

---

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He. Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie. Yankai Lin, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting Language Models with Scaled Al Feedback, July 2024. URL http::ttttx:iii/ http:///aaxiv..o/ab//////// org/abs/ 2310700777..00 arXiv:2310.01377 [es]. 3, 3,37,39 39
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 1xXii:22044044755. 2024. 9, 3,37,39, 43
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. 10
Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. Advances in Neural Information Processing Systems, 37:52354-52400, 2024. 2, 4, 7, 39,9,, 42
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: / wwt./ooo///h/// com//ugggigg/ac////ecooooooooooo acc.a//cccccccceeattttoaa 2022. 41
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self- self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023. 10
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501.12944 2025. 2
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792, 2024. 20
Jiwoo Hong, Noah Lee, and James Thome Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11 11170-11189, 2024. 9
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pp. 1645-1654. PMLR, 2017. 2, 3
Fengqing Jiang. Identifying and mitigating vulnerabilities in I1m- llm-integratte applications. Master's thesis, University of Washington, 2024. 37, 38
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part 1, pp. 99-127. World Scientific, 2013. 10
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 41
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint a4Xiv:24/1./551 1.44/1./5.00 15/124, 2024. 2, 3, 8, 37, 38
Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang, Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, et al. Bridging offline and online reinforcement learning for 1mss. arXiv preprint ar 250.2256...199, 2025. 20
12

---

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of messtruction-follwiing models. httts://githhn www.//tss-tlaa/////ooooo http:/laaalppcaaaeeeaeeeeeee eval,
5 5 2023. 3, 43, 44
Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model. arXiv preprini NNNv:2507.00822 2025. 10
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505 24864, 2025. 20
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 37:124198-124235, 2024. 7, 9, 10, 39, 42,
Skander Moalla. Python Machine Learning Research Template, 2025. URI. htt.httts://gtthbb.co com/ wttp:/ppttoooooleeeeeeeeeeeeeeee
38
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https: // //arxiv... org/ aaaaabb//222 13228. 5
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19/59, 2024. 9
Pierre Harvey Richemond. Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline Regularised Reinforcement Learning for Large Language Models Alignment, May 2024. URL http: www.ii.wgg.aaaaaa.aao. 2200g//////445..o. 19107. arXiv:2405.19 19100 [cs].
10
Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, and Bowen Zhou. Online dpo: Online direct preference optimization with fast-slow chasing. arXiv preprint arXiv:2406.05534 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023. 2, 3, 4,
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejma, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37:122207-12222.. 2024. 2, 44
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unin- tentional unalignment: Likelihood displacement in direct preference optimization. In International Conference on Learning Representations, 2025. 5
Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Lauut Sándor Toth, and Sam Work. Ta- pered off-po reinforce: Stable and efficient reinforcement learning for lmss. arXiv preprint
arXiv:2503. 142866 2025. 10
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL htt::t//p///tt orgg/ba////// 06347. 47Xiv:1707.06347 [cs]. 2,
13

---

Pier Giuseppe Sessa, Robert Dadashi-Tazess Tazehozi, Leonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Rame, Bobak Shahriari, Sarah Perrin, Abram L. Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos Garea, Amélie Héliou, Aliaksei Severyn, Matthew Hoffman, Nikola Momchev, and Olivier Bachem. BOND: Align- ing LLMs with best-of-n distillation. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttpe////oppeeeeeeeeeeeee wwwpenrreieee.neefoou//iiiiie
www.:eeetfeooo mew//ooumm///ee fid=otxXMiSufff 10
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024a. 2
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024b. 2.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. In First Conference on Language Modeling, 2024. 9
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020. 2,
Yunhao Tang, Taco Cohen, David W Zhang, Michal Valko, and Rémi Munos. Rl-finetuning llms from on-and off-policy data with single algorithm. arXiv preprint arXiv:2503. 2025. 10
Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul. Younes Belkada. Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. First Conference on Language Modeling, 2024. URL https: atttps://ppenre //ppnr/ve////eeee////// net/ forum?id=aKk44v266.. 39
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https::/t //gi thub. www.///giiggaac///eeeeoooooooeee trl, 2020. 38, 41, 43
Haoxiang Wang. Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 10592, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18613/1 1/2024. sonddnns-emllpp 2024. fwwdiiiinnnnniinnn
URL https: hatp::://aaaaahhlloggooo org/
3, 9, 37, 39
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong.
Complementing ratings with preferences. In The Thirteenth International Conference on Learning Representations, 2025. 3
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45. Online, October 2020. Association for Com- putational Linguistics. URL https: / www. aclweb. org/ hesp//nnoooggy//// /2000. emnlp-demos 6. 44
Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint 2504 14655, 2025. 3, 37, 39
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of
LLM performance in machine translation. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the
14

---

41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 25204-55224. PMLR, 21-27 Jul 2024a. URL https://pproceeding.. mlr. A.r.ppress/v23b/xxuess
html. 9
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing, October 2024b. URL http: http:://aaxxvvvveeeee wrtg////2/////6666// 08464. mrXiv:2406.08466
[cs]. 3, 37, 39, 41,44 44
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025. 20
15

---

Appendices
Contents
1 Introduction
2 Preliminaries
3 Quantile Regression Policy Optimization (QRPO)
4 Experiments
5 Related work
Conclusion and discussion
10
A QRPO loss and gradient interpretation
17
Guidelines for using QRPO with quantile transformation functions
21
C Statistical Consistency and Convergence Guarantees
23
The partition function moment generating function at 1/3
25
E Distribution of the quantile reward R.
26
Partition function for the quantile reward R. and the generalized transformed reward
27
Sufficient conditions for the function in the reward transformation for the existence of finite partition function
28
H Noise tradeoff in the calibrated target: estimating the partition function estimating the quantile reward
29
I Noise amplitude in the partition function estimate for the initial reward
30
J Noise amplitude in the transformed reward estimate
31
K Equivalence of all quantile transformation functions strietly increasing and upper- bounded on the interval [0, 1]
32
Asymptotics of the partition function / for the family of quantile transformation func- tions strictly increasing and upper-bounded on the interval [0, 1]
35
M Detailed experimental protocol
37
N Detailed empirical results
48
16

---

A QRPO loss and gradient interpretation
0.7
Hlvrr
Ret
0.6
1.0
_8
0.5
Densily
0.001
0.4
0.6
0.3
0.2
7746
0.1
0.2
2396
00
2
0.0
-2
0
Reward
Reward R
Figure 5: Distribution of the initial reward R for given prompt under the reference (TTref) and optimal (x* policy distributions for different values of 3. The optimal policy is maximizing the RL fine-tuning objective (Equation 1) with quantile reward Ry, In this plot, the reference reward distribution is assumed be Gaussian, which can serve as good example for rewards obtained from a reward model in general chat task. With this assumption, we can compute both the reference and optimal reward distributions analytically. Refer to the final paragraph for the derivation. Left: Gradient update direction for samples with different reward values. The quantile reward R. = / log Zu corresponds the reward the intersection point of the densities. This value plays the role a threshold, which separates the samples with rewards below the threshold have their probability decreased, and samples with rewards above the threshold that should have their probability increased. Right: Position of the optimal policy reward distribution for different values of B. A smaller B leads a larger target distribution shift, resulting in gradient that decreases the probability of the majority of samples around the reference policy (see left plot for the intuition).
should
(Ro(o.)) Blog Zg Blog To x))
27
The QRPO loss has the form:
LQRPO
EE 2.
Tree (y x))
with
Zg B
exp
-11
or with an additional transformation the quantile reward R = J(R.g) and its corresponding 2 as described in Equation 13 and Appendix B.
The loss has a simple and intuitive meaning of pushing the probability of each sample in the training dataset (offline online) towards the probabilities of the optimal policy that maximizes the RL fine-tuning objective (Equation 1) with the transformed reward R. (or R). Writing the model updates given by the QRPO loss gradient and highlighting terms in the equation we get:
[[1 [1 - log Zg
log Toly (x) Trer (1) x))
-Vooouo 2/3 E
Vo log Toly x))
calibrated target
(14)
To minimize the loss, starting from an initial checkpoint equal to the reference policy, the update increases (respectively decreases) the policy's ' probability for samples with positive (respectively negative) calibrated target. This adjustment continues from an initial log-ratio of zero until the log-ratio aligns with the calibrated target.
In light of known issues with existing policy fitting methods such their tendency decrease the probabilities of both chosen and rejected samples (as seen in DPO-like approaches)-it is particularly interesting analyze QRPO from the same perspective. As shown above, the start of training, the QRPO update increases the probability of a sample only if its calibrated target is positive. Since the term 33 log Zg constant across all samples (i.e., independent of x and y as the quantile reward normalizes the signal in each prompt), it acts as a threshold: sample's reward R. (z. y) must exceed this value for the probability to increase. For example:
with 0.1, the threshold is 3 log Zg s 0.77.
with 0.01, the threshold is z 0.95,
and with 3 0.001. it reaches 0.99.
17

---

log-ratio of completions (validation
lag-ratio rejected completions (validaatoon set)
40.
-1.
Figure 6: log Tee (s12) dynamics for chosen and rejected responses when training with QRPO on a preference
ag ()la))
dataset in the offline regime with 3 = O.I. We plot the average completion log-ratio over all the prompts in the
evaluation set. log-ratios start 0 when the trained policy is initialized Iy xx) nr.. (y I). and during training become positive if ITe (y) | 2r) > / (y x) and negative if ma(y x) Teee (y/ x). desired, we observe that they become positive for chosen completions and negative for rejected completion. Indeed, with
g = 0.1 the quantile threshold for samples to incrcase their probabilities is at R2 = 0.77 and in this dataset chosen samples have average R2 of approximately 0.91, while rejected samples have average of around 0.45. The initial decrease of the chosen log-ratios be attributed generalization caused by the gradient being larger for rejected completions, until clearer separation between them is learned by the model.
Recalling that R2 corresponds to the CDF value of the reference policy's reward distribution (Defini- tion 9), we can interpret these thresholds in terms of quantiles. Specifically, if the training data points are sampled from the reference policy (off- policy data):
for 3 = 0.1, only the top 23% of the samples will have their probabilities increased (see Figure 5 Left),
for 3 = 0.01, only the top 5%,
and for 3 = 0.001, only the top 1%
A similar pattern holds in the offline case: for sample from offline dataset to be favored (i.e., to have its probability increased), its reward must lie within the top-performing percentile of completions under the reference policy, with the required percentile depending the value of [..
To better understand why only a small fraction of samples see their probabilities increased, is helpful to visualize the reward distribution and examine the position of the optimal policy distribution for different values of B. Figure 5 illustrates how the optimal policy distribution shifts as / changes and indicates the update direction for samples based on their rewards. We can see that for small values of 9, only a very small subset of samples from the reference policy will have their probabilities increased. Consequently, when training with off- off-policy data (i.c., samples drawn from the reference policy), it's normal to observe an overall decrease in the probabilities of training samples except when using very large B. In the offline setting, sample's 's probability will increase if it has sufficiently high reward, specifically if it lies in the region where the optimal policy places a high probability mass.
We observe exactly this behavior for sample probabilities in our experiments. In the off- off-ppolcy regime, we consistently see an overall decrease in the probabilities of training samples for all tested values of 3 [0.0001, 0.1]. In the offline regime, we use preference dataset constructed for preference training (e-g.. for DPO), which includes "chosen" and "rejected" samples for each prompt. Even though QRPO does not require preference pairs, we can use this separation to track sample probabilities dynamics for "better" and "worse" examples. In this dataset, chosen samples have an average R, of approximately 0.91, while rejected samples have an average of around 0.45, We observe a decrease in probabilities for both chosen and rejected samples when 3 E [0.0001, 0.01]. However, for 3 0.1, we see increase in the probabilities of chosen samples and a decrease for rejected ones (see Figure 6). These results align precisely with the R. = 3 log Z threshold values discussed above.
is important to highlight that even though training with a small 33 results updates that predomi- nantly decrease the probabilities of training samples, this regime can still lead to effective reward optimization. Empirically, we observe that due to the policy's 's generalization capacity, reducing the probability of lower-reward samples more than that of higher-reward ones implicitly increases the
18

---

probability assigned to even better completions. Notably, our best results on the general chat task were achieved under such "aggressive" training regime with 5 0.0003. However, this strategy relies entirely on implicit generalization to shift probability mass toward higher-reward regions, without any explicit control over where that mass moves; thus, potentially resulting in inaccurate policy updates. In practice, we observed signs of such inaccuracies when training with small (3: while the model initially improves rapidly, it later undergoes sharp degradation, likely due to the cumulative effect of these uncontrolled updates disrupting performance.
While this form of low-regularization training may work well for single-task settings (as in this and most other research papers), where the input domain is narrow and potential side effects of inaccurate policy updates on other domains go unnoticed, becomes risky in broader contexts. In large-scale post-training pipelines aimed at improving multi-task performance, such uncontrolled drift may degrade results on tasks not represented in the training data. For this reason, we consider the "aggressive" small-3 training regime unsuitable for scenarios where robust generalization across tasks is critical. Conversely, training with a larger 3 constrains the optimal policy to a lower reward region compared to small-B regimes, potentially limiting overall performance. However, large-scale post-training pipelines typically mitigate this limitation by performing a few iterations of training with relatively high 8, interleaved with collecting fresh on-policy data. This iterative approach enables gradual improvement of the policy while maintaining explicit control over probability mass drift (by leveraging on-policy data for the intermediate checkpoints), ultimately allowing the model to approach the performance levels achievable with small-3 training, but in a safer and more robust manner. It can be shown theoretically that performing NN iterations of training with 3 while updating the reference policy to the latest policy between iterations leads to the same optimal policy as training in single stage with 3/N. Indeed, to get the optimal policy after the second iteration, we can the closed-form solution (Equation 2) for the optimal policy and replace the reference policy with the optimal policy from the previous step:
1
I (3 (
Ti2) (y x)) =
T (y exp
(z)
1
1
(FR(O.))) exp (1 ( R(
Z (x) Z(as(to) Tyes xx)
1
2 (2 R(a.v))
(z) Fres 1)) exp
Eventually, if one does N iterations with 3 the final optimal policy is
m(N)- = Z(x) Tres (v x) exp
N (
R(z..))
which equals the optimal policy in the closed-form solution for B/N, providing a principled explana- tion for why large-3 iterative schemes can reach similar final performance to single-stage small-3 training, but in a more stable manner.
Interpretation of the gradient in the on-policy case Observe that in the derivation of the gradient in Equation 14, we pulled the derivative through the expectation without taking into account that y can be online (as it would impact the gradient). Indeed, in the case of using online data in QRPO, we do not need take a derivative of the expectation over y ~ To( x)) like it is usually done in RL as we want the equality in Equation to hold for all available y, whereas taking a derivative of the expectation over y will reduce the support of its distribution to the cases where the equality in Equation 5 is easy to achieve. Hence, for the online case, we have the same gradient expression, but we can interpret it differently, as now the rewards are changing with the online sampling from the policy:
Blg (e ()) Tre (y) xx)
-VoLRRO 2/3
E aND yo 1))
Rg v)
B log Zg
Vo log To x)
reword with KT. penatty
Notice that it can be seen as
standard policy gradient with an advantage made from a reward that includes a KL penalty
( [RR((,,)) - Blog (a()))) [ - 3 log Zg)
19

---

term, typically included in the reward in online RL methods, and baseline term. One may argue that the baseline term / log Z is a constant, thus, may not be a good substitute for the baseline term which is typically taken as the average performance of the training policy for each prompt, such as the critic for PPÔ and the Monte-Carlo average return for GRPO. However, the baseline signal that QRPO brings is in two folds: the first fold already comes from the transformation to a quantile reward, which re-centers with respect to the performance of a reference policy and normalizes its variance; the second fold comes with 3 log Zn which is subtracted to reflect the difference to the optimal policy,
Indeed, we show below that 3 log Zu 3 log (z) Byy=x*(-la) (z, v) 3 log æ--イ(リla)
="(ul=) is
the average performance of the optimal policy with the reward including the KL penalty. Thus, 3 log 4g is precise baseline estimate when the model approaches the optimal policy.
π (v x)
From Equation 5: Vy,
R(x,1) 3 log Z(x)
3 log
Freef (y) x))
B log ((x) R(xx v) - 3 log x T' σ)
Kres (y x))
7"( x))
E
B log ((x)
E gra (-la)
R(x,8) v) - -Blog
y~aa"(110)
rre (y [r)
π (y xx)
4
Blog ((x)
E
R(x,y) - -Blog
yoar (-Ix)
Tref f (y [r)
π (3 x) Tref (y) x))
And when using the reward R.
B log (z)
=
E gur (+1x)
R2 - 8 log
On-policy training further improves QRPO
performance It is well established that on- policy training improves the performance of pol- icy optimization methods (Lanchantin et al., 2025; Guo et al., 2024), as it allows for better control over output probability mass drift. Although on- policy training is beyond the main scope of this work, we nevertheless verify that QRPO also ben- efits from this regime. Specifically, we conduct ab- lations where online data is incorporated alongside offline data during QRPO training on the Magpie- Air dataset. This results in 26% greater reward improvement compared to training with offline data alone (see Figure 7), highlighting the strong potential of QRPO in on-policy settings.
Test rewards
QUPO oftina criine
0A20 offinne data
0.170
6.130
6.1220
0:225
30
150
150
200
252
300
392
trainiglobal_sn Step
Figure 7: Performance of QRPO under two training regimes: using only offline completions and using mix of offline and online completions.
Dynamic-reference KL for stable and flexible QRPO QRPO inherently optimizes KL- regularized reinforcement learning objective (Eq. 1). While there is ongoing debate about the necessity of explicit KL penalties, particularly in the context of verifiable rewards (Yu et al., 2025), recent work presents a more nuanced perspective: Liu et al. (2025) demonstrate that combining KL regularization with periodic updates of the reference policy recent checkpoints both stabilizes training and prevents stagnation, thereby enabling continued improvement during prolonged RL. Consequently, we do not view the inclusion of KL regularization as a limitation of QRPO; rather, we recommend adopting a dynamic-reference approach, where the reference policy is periodically updated to facilitate further reward optimization. Moreover, this strategy helps prevent saturation of the training signal when the trained policy approaches the top quantile, ensuring the quantile reward remains informative throughout training.
Figure 5: Deriving the reward distribution for the optimal policy analytically To derive the distribution of the initial reward R(x, y) under the optimal policy when optimizing the quantile reward R. (It y), we start from the closed-form solution in Equation 2:
1 Z, Trer(y x)) exp ( (R)(o..))
T (y x:) =
(15)
20

---

Next, we sum both sides over all completions corresponding to a given reward r = Rxx, y) for a particular prompt obtain the probability P a) of observing reward under the optimal policy:
P (r 2)
E
(y
E
1 Zn
Tre (v) x) exp (+R(((0)1))
y1 R(
31 R(z.))=r
Σ
Tres (1/ (1)x)))) (2) exp Fres(asr))
Zg
9: ((..))
exp
{ (Frer(es)))
Σ
Tres(y x))
Zg
B
R(z,a)=r
= Z, exp (3Free(aar))
Pref (r 2)),
where Pref(e (- æ) denotes the reward distribution induced by y Trafl- x). We used the definition of the quantile reward from Equation 9, namely (x, y) (rr R(x, v)) which is also equal to Fret (2, in the conditioned sum, to express the exponential term as a function of r.
In case
approximate the reward with a continuous distribution (e.g., in the case of using continuous reward model), we can use the probability densities Prof(: x) and p" (- x) to express the reference and the optimal policy reward distributions:
(r x) = Z, Pres (r 1)) exp (3fresleer))
(16)
Hence, assuming that Pret(- a)) is standard normal we can illustrate the shift in reward distribution of the optimal policy relative to that of the reference policy in Figure 5. This figure should interpreted qualitatively, as the actual distribution P.e((: æ)) can be arbitrary in real-world scenarios.
B Guidelines for using QRPO with quantile transformation functions
Table 4: Closed-form partition functions for common reward transformations. erfi is the imaginary emorr function and CDF A/o.1) the inverse CDF of the standard normal distribution.
Transformation f(t)
Partition function (3)
B(e1/3 -
B
log
B + 1
VT 3
t²
erfi erfi(1 (1/(3)))
Vi.
23 B + (1 33 2/1
CDF !
No(01) (t)
exp( 2/²
I + ((o.1)
exp 5 +
x² 2/2
The generalized transformed reward for QRPO has the following expression (Equation 13):
R( y) f (R,((r.)))
In this section, we discuss the common questions that need be addressed when using custom quantile transformation function f, notably
21

---

how to choose it,
how to compute its associated partition function, and
how to choose a suitable scale for the hyperparameter |3 to balance it.
The impact of quantile transformation function The optimal solution for the -ll-regulrrzzed RL fine-tuning objective (Eq. 2) depends on the shape of the reward distribution; therefore, the primary function of quantile transformation f is modify the transformed reward distribution to have some useful properties. One can easily obtain any desired distribution by using its inverse CDF as a quantile transformation function (since for any distribution Y. applying its inverse CDF to uniform sample, Uniform(0 [0, 1], results in CDFy (z) ~ Y). For example, one might wish the reward distribution to be Gaussian with mean jt and variance o². However, after transforming it into quantile reward Ry. the resulting reward is uniformly distributed. In this case, the following transformation can be applied to recover the desired distribution: R = f(R.,) = a CDF ((..1) (Re) + μ..
Choosing a quantile transformation function Generally, it natural to require a transformation function be monotonically increasing ensure a meaningful transformed reward. This guarantees that higher original rewards correspond to higher transformed rewards, preserving the reward signal's intended structure. Furthermore, there only two requirements for the transformation function f to be suitable for QRPO:
1. it must be defined on the interval [0, 1]. and
2. its corresponding partition function must be finite.
The first condition is straightforward. and we provide some guidelines to find functions that meet the second condition.
Using moment generating functions One possible approach to address the finiteness condition
is to get insights from the distribution of transformed rewards R. The partition function (xx is simply a moment generating function (MGF) for the distribution of transformed rewards under the reference model: Z(x) IE+ [et]] Mz(t), where 1.2 R(x, y) for y Tref( 2) (see Section D for the derivation). Thus, check the finiteness of (z), one derive the distribution of the transformed rewards for completions sampled from the reference model, and then check the value of the corresponding MGF for t
Sufficient conditions There is a sufficient condition for the finiteness of 2: if the transformation function is pouerebounded on the interval [0, 1]. then the corresponding partition function is guaranteed to be finite (proof in Appendix G). However, it is very important to note that this condition not necessary, i.e., there exist some unbounded functions (we consider the upper bound) such that the corresponding Z is finite. An important example of such an unbounded function is the inverse CDF for the normal distribution (the transformation we considered above to obtain a Gaussian distribution of transformed rewards).
Computing a partition function After applying a quantile transformation the initial reward values, the distribution of the resulting quantile reward Rg is known; therefore, the corresponding partition function Z, can be calculated analytically. Moreover, starting from a quantile reward Rg one can apply any additional transformation fRR,) to it while still preserving the ability to compute the corresponding partition function: the partition function corresponding to a specific transformation function f can be computed as simple integral (derivation in Appendix F):
2(0) - exp (tro)a
2(x)
If the integral cannot be computed analytically, one can compute it numerically for each specific value of 3 that plans to during the training as the value of (xx) is constant. We provide table with some transformation functions and their corresponding partition function values
22

---

Scale of the 3 hyperparameter 3 is one of the most important hyperparameters when optimizing a KL-constrained RL objective (see Objective 1), and like other alignment methods, QRPO requires this parameter to be properly tuned. As can be seen in Objective 1, if the reward is multiplied by factor a, then 3 should also multiplied by a to preserve the
ratio.
A feature of the quantile reward is that it normalizes the reward between [O, 1], making 3 independent of the original reward scale and transferable across domains. However, additional transformation on top of the quantile can change the reward scale again. Thus, if transformation function changes the typical reward scale by some factor, then, roughly speaking, / should scaled by the same factor. Hence, the typical range for 3 depends on the transformation applied to the quantile. This is rather a rule of thumb, as the shape of the reward can also matter. This aspect important to keep mind when trying different transformation functions.
We make quantile transformation function f is upper-bounded on the interval [0, 1] (see Appendix K for the intuition). The key factor determining the relative scale of the transformed reward is the left derivative of the quantile transformation function f at æ = 1. Therefore, when comparing two upper-bounded quantile transformation functions, the ratio between their respective / values should be approximately equal to the ratio of their left derivatives at zero, move higher orders until you find the first non-zerv left derivative common to both functions. Let k be that order. Then the appropriate 3-scaling factor is the ratio of their k-th left derivatives. If the first non-zero left derivative appears at different orders for the two functions, no single constant can scale 3 consistently.
rigorous observation regarding the correct scaling of B3 in the case where the
1. If both left first derivatives at æ = are equal to
C Statistical Consistency and Convergence Guarantees
In this section, we analyze the QRPO objective in Eq. (11) (and its practical variant Eq. (12)) through the lens of (i) exactness of the partition function, (ii) statistical properties of the empirical quantile reward, and (iii) optimization with stochastic gradients.
Notation. Write the calibrated target as s(x, v) == (I, v) - 3 log Zg and the model log-ratio as (y(a)) The population QRPO loss is L(0) B[(s(x,)) v) (0 v) y]}
go(a; y) B log (yle)
C.1 Exactness of the partition function
Assume that, for each fixed x, the reference reward distribution R(xr, y) for y Teall. xx) continuous (as with standard regression reward models). Then the quantile reward R,, y) = Fiet R(x, v)) is uniformly distributed on [0, 1] for every æ. Therefore the partition function analytically exact and z-independent:
7₂(x) = exp (2 [
1/3 1
31=B==B000 (d10-1))
(17)
Hence QRPO introduces no approximation or modeling bias through Zy.
When R takes a small number of values, R, is no longer uniform; however (z) = En Pk(22) (z) e/x/$ remains exactly computable as finite expectation over the atoms {tx} with masses (p)(F)} (see App. F). Thus Z, is still exact.
Remark (discrete rewards)
Practical constant used in Eq. (12) If one replaces / Bog Zg by $ B + 1 for stability, the calibration error is Ag 3 log Z, (8 log B + 1) 3 Blog(1 E 1/3) < 0. For 3 < 0.1. 10 -5, is negligible.
|Ag $ 1/8
C.2 Empirical quantile reward: unbiasedness and concentration
For each x, let ......, ... Un Tyer(- (- x)) be i.i.d. reference samples with rewards ra = R(z, 31). and define the empirical quantile estimator R, y) 1 Σi= 1fr: < R(x, y)}-
Unbiasedness For all (x, y). E R2 (z. y) xz = (x,
23

---

Uniform consistency and sinite-sample deviation By Glivenko-Cantelli, sup, (Pn,1Pa(z,r) Fre r) - co. Moreover, the Beettkky-KKifee-ooofoooooe inequality yields, for any E > 0, the following distribution-free bound:
Pr [2, v) Rg(2,8)1 >E {
s 2 2nc²
Expected error
Var (2, v) k.
(2, y) Rg(
I
1
E
R4
Ry
1
4n
2/n
Thus the only stochasticity introduced by QRPO zero-mean label noise on the target, with sub- Gaussian tails (from the exponential bound) and ((1/vn) rate.
C.3
Population argmin is unchanged by unbiased quantile noise
(z. v) Let [1x, y) = (x,
R, (x, v). so that (zE(z, u) xz y) = O. Define B(x, s(x,)))+ +) v).
[a(y)l) Then the population loss when where s(r, = (x, y) B log Zg and go v) log Towr(y
training against 3
[())= = E (3(z.)) go (z, w))
(()) + ((0)+BB[E) [E(r,
because [<(,,,)) n, vl - 90)] = O.
the
cross
term
vanishes:
[E(r,,)) v) (s(ar, v) go s))]
Oracle risk invariance The set of minimizers of Σ(0) equals that of L()). Unbiased quantile noise adds only 0-independent constant the population risk.
Gradient perturbation bound (per sample) gradient difference
With the same E(x, y) as above, the per-sample
V Vol 2 y) Vago (x,y). y).
If Vege v)ll S G (c.g., via gradient clipping), then for any > 0,
Pr (IV Voll < 2G x) 2 1 - 2e 2nc
Equivalently, with probability at least 1 - 6, vol V S 2G
log 2 5
2n
C.4 Convergence of (stochastic) gradient descent
The QRPO loss is nonconvex in 0, but the preceding results ensure that the population objective is unaffected by empirical-quantee noise. Assume (A1) the loss is lower-bounded and L-smooth in 0; (A2) stochastic gradients have bounded second moment; (A3) either the empirical quantiles are resampled independently across iterations (or n - oo so that & > 0). or one optimizes the fixed empirical objective with standard SGD. Then:
1. With resampling (or x)), SGD produces gradients that are unbiased for VL(() and converges to stationary point of the oracle loss L()) under standard step-size conditions
Σ, T Σ, ni
With fixed precomputed R. (Algorithm 5), SGD converges to stationary point of the empirical co the empirical objective converges
QRPO loss. By DKW and Civvenko-Cantellln uniformly to the oracle objective, so the stationary points approach those of L()).
Summary. (i) Zg is known in closed form (or as exact finite sum in the discrete case), so QRPO introduces no bias via the partition function; the practical constant in Eq. (12) differs from log Z, by a negligible 01/- shift. (ii) The empirical quantile reward is an unbiased estimator with exponential concentration e 2nn= and expected error (1///n). (iii) This induces only zero-mean label noise, leaving the population argmin unchanged; SGD converges to a stationary point of the same oracle objective under standard assumptions.
24

---

D The partition function is a moment generating function at t = 1/B
In this section, show how express partition function ((x) terms of a reward distribution for completions from the reference model. Assume that R(x, y) is the reward optimized in the objective I, it can be an initial reward or transformed one. Then, according to the definition of the partition function (Equation 2):
(xx)
Σ
Trefly x)) exp (epp(x R((
y
Σ
Σ
Tret(y x) exp R(x, v))
[Stratify the commation of by possible reward values. see discussion below regarding the validity of this reordering]
y: R(a.y)=r
Σ
(ye 2) exp (3r))
[Change Ria, y) have reward R.r,
y's
V: R(x,v)=r
Σ exp (r))
Teo xx)
[Pull exp (3r) ou of the inner zum it doesn't depend on y
y: ((r,))=r
Σ exp (/21)
Pr y
(R(x, v) =
[The sum becomes cumuulative probubility of y the roward my it's probbility obtaining the roward v]
Pre((1la)
Eexp( exp (r)
Pror(r xx)
eep (arr)rr = Mr 1= (3)
= Pew
[Use the definition of MGF]
Here we use notation (r(z) that represents probability of obtaining reward when sampling a completion y for a reference model Vyee(y a) and computing the corresponding reward R(x, y)-
In the last line, we applied definition of a moment generating function (MGF) Mx = Ex (exp (tX)].
Note on the validity of the summation reordering Although the sets of possible and r can be infinite, the reordering of the summation is legitimate for the following reasons. language model by definition can generate only finite-length strings, with generation halting with probability 1 (a property often referred as the righmess sets of possible and are infinite, they are countable. Moreover, since all terms in the sum are non-negative, the series converges absolutely, and the reordering terms therefore justified.
language model). Consequently, although the
We derived an expression for the partition function that depends only on the reward distribution and independent of the probability distribution over completions. This result greatly simplifies both the analysis and computation of the partition function. Specifically, if the reward distribution is known and the moment generating function (MGF) is finite the chosen value of (3, the partition function can always be computed analytically or numerically. Conversely, if the partition function is known as function of variable 3 and after the change of variables t }/2 is still defined on an open interval containing = 0, then the reward distribution is uniquely determined (E.g., using the inverse Laplace transform) and can be recovered analytically or numerically.
This leads to an important insight: practice, knowing the reward distribution implies knowing the partition function (provided it is finite), and vice versa. Therefore, it is not possible to obtain an analytical expression for the partition function if the reward distribution can be arbitrary. This insight is central to the QRPO method, which uses quantile transformation to ensure the reward distribution is known and thus the partition function becomes analytically tractable.
25

---

E Distribution of the quantile reward Rg
Let y Krof(: x)). We want show that the random variable
Pr Rg( y' y/~trer(((a)
{R(x, S R(x,y))
uniformly distributed on [0, 1] for y ~ Traf( x)) in case R(x, y) can be treated as continuous variable. This is well-known result; we include it for the completeness of the theoretical part.
The cumulative distribution function (CDF) of rewards under the reference policy (Eq. 8) is
Fref (z, r)
Pr
(R(,, sr).
Hence, the quantile reward (Eq. 9) also
Rg(x, (ze R(x, v)).
Proof intuition There is simple intuition to that (I, y) is uniformly distributed for y Trey(- I) by thinking about its cumulative distribution. We pick a value a E [o, 1] and ask what is the probability that (I, y) I but since (2, y) Firr(z, R(x, v)), this is equivalent to asking what the probability to cumulate a probability under the reference reward distribution, which is by definition
Formally Assume that, for the fixed input x, the map - Frer(r; r) is continuous and strictly increasing (i.e., R(x, y) has continuous distribution). For any [0,
(r,R,((2,))) Sa) = =a)==
R(x, y)) S sa)
= =P((R(s, v) & (r
Frot (I, (r,
=
which is the CDF of Uniform(0, 1). Consequently,
(I, y) Uniform 1) for y ~ Tyer( x).
As an illustration, we show in Figure empirical distribution of the quantile reward.
Distribution of the quantile reward Rg
1.0
0.0
Density
0.6
0.4
02
0.0
02
04
08
1.0
Rg(x y) for y marl (x]
Figure 8: Empirical distribution of the quantile reward (2, for completions generated from a reference model Tref-
26

---

Partition function for the quantile reward R and the generalized transformed reward R
Here we compute the partition function Z for the case of generalized transformed reward Rx, y) (I, y)) (as introduced in Equation 13). To get Zg for the quantile reward Rys one can just take f(t==t =t
We use the same notation as in Equation 9:
Rg(2, 9)
Pr of y'wotref (+1z) [R(x, 5²) R(x, = (2, R)),
R(z. v) (x,
for any function f defined the interval [0, 1].
Following the results obtained Appendix D, we can express the partition function in terms of the moment generating function (MGF) for the generalized transformed reward R(x, y) distribution for ~ (- r):
Ž = E Prer((t)) (ar)
[exp( (t1)
M₁ t
3 )
where, similar to Appendix D, we introduce Prot(: the probability distribution the generalized transformed reward R(x, y) induced by Tree(. (-
If one knows the distribution of the transformed reward, the partition function can be computed directly by evaluating its MGF at t = 1/B. For instance, in Appendix E, we show that the quantile reward R. (zz, y), when treated continuous variable, follows Uniform(0, 1) distribution. The MGF for Uniform(O, O0, 1) distribution is given by Mr, (t) {e* 1), and thus the corresponding partition function is
Zg - = Mr, (1= 11) (t= B(e) - 1)).
More generally, when rewards can be approximated by a continuous distribution Note that, generally speaking, when the reward can be treated as a continuous variable (such as in the case of a reward model or coding test-case pass rate), the distribution the generalized transformed reward R can, in principle, always be computed. This because the distribution of R. is known, and R. is defined as known function f of R. Moreover, it is not even necessary to explicitly derive this distribution in order to compute the partition function; instead, one can directly apply the change-of-variable formula for expectations (also known as the "Law of the Unconscious Statistician" (LOTUS)):
= Prette))))))))) eep
E, Tepp (f)(Ga)
exp (ap(t((t) dt.
We obtained I-D integral that can be computed for any transformation function f (either analytically numerically for some specific values of (8). However, note that QRPO requires this integral to be finite, thus introducing some basic restrictions for the transformation function f.
If the rewards cannot be well-approximated by continuous distribution, directly use the discrete distribution the rewards This situation typically arises when the reward distribution has only small number of possible values. In this case, the expression for the partition function cannot be computed in the same way as in the continuous case because we can no longer rely on the fact that the quantile reward R, follows uniform distribution. Consequently, the chain of reasoning used in the continuous setting-starting from the known distribution of R.p, then obtaining the distribution of R, and finally computing the exact expression for the partition function-brect down.
27

---

Instead, we need to explicitly approximate or estimate the distribution of rewards. Specifically, we need know the distribution of either the initial reward R, the quantile reward R, or the generalized transformed reward R. In practice, usually easiest to work with the distribution of the initial reward R.
The good news is that in such discrete settings with a small number of possible values, the distribution of the initial rewards can often be estimated much more easily and reliably, for example by simple empirical counting of occurrences the initial reward values {Pier(T; x)}; for each prompt by sampling from the reference model y Fref(: z), we can, similarly to the continuous case, apply the change-of-variin formula for expectations to compute the partition function:
finite sample. Once we have estimated probabilities for
7(x) B Pou((11)))
eep (1t))
E E
exp (exp(F((rer((aa))))
Prey x) exp ( E
( (I x) )
Here, Pret(: x) denotes the probability distribution of the initial reward R(x, y) induced by y Tre z), as introduced in Appendix D and similar to Prer(- x)) introduced earlier in this section; Pr x)) is an empirical estimate for Pref(- x);; Fres (z, r) is a CDF of initial rewards under the reference policy as introduced in Definition 8. K denotes the number of possible initial reward values.
The expression for the discrete case derived above is intended to be used in the case when the number of possible reward values is small (i.e. < 10). For other cases, the continuous version is likely to be decent approximation for the partition function, and is more preferable, since it does not require one to estimate the probabilities (re .r);-
G Sufficient conditions for the function in the reward transformation for the existence of a finite partition function
We derive sufficient condition on the function f in the proposed reward transformation (Eq. 13) that ensures the finiteness of the corresponding partition function. The partition function given by:
(x) Σ Tref (y ()) exp (z,
(
Recall that the partition function can be expressed as moment generating function for the distribution of the reward used in the objective I (see Appendix D for the derivation). Here we consider objective 1 with the generalized transformed reward R:
exp }:
E Bi
where (r 1)) denotes the probability distribution of the generalized transformed reward R(x, v) induced by the reference policy ~ Tres(- z).
By applying the change-of saeee-of--rirlaa variables formula for expectations (also known as the "Law of the Unconscious Statistician" (LOTUS)) we obtain:
2 (p-(tat(())) [exp [br]]] -E. [exp (F
= =Σ exp (FFee(((e, Piet(r 1x),
where (res(r .r) denotes the probability distribution of the initial reward R(x,, y) induced by the reference policy y Tres(: (: z).
28

---

Then we can derive a very simple sufficient criterion for the finite partition function existence:
-00== exp
expp(b (Fre(((( Proe(r (x) s Σ exp
max S((²)) Pret(r) xx) " ==
=ex
( max B (
f(u)) 0o,
if the function f is upper-bounded on the interval [0.
Other transformations The condition for transformation function to be upper-bounded is suf- ficient for a finite partition function existence, but is not necessary. Thus, one can also consider other transformation functions that are not upper-bounded on the interval [0, IJ. However, this would require a more careful consideration of the finiteness conditions of partition function.
H Noise tradeoff in the calibrated target: estimating the partition function vs. estimating the quantile reward
B 0.3
8-02 0.2
f 0.1
10a
107
Modified reward R₃
Modified reward Rg nnttlal reward
10
Modified reward Ra
Nolse Amplitude
Inittal reward
10'
Initial reward
107
10
1044
10
30-3
10a
10'
10-1
107
10-2
10a1
3.0
0.2
0.4
0.5 [[Rg(K,)] //]
0.5
0.0 0.8
1.0
0.0
0.2
0.4
0.6
1.0
51Rg(x,)) y)]
FR y1]
Figure 9: The noise magnitude in the regression objective in log-scale when the reward is left unchanged and (zx estimated directly (initial calibrated target) vs. when the quantile reward is estimated (modified calibrated target) and 7(x) is computed analytically. The z-axis represents the true quantile of the reward value relative to the reward distribution of a reference model. We use = 10 samples to estimate both the values of the empirical partition function and the transformed reward in this plot.
Noise amplitude and intractability of partition function estimation for the initial reward Direct regression methods described above require estimating partition function Z that is computationally very expensive. To support this empirical observation, we provide theoretical analysis of the noise amplitude in its empirical estimate, assuming that completion rewards for a given prompt follow a Gaussian distribution. The standard deviation of a logarithm of an empirical estimate of partition
a²
function 2 has the following expression for a large number of samples >> e (derivation in Appendix I):
B
sLd log (()) =
e
(18)
where n is the number of samples that use for the estimation of Z(x) for each prompt x, o² is variance of the reward R(z y) for y ~ (- x). and B is the parameter from the RL fine-tuning objective. The derived formula shows that, for example, when o = 1 and 3 0.1, approximately n 100 ~ 10 samples per prompt I are required to achieve a reasonable signal-to-nonl -o-is ratio in the estimated partition function that is an entirely intractable number. In fact, while the actual reward distribution may not be perfectly Gaussian in practice, any distribution with a right tail will face the same issue due the exp(3 ! R) term in the partition function expression. A good workaround would be to compute (1x) analytically, but this requires us to know the exact distribution of the reward. Hence, we need transformation for the reward that leads to known final reward distribution.
29

---

Noise amplitude in the regression objective We showed that the regression objective in Equation 6 suffers from high noise due the estimation of the partition function Z. By introducing a transformed reward, we have completely eliminated this noise in the partition function by using exact expression computed analytically. However, some noise still remains in the regression objective, now originating from the transformed reward R, (2r, y) itself, as it requires estimating a quantile.
To assess the impact of this change, we compare the noise level in the objective 6 after applying the transformation with the initial noise level. The standard deviation of log (ax) for the initial reward was previously shown in Equation 18. The standard deviation of R2 (z, y) is given by the following expression (proof in Appendix J):
J. B v)1 E (x, Vn
stdd (Ro (2,((,,)))) u))
Note that the standard deviation of the transformed reward is not constant and depends on the initial reward value through its exact quantile, E (x, y/]. Figure 9 compares the noise magnitude before and after the reward transformation. As shown, the noise in the initial reward is significantly higher, even for moderate values of B. This is particularly relevant given that typical values in RL fine-tuning methods are around 0.1 or lower.
This theoretical analysis demonstrates that the proposed approach of introducing reward transfor- mation reduces the noise in the objective function dramatically*. which enables the general approach of pointwise policy fitting to work well in practice.
I Noise amplitude in the partition function estimate for the initial reward
Here we compute the variance of the log (x) estimate for the general case of using the initial reward for the policy fitting regression objective. Recall that the partition function can be expressed as moment generating function for the distribution of the reward used in the objective (see Appendix D for the derivation):
exp }+
Z = Ey (-1z)
where (r | I)) denotes the probability distribution of the initial reward R(x, y) induced by the reference policy y ~ Tref(- x).
Therefore, since the exact distribution Pref(rias (x) is unknown, we cannot compute Z(x) analytically. In the remainder of this section, describe Monte Carlo approach for estimating log (xx). Specifically, we analyze the variance (noise amplitude) of this estimate to assess the feasibility of the approach.
We estimate the variance of log Z(x) by modeling the process of generating n completions from the reference model and then approximating [zz) with finite sum. To do this analytically, we assume that Prey x)) is Gaussian, but this does not reduce the generality our conclusions, since can be shown that any distribution with an infinite right tail has the same properties worse
We can also assume that (r(a)) Gaussian with zero mean since:
]]
n(xx)
1 exp 3
}.
log (xx) log E. M(tt)))))
exp
3"
3
+ log ((g,
so it is clear that the variance of estimated log (xx) does not depend on p(z).
Now suppose we have sampled n completions from a reference model and obtained n samples Ts ~ o2). Then exp
(1rV) Lognormal(.. (0, e² and
1 exp ( ]
1 exp 3 (r:))
284 2
var
e
'tt least for rewards that can be treated as continuous. We have derived QRPO for reward distributions with small number of possible reward values for completeness, but this is not the focus of our work, and the noise tradeoff in this case remains an open question.
30

---

Unfortunately, the distribution of the sum of Lognormal variables has no explicit expression. There- fore, we will operate with the variance.
Then the estimator (xx) = Ei= exp
(3r))
have the variance
var (2(x)) = var 1 Σ exp 3
var exp 3r1
-1 e
e
n²
The last step is to calculate the variance of log (z)) Unfortunately, since there is no explicit expression for the distribution of 2(x), we cannot rigorously compute the variance of log 2(r).
Furthermore, the knowledge of var ((x)) is also insufficient to rigorously calculate the variance of log (x). However, approximate it for large enough Indeed:
E
[(z)
2
std ((x))
22
32
then for e0 the distribution of (2x) is narrow and we linear approximation for log
at the point x = TE [(xx)]
e that is, log x 2 e x +
23 1. Then finally
o²
var (log ((x)) x var (e
(e= 25 ((x) 2,32
o²
1
var (xx)) = 12
-1)-
e 1
2
std 2(x))
V e
- 1.
J Noise amplitude in the transformed reward estimate
In this section, we calculate the variance for the proposed modified reward estimate (z, y). Suppose we have n completions yi from the reference model Tres(- x)) and the corresponding rewards Fref: R(x, y:). Then the estimate R.y y) is computed
1 Σ R n
Re v)
1(r & R(x,y)) y)3 var (1 fra S R(x, v)}) = Fre, R(z, y)) - Fres (z, R(x,
a Bernoulli random variable with the variance
where (x) a cumulative distribution function (CDF) for the distribution of the initial reward under the reference policy (see definition 8). Then
var (R, v)) = - Fref(( R(x,8))(1 - R(x,)))),
std (R, 1/) = /n V Frof R(,, Frof (x, R(x, ()))
Noticing that
R2, y)] = (x, R(x,x))
we can finally rewrite the std (R, y)) in the following way:
std (R, v)) = nn R yil (1 E (xg(2,))) v))).
31

---

K Equivalence of all quantile transformation functions strictly increasing and upper-bounded on the interval [0, 1]
In this section, we show that when /3 is sufficiently small and the initial reward R. can be approximated as continuous variable, the optimal policy for a transformed reward R(x, v) = f (R, y)) remains invariant under any choice of strictly increasing continuous transformation function f(t) defined on [0, with finite values and non-zero left derivative at / 1.3 Formally, as 3 decreases, the policy converges to the same solution regardless of which function f in this family is used.
We assume mild regularity conditions, specifically f [0, 1].
Recall that the optimal policy 2 has the form:
Z(x) Tres exp {} R(x, v)
1
(3 x)
where
R(x, y) = (R.(x,
7 Σ Tres(y (y x)) exp
( (((
We start with some standardization of the initial conditions.
K.1 Rescale and shift
Under our assumptions:
1(1) C1 x0, C1> -o,,
f2(11 C2 < ~0;
where f- denotes a left derivative.
Also since (tt is strictly increasing on the interval [0, 1] and its left derivative has non-zero value at t t= 1:
C₂ > 0.
First, note that any constant shift of the function f does not change the optimal policy x x² x)). Indeed, if there is shift f(t) f(t + then
exp ( (Rg(x.
a
( (R,(o,,)))
+ exp
exp
3
a 2(x),
Z(x) + exp
3
and the terms exp changing the optimal policy:
( cancel each other. Hence, we can do the following transformation without
f(t) + f(1) Ci.
obtaining the following initialization property w.l.o.g.:
S(1) =
Second, note that for any constant a, transformation f(t) - at together with 3 ag also does not change the optimal policy. Hence, we can simultancously transform f(t) and /3 in the following way without changing the problem formulation:
f(t) C2
8 C2
f(t) +
3
Functions with zero left derivative t correspond another equivalence class, More formally, can be shown that there exist classes of equivalence. The n-th class consists of functions whose first n left derivatives are all zero at t = 1 but have non-zero + left derivative at = 1. We make a derivation only for the case of i.e. non-zero first left derívative. Others be done similarly.
32

---

After this transformation, we obtain a useful initialization property that we will use in our proof w.l.o.g:
1-(1) =1.
Intuitively, this scaling argument means that two different reward transformation functions differ only in their scale near 1, one can compensate by adjusting 3. This rescaling ensures compara- bility of different transformation functions and does not change the essence of our invariance claim: although different scales of f near z 1 may alter the rate which the policy converges, they do not affect the limiting policy itself.
Throughout the following proof, whenever f does not already satisly f(1) = 0 or f_(() = 1, we
perform these shift and rescaling steps first to normalize its value and derivative at x
K.2
Main proof
To prove the invariance of the limiting optimal policy under any choice of the transformation function for small enough B we show that KL divergence between the optimal policies corresponding any two transformation functions f and g among the considered family of reward transformations decreases linearly with B - 0.
We consider the following optimal policies:
1 mÿ x.)
1
Tresly a)) exp
T (R,
Zz(xx)
1 Ty x)) Zg (x)
Tre x) exp
@ (Ry(
where
(x = Σ Tre (y] x)) exp
(R (R)(
3
1
(g(x)= = Σ Erees(W x)) x)exp
399 (Ri (2,
[z, y) Fres (z, R(x, v)),
where (x) a cumulative distribution function (CDF) for the distribution of the initial reward under the reference policy (see definition 8).
(TI( x) [.(. xx)
Zp(z] I Tre Tres x)) exp (#s (R., 8))
Σ 1
exp exp (gS (Ry (R.(( log
Zz(x) Tres(y 1))
2o(z) Tres(y x) exp ( (R, y))
Σ Σ
(( =r Zx(x) Tref exp
exp
[ (Fres(a,r))
1 (Fre((os)))
3 (Fre) + log Zz(x)
B
2
Σ (x(x)
1
exp (Fr(tr)l)))))
1
1 { (Fro
3 (2,)) + log Zg(x)
2f(x)
Σ Tres
(:R(a,))
Pres(r(a)
Σ
1
1 exp 1 ( (Fros(e,,))
{
3 (Fres(e, + log 2z(x)
2z(x)
7r(x))
x (res())a))
33

---

where (re r) denotes the probability distribution of the initial reward (zx, y) induced by the reference policy y ~ Tref(" x).
Recall that we require the distribution of the initial reward r to admit a continuous approximation (such in the case of reward model coding test-case pass rate). We denote the probability density function of this continuous approximation as Pref(r 2)). Then the derived sum can be turned into an integral:
D (3)( ()) (1
-/2 (z)x) exp (Fre(((r)))
{ (Fre((orr)) 3 (Fres(a,))) + log Zg(2x)
2z(x)
x *Pres(rla)d
-A exp (Fns(((r))
(Bos(e()))) 3 (Fros(E3 log Zz(a)
(x)
7y(x)
x ddFrs(Ez,7)
= - B (z)x)) exp (5)(())
Zz(x) Zz(x)
f g (z) + log
x de
Since we require f to be strictly increasing smooth function defined on [0, with finite value and non-zero left derivative at t = 1 we can apply Laplace's 's method for an endpoint maximum to compute the asymptotics terms of beta.
Here the Laplace method formulation for reference:
Theorem (Laplace's method for an endpoint maximum). Consider the Laplace integral
F()== = f: ((t) dt, 4+>
F(A)
Let 1 [a, bj be finite interval and assume that
1. [t) attains its maximum on [a, b] only at the right endpoint = b;
2. h, J E EC((a,))
3. h, f.J E C a neighbourhood of = b and f'(b) + O.
Then, as X 00,
F()) ~ A1(
de x 1
k=0
with coefficients
dx = (- Radda
h(t)
dt
f'(t)
t==b
Moreover, this expansion may be differentiated with respect to X arbitrarily many times.
In our case we have
8² A(t) (x) (1()) (t)) 2y(x)
1
7x(x)
0, b=1,
Applying this theorem integral, obtain the following asymptotics for the KL divergence:
log (x)
DRI( (x) mg x)) 3 (x)x) 7z(x)
O(B3),
where we used the fact that f(1) g(1) and f1( g0((1) after rescale and shift described in Appendix K.I.
34

---

Finally, we need to show that the coefficient Zy(x) log (x(=) expansion with respect to B. The partition function has the following asymptotics (see Appendix L for more details):
has constant leading term in its Taylor
2,(z)
7,(x) +0)
Zg(x) = 3 -0
Hence
(z(x)
+ 11
++
1
3 (1( 00
log
Zg(at (2(z))
(1) f² 8 0
log
(z(w)) (2(xx)
(d(())-((1))) +0(3).
(x)
Hence, by plugging the Taylor expansion for the coefficient - asymptotics obtain:
2,(0) log Zg(2) Zy(x)
into the KL divergence
B +
Thus, we show that for any two functions f and from the family of transformation funccioos that are strictly increasing, continuous, defined on [0, 1] with finite value and non-zero left derivative at = 1, under mild conditions, we asymptotically obtain the same optimal solutions. Moreover, the KL divergence between the corresponding optimal solutions decreases linearly with B, which means very fast convergence. In practice, this means that one can take an arbitrary function from the considered family transformation functions, given that sufficiently small.
L Asymptotics of the partition function Z for the family of quantile transformation functions strictly increasing and upper-bounded on the interval [0,
We compute the asymptotics of the partition function with respect to 3 for any choice of strictly increasing continuous transformation function (t) defined on [0, 1] with finite value and non-zero left derivative
We consider the case when the initial reward R (and thus the quantile-rewa reward R, and generalized transformed reward R) can be approximated as a continuous variable.
We assume mild regularity conditions, specifically f E ( 1]. We also assume that f is normalized according to the rescale and shift procedure described in Appendix K.1.
Recall that the partition function can be expressed as moment generating function for the distribution of the reward used in the objective I (see Appendix D for the derivation). Here we consider objective 1 with the generalized transformed reward R.
Ž = B Peot
exp (
where Pror (r v) denotes the probability distribution of the generalized transformed reward R. (xx,, y) induced by the reference policy y Fres(: ar).
By applying the change-of-varian Unconscious Statistician" (LOTUS)) obtain:
formula for expectations (also known as the "Law of the
E001) exp (ff(ro))] = f
- exp (1S(3)) dz.
Ž = E3 F
exp
1
Here we used the fact that the probability distribution of the quantile-reward R. y) induced by the reference policy y Tres(: æ) is uniform from 0 to (sce Appendix E for the derivation).
35

---

Since we require f to be strictly increasing smooth function defined on [0. 1] with finite value and non-zero left derivative = 1 can apply Laplace's method for an endpoint maximum to compute the asymptoties in terms of beta.
Here is the Laplace method formulation for reference:
Theorem (Laplace's method for endpoint maximum). Consider the Laplace integral
F(A) = [- M()) di,
-
Let 1 |a, b) be a finite interval and assume that
1. [(L) attains its maximum on [a, b] only at the right endpoint = b;
2. h, f E =C(la,b))
3. h, f E C a neighbourhood of = b and f'(b) + O.
Then, X 0C,
F(A) exch)) Σ
dx x
k=0
with coefficients
dx = (-ro
a)
(F()
1t=b
Moreover, this expansion may be differentiated with respect to X arbitrarily many times.
Applying this theorem to our integral (by taking a 0, 1, = 1/2 and h(x) = 1) we obtain the following asymptotics for the partition function:
(z) = +
Note, that we assume the function f to normalized according to the rescale and shift procedure described in Appendix K.1 (as require this property in the transformation functions equivalence proof K). Without this normalization the asymptotics will have different form which can be derived similarly using the Laplace's 's method.
36

---

M
No Detailed experimental protocol
Table Initial models used experiments.
SFT-only Model
Base
Fine-tuning Data
Llama 8B Tülu SFT (Lambert et al., 2024)
Llama-3.1-88 Mistral-7B-v0.2
Tilu SFT dataset
Mistral 7B Instruct v0.2 (Jiang, 2024)
Unknown public instruction datasets
Table 6: Datasets used in experiments.
Dataset
Task
Truin Size
Test Size
Filtering Criteria
Filtered
Magpie-Air (Xu et al., 2024b)
General Chat
98,000
2,000
Length > 2048 tokens
< 1%
UltraFeedback (Cui al., 2024)
General Chat
61.135
2,000
Length 2048 tokens
1%
LeetCode (Xia al., 2025)
Coding
2,641
228
I Length > 2048 tokens or 20 test cases
< 3%
Table Rewards used for training and evaluation.
Task
Reward
Description
General Chat
Reward model with sequences truncated 2048 tokens I Length-cone controlled AnmoRM (new) Length hias canceled via regression as in AlpacaFval 2 (Duhois et al., 2024)
ArmoRM (Wang al., 2024)
General Chat
Coding
Python Sandbox (new)
Average pass rate over 100 test cases per problem
Table 8: Sampling parameters for reference and evaluation completions.
Sampling Purpose
Temperature
Top-p
Reference Completions
1.0
1.0
Evaluation (Llama 8B Tülu 3 SFT)
0.6
0.9
Evaluation (Mistral 7B Instruct v0.2)
0.7
0.9
Table Distribution shift settings RL fine- fine-tuning.
Setting
Additional SFT Chosen Before RL Pairwise Preference Generation & Annotation Method
offline
Offline dataset annotations
off- fff-policy-best
No
Best and worst from 6 completions (ArmoRM scores)
Ranked random pair from completions (ArmoRM scores)
SFT-offline
Yes
Offline dataset annotations
SFT-off-policy-best
Yes
Best and worst from 6 completions (ArmoRM scores)
SFT-off wwwfoooolic--rnddeee
Yes
Ranked random pair from completions (ArmoRM scores)
Table 10: Hyperparameters for supervised and RL fine-tuning.
Phase
Epochs
Learning Rates
Batch Size
Optimizer
LR Schedule
Gradient Clipping
SFT (Chat)
10-7
128
Adam
Casine w/ 10% warm-up
10.0
SFT (Code)
3
10-7
128
Adam
Cosine w/ 10% warm-up
10.0
RL.
1
{1,3, 10) x 10
128
Adam
Cosine w/ 10% warm-up
10a (effectively none)
Table 11: RL fine-tuning hyperparameters: KL-regularization parameter (3) sweep and QRPO number of generated reference rewards.
Algorithm
Chat Task
Code Task
QRPO Reference Rewards
DPO
0.01. 0.03, 0.1 2.5, 10
0.01, 0.03, 0.1 2, 2.5, 10
SimPO
REBEL
1c-6, 1c-4, le-2
1c-4, 1c-2, 10
QRPO
3c-4, 0.001. 0.003
0.003. 0.01, 0.03
n (Magpic-Air), 3 (UlraFeodback), 20 (LeetCode)
37

---

The code- def Lypo_Loss(beta, logps, ref_ Logps, rewards, ref_rewards): ""*Compute the QRPD loss for a batch of pronpts.
Angs:
beta ('torch.Tensor: (1,)"):
The beta paraneter for the QRPO loss.
Lagps (etsa:: beatch_size,)")
Tensor:
Log probabilities of the training completions for the model.
ref_logps (" ("torch. Tensor: :
the training completions for reference model. rewards "torch.Tenson (batch_size,))):
Log probabilities
10
11
Rewards of the training completions.
Tensor: ref_rewards Tenso:: (batch_size, nun_ref_rewards)
Rewards of the reference completions generated by the reference model.
14
Returns:
15
loss ('torch. Tensor (otorch (iatch The computed QRPO loss.
Lag_ logps ref_logps
quantile_rewant (ref_rewards rewards. (nnaddssunsqueezeddnn)))) float (daan()
19
log_2 torch, (be(taa)) 1 beta
20
Loss aovantil__rewardee beta * log_2 beta log_ratio)
return Loss
Figure 10: QRPO loss reference implementation in Py
M.1 Codebase
We release a reference implementation of QRPO based on a HuggingFace TRL (von Werra et al., 2020) trainer also supporting the baselines trained (DPO, REBEL, SimPO) at https: https://githun com/CLAIRE Conm/CLARE-LLao/quaee ott.a//CCAaaaaaa/oaaaaaeaaaaaaaa base is based the Python Machine Learning Research Template (Moalla, 2025). It contains
All the scripts used train and produce all the results presented the paper (including reference data generation, training, and plotting).
All of our infrastructure and experiment management code for running and managing (tracking failures, etc.) experiments at scale SLURM cluster.
A reference implementation for scalable code sandbox on SLURM clusters with container runtimes, that does not require elevated privileges.
The specification of our software environment for reproducibility (OCI container image, pip dependencies, etc.). We acknowledge that few users have access to the NVIDIA GH200 platform reproduce our exact environment and numbers throughout the training pipeline, so this specification mainly serves as a reference.
M.2 Initial SFT-only Models
We consider popular middle-scale models that have been instruction fine-tuned (IFT/SFT) but have undergone RL fine-tuning." We choose al DelenaiLLlaaae 1-Tuuu---------57 (Llama 8B Tülu 3 SFT) (Lambert et al., 2024), a fine-tuning of the meta- m 1-8B base model on the Tülu 3 SFT dataset (Lambert et al., 2024), and miscrallil/MMssraal----itttttrrn 7B Instruct v0.2) (Jiang, 2024).
(Mistral
TThis is important as experiments emulate the RL fine-tuning stage after the supervised fine-tuning stage a post-training pipeline, and importantly because reporting improvements only specific benchmarks over already (usually closed-soure RL fine-tuned models is biased view on performance profile these models which have been carefully tuned to balance multiple benchmarks. E.g.. taking Jlama-3-8B. I-B-rrntttutt and improving its AlpacaEval 2 scores by RL fine-tuning it on narrow chat dataset is not faithful way to report an algorithmic improvement does not show where the model has regressed.
38

---

M.3 Datasets
For the general chat task. we use two popular aligmment datasets, extensively used in previous work (Meng et al., 2024; Gao et al., 2024) consisting primarily of synthetic data.
Liipriee-AAiiggaaaaaa----------- 100 1 (Magpie-Arr) (Xu al., 2024b) is strong align- ment dataset containing 98000 training samples (and 2000 testing samples) with synthetie instructions and completions generated by -Jama-3-8B-Instuee
targeting mainly information secking, and other categories such as creative writing, advice seeking, planning, and math. The chosen (resp. rejected) completions have been selected by picking the best (resp. worse) completions in a set of completions rated by the RLHFlow/
-aailaama-----aa- 8B-v0. . reward.
httpe//aacee//uuutaffffeeeee www.eeeeaaac..bbbaieeeeeee.e (UltraFeedback) (Cui et al., 2024) consists of 61 135 training samples (and 2000 testing samples) with instructions targeting instruction following, truth- fulness, honesty, helpfulness, and completions generated by a large pool of models with different capabilities (between 7B and 70B open models and closed models including GPT4), annotated by GPT-4. This is the pre-processed version of the original Ultraeedbacck dataset that used train Zephyr-7B-8 (Tunstall et al., 2024), where for each prompt, the completion with the highest overall score is taken as the chosen completion, and one of the remaining three random as the rejected one.
(LeetCode) (Xia et al., 2025), which contains 2641 training samples (and 228 testing samples) comprising easy, medium, and hard LeetCode problems. Each problem has a median of 100 test cases and contains a correct solution that we use for additional SFT before performing the RL fine-tuning phase.
For the coding task, we use seaffaccdeeLeetCCdeDtttatt
For all the datasets, in both the train and test splits, we filter out the samples where the length of the prompt plus the completion is longer than 2048 tokens. For the LeetCode dataset, we additionally filter out the problems which have less than 20 test cases. This results in less than I % of filtering for Magpie-Air and UltraFeedback, and less than 3% of filtering for LeetCode. This is to match our training and reward model context length of 2048 to avoid prompts for which the model often generates long sequences that get truncated at training time (off-policy) and by the reward model and would not give a meaningful reward signal.
We additionally split the test set in each dataset into two equal splits: a validation subset for hyperparameter selection and test subset report the final numbers. For the LeetCode dataset, we maintain the distribution of easy, medium, and hard problems in the subsets (stratification by the difficulty label).
M.4 Rewards
For the general chat task we use the 2024) reward model with a maximum sequence length 2048. It is used to compute the rewards directly used by REBEL and QRPO in all distribution settings and to label the chosen and rejected preferences for DPO and SimPO in the off-policy setting. This also holds for Magpic-Air Air in the offline setting its chosen and rejected annotations have been annotated with ArmoRM as well by Xu et al. (2024b). For UltraFeedback the offline chosen and rejected annotations are not annotated by the ArmoRM, but most of the best models for the baselines turned out to be from the off-policy setting due to the low quality of the UltraFeedback offline completions.
(ArmoRM) (Wang et al.,
Although ArmoRM has verbosity component to reduce length bias, it still suffers from out- of-distribution length bias. i.e., its length bias has been minimized on specific outputs (original Ultraeeedback), but is still present when computing the rewards on chats that diverge from the UltraFeedback outputs, which we did observe in our experiments. We therefore also report a length- controlled reward whose coefficients are computed for each model based on regression that cancels the length component as done in AlpacaEval 2 (Dubois et al., 2024) (see Appendix M.5 for more details on the length-controlle reward computation).
For LeetCode, we use Python sandbox for executing LLM -eeerrtee code in safe environment with exclusive and controlled hardware limits, which we built for this project. The reward for a solution to problem is the average number of test cases passed the suite of tests for the problem. Each problem has a minimum of 20 tests and a median of 100 tests, and we cap the number of tests to run for each problem to the first 100 tests for computational efficiency. We use this average pass rate reward instead of binary reward indicating whether all tests passed for two main reasons: (i)
39

---

because the return value of the solution for edge cases is often not specified, still those appear in the test cases with arbitrary targets causing high rewards collapse when binarized (e.g., passing the test cases with rate of 0.99 because of single failed test on an under-specified edge case would be converted to 0 binary reward). (ii) because we assume a reward that can be treated as continuous in the theory of QRPO.
M.5 Length-controlled reward (I.C-Reward) computation
For every prompt x in an evaluation split (validation or test), we first collect completions produced by the reference policy Tref and compute the prompt-wise mean and standard deviation of the original reward and the token length 7. corresponding to the reference policy:
PR (x) = By- [(x
o (z) stdy (1 [)],
32 (x) = By [ty)]
(f(()) sldy [T(]]
These prompt-wise statistics absorb two major confounders: (i) the intrinsic difficulty of the query, captured by the average reward p (z) and its variance of (f((()) and (ii) the target answer length implicitly specified by the prompt, captured by pf. (x) and of. (z).
Step I: prompt-wise normalization For each completion produced by a trained model me we compute the normalized reward and completion length using the normalization coefficients for its prompt:
Raom (z, v) R(x, u) PR (z)
Luorn (I, v) L(y) (x) L (x)
R (r)
Step 2: fitting the length coefficient for a trained model Given a trained model's outputs on the entire evaluation split (validatioon test), we regress the normalized reward the normalized length by fitting a linear model: Ream (x. 3) Lnomn (x,, v) bbb+
where k captures the global linear dependence of ArmoRM scores on completion length for this model in the evaluation split and b absorbs any fixed shift.
Step 3: debiasing the rewards for a trained model Finally, we subtract the length contribution (bias) learned for the trained model and transform the result back to the original reward scale:
Ric(I, v) = RRorr (x,, y) - Inorm (z. v)) (x(
(x)) + 1
Written directly in terms of the original magnitudes this
Ric(2, v) = R(z, y) - L(v) (((((2) of (x((x)
(x)
By construction RLc has zero first-order correlation with length while preserving the prompt-wise mean and variance of the original reward distribution, ensuring that improvements measured with LC-Rewards reflect qualitative gains rather than verbosity.
M.6 Sampling for reference completions and evaluations
When sampling reference completions, we use a temperature of and top-p of 1 to keep the generations faithful to the distribution of the reference policy. This keeps the implementation faithful to the theoretical derivation of QRPO and at the same time maintains diversity in the reference completions and rewards. We did observe in early experiments that sampling with a lower temperature and smaller top-p resulted in less diversity reference rewards and a worse evaluation performance.
We have also derived a version of QRPO for rewards with small number of possible values in Appendix F which would be compatible with the binary reward in this case. We defer this setting future work.
40

---

We defer the exploration of the hyperparameters related to the generation of the reference completion to future work.
When sampling evaluation completions, we use the recommended temperature and top-p for each model, that is temperature of 0.6 and top-p of 0.9 for Llama 8B Tülu SFT and temperature of 0.7 and top-p 0.9 for Mistral 7B Instruct v0.2. The parameters of Llama 8B Tülu 3 SFT are taken from its default generation configuration file and the parameters of Mistral 7B Instruct v0.2 are recommended choice within API servers.
All the sampling is done with vLLM (Kwon et al., 2023) and we keep all the other sampling parameters as default. In early experiments we generated up to 200 completions per prompt, and since this is embarrassingly parallel problem, our codebase includes a scalable sampling pipeline with data parallelism to effectively scale this task to a cluster scale.
To generate completions for the LeetCode dataset, we modify the original prompt to make the model aware of the initialization code that runs before its suggested solution. This code includes generic imports and definitions of helper classes (see Figures 11, 12, and 13). The imports and definitions are used in the template solution code that the model gets in its prompt and in the test suite, and are therefore critical in its crafting of the problem solution. The original initialization code is not constant across problems and we found it to lack robustness and to be subject to bad practices such as having several wildcard import statements. We therefore combined and refactored all initialization prefixes into a single unified one shown in Figures 12 and 13.
M.7 Distribution shift settings
is well known that although policy fitting methods can be used to fit the optimal policy with any data distribution, they do exhibit different levels of performance depending on the distribution shift between the model being trained and the completions in the training dataset. For example, it can be recommended to perform SFT on the chosen completions when the completions in the preference dataset are not from the model being aligned. We therefore let the distribution shift setting be hyperparameter and select 6 different distribution shift settings to put all algorithms in their best setting. We consider the SFT-chosen and no-SFT cases where the model is first fine-tuned on the chosen completions or not (our initial models are already instruction fine-tuned, this is an extra fine-tuning to reduce distribution shift). Subsequent to these two settings we have the offline setting (data from the dataset) and two off-policy settings, where replace the chosen and rejected completions in the dataset (keeping the prompts) with the completions generated by the model before training (thus 2 (1 + 2)- 6 settings). We refer to the off- off-policy settings by (i) off-policy-besr when we follow Xu et al. (2024b) in picking the chosen (resp. rejected) from the best (resp. worst) out of completions scored by the ArmoRM reward model, and by (ii) po--poccc--addon when we only pick them by comparing and anmotating random completions from the completions. Unlike popular belief, observe that DPO obtains better results on off- off-policy-ree ponicy-random than on off-policy-beer in multiple cases (see Table 12 for example).
For the LeetCode dataset, it is small and does not contain preferences, we only use the off-policy random setting to generate 20 completions per prompt and group them in 10 random pairs that we annotate with the code exccution reward to make preferences and increase the dataset size by 10 folds (cach prompt gets 10 pairs).
M.8 Hardware and software acceleration
We train all the models (SFT and RL fine-tuning) with the same distributed training setup. We dis- tribute training with HuggingFace Accelerate (Gugger et al., 2022) using the DeepSpeed (Aminabadi al., 2022) plugin at ZeRO stage (optimizeer state partitioning) over GPUs from 2 nodes with NVIDIA GH200 each.
Our SFT trainer and RL fine-tuning trainer (same trainer supporting DPO, SimPO, REBEL, and QRPO) are based on the HuggingFace TRL (von Werra et al., 2020) SET and DPO trainers.
'The data was available on show this data anymore. These are also the parameters suggested when asking Le Chat.
ai/, but the website changed interface and does not
41

---

M.9 Baseline algorithms and hyperparameters
We perform full model fine-tuning with a batch size of 128 (128 pairs of completions for RL fine- tuning and 128 individual completions for SFT). For both SFT and RL fine-tuning we train using the Adam optimizer and a cosine learning rate schedule with 10% warm-up steps.
For SFT (when fine-tuning on the chosen completions) we use learning rate of 5e - 7 and train for one epoch in the chat task and three epochs in the code task (smaller dataset) The SFT loss is computed on all the tokens in the sequence (prompt and completion). This is the default in the TRL trainer. We apply gradient clipping by norm with value 10.0 (to prevent potential spikes, though the norm stabilizes below 10.0 very early in training).
For RL fine- fine-tuning, we sweep over three learning rates [1c -le-7,33 7, 7, 1c 6) and train for one epoch. We run three 3 values for each algorithm, following log scale, unless specific hyperparameters are recommended by the authors of the respective algorithms. For the chat task, for DPO sweep over [0.01, 0.03, 0.1] which is the same magnitude as in Meng et al. (2024) and are also commonly used in the literature, for REBEL sweep over [1e 6, le 4, 1e 2] as done by Gao et al. (2024) in a similar experimental setting using UltraFeedback and ArmoRM,, for SimPO we sweep over [2, 2.5, 10] as recommended by Meng et al. (2024) keeping the margin 0.5 as recommended by their public codebase to maintain the same budget of hyperparameters for all methods, and for QRPO we sweep over [3e- 4, 0.001 0.003] which we found to be good range in preliminary experiments on another chat dataset. The code task is harder and we expect worse offline-to cefline-to-onliee to-onl generalization requiring to aim for a more conservative optimal policy and change in the rewards. Therefore we adjust the range of 3 values used in the sweep for each method depending on the impact of / on the method to obtain similar impacts: for REBEL. we increase 3 by orders of magnitude and we sweep over [le 4, le 2, 1.0]. one order to be more conservative and one order to match the difference between rewards which increases from the chat task by around one order of magnitude since REBEL is sensitive to the reward scale to preserve the gegnal-o-regulaaan
ratio (higher /3 values have also been used by Gao et al. (2024) in their summarization experiment); For QRPO we increase by only one order of magnitude and sweep over [0.003, 0.01, 0.03]. we only need one order to be more conservative and do not need to take into account the change in reward scale as the quantile reward is invariant to it; For DPO we keep the same values for B, since this parameter is not tied to an external reward scale (model learns an internal reward implicitly based on preferences). Moreover, we do not adjust for additional conservatism, as B = 1 is too conservative. Finally for SimPO, the training completions are at least twice shorter therefore /3 is already effectively increased in the loss and the values do not follow a log scale so we can't increase by an order of magnitude and thus we keep the same values.
To estimate the percentile rewards for QRPO, we use 1 reference model rewards for Magpie-Air, n = for UltraFeedback, and n = 20 for LeetCode. Note that when using n = 1, the quantile reward is obtained by comparing the reward of the training sample to the reward of the single reference reward and yields a binary quantile reward of or 1.
We use the same RL fine-tuning trainer for all methods, changing only the training loss with the configuration file, so although QRPO only needs one completion instead of a completion pair, we consider the pair as single sample in the batch so that all methods use the same number of training steps when consuming one epoch. QRPO applies its loss independently on each completion of the pair and can be seen as effectively using double the batch size, but is actually using exactly the same number of completions.
M.10 Evaluation methods
For each training run, save five equally spaced checkpoints and compute the online rewards of completions generated by the checkpoints on prompts from the evaluation split of the dataset used in the run. The rewards are based on the ArmoRM reward model for the chat task and the code execution reward for the code task. We repeat the online evaluation three times to obtain estimates of the mean performance and its standard deviation.
We split the evaluation dataset into a validation split (selection) and a testing split (reporting). For each baseline and task, we use the mean online reward in evaluations on the validation split to select the best checkpoint across hyperparameters and checkpoints (learning rate, beta, distribution shift setting, checkpoint number) for given algorithm, model, and dataset, and report mean performance on the test
42

---

split and its standard deviation. We start by selecting the best checkpoint across hyperparameters for each distribution shift setting. Then we perform a more careful selection among the best checkpoints for each distribution shift setting as we believe that the training data distribution shift can qualitatively affect different characteristies of the trained model, not limited to the online reward: We first shortlist checkpoints that are in the 98%-CI (one-sided) of the best online reward value among the checkpoints. Then, we refine the shortlist to checkpoints that are in the 80%-CI (one-sided) of the best online length-controlled reward (LC-Reward) value. This procedure allows us to take into account both online Reward and Lewar to make a more robust selection. If we have more than one checkpoint present in the final shortlist, we choose one with the better online reward.
For downstream evaluations, on the chat task we further evaluate the best checkpoint of every algorithm on AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024) and report the length-controlled win rate against GPT-4 preview 1
denoted as gpt4 turbo in the AlpacaFval repository) as the final performance of an algorithm on a (model, dataset) configuration. A AlpacaFival 2.0 widely used benchmark by the community, assessing broad conversational skills across a wide range of queries with total of 805 questions. On the code task, the reward on the test split of the evaluation set can already be considered as downstream as they measure the performance of the model on the popular LeetCode coding task.
M.11 Statistical significance
The claims in this work are supported by experiments that include LLM training and evaluation. There are three main sources of randomness present at two levels. (i) At the training level, our
results can be influenced by the reference model generation randomness used to compute the quantile rewards for QRPO, the random shuffling of the data samples in the dataloader, and the hardware acceleration non-determinism. (ii) At the evaluation level, our results can be influenced by the sampling randomness during the evaluation of trained models and again by the hardware acceleration non-determinism. Unfortunately, it is computationally prohibitive to repeat the training several times for each setup to estimate the noise in the results, thus the first source of randomness cannot be addressed directly. However, we address the noise introduced by sampling from the models during evaluation by repeating each sampling procedure three times and estimating the average value and a standard deviation (we cannot repeat the experiments on different hardware to control for that source of randomness). All conclusions drawn from the obtained results were made by taking into account the obtained noise estimates. Furthermore, we employ a 3-split strategy to train, select hyperparameters, and report performance which has not been employed in previous offline alignment work. This is possible in our case because we compute online rewards during evaluation for all checkpoints and therefore can perform a robust hyperparameter selection.
M.12 Common implementation pitfalls and experiment protocol limitations
We have identified several pitfalls in alignment algorithms implementations and experimental proto- cols while experimenting with various fine-tuned models and tue-ttning libraries and attempting to replicate previous published results. We provide a non-exhaustive list below aimed to raise awareness about these details that result in non-negligible performance differences. In particular, examples mentioned in this section are not meant to point out issues in specific codebases. We recommend the open-source community adopt more robust testing for these cases. We have observed that the codebases with the most bugs are those that try to support the largest number of use cases and users, introducing frequent bugs with the frequent addition of new features, and we encourage the community to weigh a bit more the robust testing end of the tradeoff.
End-offsequence (EOS) token The EOS token is part of the sequence the model predicts to a have a sound language model distribution, however we have observed several implementation issues with regard to it. For example we have found that:
There was a bug in SFT Trainer of the TRL library (von Werra et al., 2020) where when the padding token was set to be the same as the EOS token as shown in their examples, the EOS token would be masked out in the loss like the padding token, leading to models never predicting and end of sequence during the SFT and then at inference time generating non-ending sequences. This bug has since been fixed in TRL (43200, #3328).
43

---

Popular models diverge in the EOS used for pre-training and post-training with their Hug- gingFace config remaining with the pre-training EOS and leading them generate non- ending sequences out of the box. (F..., the Magpie-Align models trained with the Axolotl (EbS and chat template).
An extra EOS token is added to the sequence after applying the chat template if it already contains an EOS in the DPO implementation of TRL. This bug is still present in TRL at the time of writing. (Added EOS after chat template.)
Gradient and loss accumulation across devices and accumulation steps There is an increased complexity to aggregate the loss of all micro- barche across devices and accumulation time steps when training in a distributed setting and using gradient accumulation to simulate a larger batch. For fixed batch size any combination of these dimensions of splitting the batch size should yield the same gradient. Yet we have observed several bugs with regard to this aggregation which leads to biased gradient and incorrect reported gradient norm in several libraries. For example,
In the HuggingFace Transformers library (Wolf et al., 2020) there was bug leading to the gradient norm to scale with the number of accumulation steps, which is solved at the time of writing (435808).
There is still new code being merged in the TRL library for which the gradient norm is not accurate (#3317).
The incorrect loss and gradient accumulation bug has had long history of being patched and then appearing again (Unsloth blogpost, #35207, #3574).
Arbitrary sampling parameters at evaluation Some benchmarks models do specify the sampling parameters for their evaluations. For example, AlpacaEval (Li et al., 2023) does not specify decoding configuration, resulting in different works reporting performance on AlpacaEval with different decoding strategies even when using the same base model making the comparison across works very hard. For example, for the same Llama 3 8B Xu et al. (2024b) use greedy decoding with a repetition penalty on AlpacaEval while Meng et al. (2024) perform grid search over multiple sampling temperatures and report their results with temperature of 0.9.
Unfair hy perparraeeee budgets We believe that new methods should compare to baselines while maintaining the same hyperparameter budget. This is because if new method introduces new hyperparameter that increases the computational budget to find its optimum, baselines could as well find better optima by searching over more hyperparameters. However, find that this is not always followed. For example, while Meng et al. (2024) perform a comprehensive hyperparameter search for their baselines, they still use four times more hyperparameters for SimPO than for DPO.
Unclear hyperparameter selection In offline alignment, at least with the current algorithms, there no accurate metric to predict online performance on downstream evaluations. For example, we have observed that the DPO loss, margins, or accuracy are not reliable metrics to predict online performance. This has also been observed by Rafailov et al. (2024) for other policy fitting algorithms. Furthermore, the best hyperparameters cannot be selected based on the online performance on the downstream evaluations, as this practice would be subject to a selection bias. Therefore, it is important for offline alignment work to report their hyperparameter selection protocol. Yet, we have found that several research papers do not document their hyperparameter selection protocol. This also leads to reproducibility issues and discrepancies between papers that seemingly use the same experimental protocol (datasets, hyperparameters, etc.), but still report different numbers.
44

---

Example formatted prompt from LeetCode
You are an expert Python programmer. You will be given question (problem specification) and will generate correct Python program that matches the specification and passes all tests.
### Question:
Given an array of integers nums and an integer target, return indices of the two numbers such that they add up target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You return the ansver in any order.
Example
Input: nums [2,7,11,15], target
Output: [0,1]
Explanation: Because nums [0] nums [1] -- 9, we return [0, 1].
Example 2:
Input: nums [3,2,4], target Output: [1,2]
Example 3:
Input: [3,3], target Output: [0,1]
Constraints:
<= numa length <= 104
-109 <= nums[i] <= 109
-109 <= target <= 109
Only one valid answer exists.
Follou-up: Can you up with algorithm that is less than (n2) time complexity?
### Format:
Your code will run after the following definitions imports, which may or may not be needed.
"python (see Figures and ...)
You will use the following starter code to write the solution to the problem and enclose your code vithin delimiters. You have to add the missing imports to the starter code to run correctly. pyyhhon
Additional imports.
# End of additional imports.
class Solution:
def tsoSf nums: [ist] target: int) List List]
###
Answer: (use the provided format with backticks)
Example formatted prompt from LeetCode from which reference solution are generated. We have modified the original prompts adding the highlighted text make model aware of the code that runs before its suggested solution.
Figure 11:
45

---

Definitions and imports for helper classes (part I)
import collections
import itertools
import functools
import math
import string
import random
import bisect
import
import operator
import heapq
import queue
from typing import List, Tuple, Dict, Any, Union, Optional from queue import PriorityQueue from itertools import combinations, permutations from functools import 1ru_cache from collections import defaultdict, OrderedDict, deque, Counter
inf float('inf')
class ListNode:
def inint___ (self, val=0, next=None): self.val val val self.nert next
def noteenooe list):
if not values:
return None
head ListNloee (valuee [0])
head for val in [a:]
node ListNode (val)
p.next node p node head
def is_same (p1__ p2):
if p1 is None and p2 is None: return True if not p1 or not p2: return False
return p1. val == p2.val. val and is_same list (p1. next, p2. next)
Definitions and imports for helper classes (part 1). These are used in the prompt (Figure 11) and prepended to the code generated by model.
Figure 12:
46

---

Definitions and imports for helper classes (part
class Treelloee:
def ..init_. (self, val=0, left=Nome,, right=None): self val val self lelf.left. right right
def tree_node (values: list): if not values:
return None
root (values
-
queue deque()
queue. (pppeed(opdd
shile queue:
node queue queue. popleft()
(values)) and [i]
node. TreeNldee (ialues queue. apppueeeppeddddde.eeeee
1
lan (values) and values [i] is not None: node right TreeNode (values(i)) queue (node. 4-
return root
def is_ tt__aam__trea_ee q):
p q: True return False
9F
elif p. val 1= val:
return False
else:
return is_same_treegf ceaane_treeep.effe q.left) and js_eame_trree(prrige q.right)
class Node:
def __init__ (self, val=0, left=None, right=None, random-None): self val val self left left right right randon
class CategoryHandler:
def haveSaneCatogory(aatt a: int, b: int) bool:
pass
Figure 13: Definitions and imports for helper classes (part 2). These used in the prompt (Figure 11) and prepended the code generated by a model.
47

---

N
Detailed empirical results
0.188
0.182
0.170
0.188
0.169
Reward
Reward
0.180
Reward
0.184
0.168
0.182
0.178
palicy, otfine, f00.000
off policy offine, B80000
0.167
palicy, f=0.1
0.180
0.176
otfine, f00.1
0.168
10"
10
10a
10a
10"
10 Number references
Number references
Number of references
(a) learning rate 1c-7
0.183
0.171
0.188
0.182
Reward
Reward
0.181
Reward
0.170
0.186
0.180
0.169
oli-poliyy, $=0.003
0.179
off-policy, f=0.01
0-0000
0.168
all ali-palcy, Pr0.1
otfine, 8=0.003
0.178
offine, 8=0.01
0.167
offine, 8-0.1
10"
10
10a
10a
10a
Number of references
Number of references
Number of references
(b) learning rate 3e-7
0.184
0.171
0.170
Reward
0.186
Reward
Reward
0.169
0.180
0.168
0.184
off-polley, $=0.003
off-poliyy
0.167
ott-policy,
otfine, f-0.003
0.178
offine, 8=0.01
0.168
otfine, 8-0.1
0.182
10P
10
10a
10a
10a
10 Number of references
Number of references
Number of references
(c) learning rate
0.184
0.171
0.188
Reward
Reward
0.182
Reward
0.170
0.186
0.169
0.168
off-policy, $-0.003
off-policy, 8-0.01
off-poley, P-0.1
0.184
ottine, $=0.003
0.178
8=0.01
0.167
ottine, #=0.1
10a
10 Number references
10a
10a
10a
10 Number references
Number of references
(d) Best learning rate validation set for each number of reference rewards.
Figure 14: Scaling curves Figure different learning rates. fixed and learning rate, still select the best checkpoint according to a validation set and report the performance on a test set, as described our experimental protocol in Appendix M.
48

---

Table 12: Llama 8B Tülu SFT, Magpie-Air dataset. The best models selected across distribution shifts marked with according to rewards and LC-rewards as described in the selection protocol Appendix M. We underline the values determining the selection shortlist using the validation split (valid).
Method
Setting
Reward (valid/test)
LC-Reward (validitest)
LR
Step
Base
0.1493 nooes 0.1519 eone
0.1495
a: 0008
0.1518 nore
SFT
0.1612 0.1649
0.1614
0.1648
5e-7
764
offline
0.1871
/
0.1903
0
0.1750
0w02 / 0.1780
0.00s
0.03
le-6
320
offpolicy-best
0.1734 + n009
0.1761
= 0.00
0.1671
= 0000 / 0.1776
0.000
0.01
3e-7
320
DPO
offpolicy-rane random
0.1774
0.1804
0.1682
oo0e 0.1716
4.00s
480
SFT offline*
0.1912
0.1943
0. 1869-
0.1904
0.01
le-6
160
SET - offpolicy- best
0.1874 nomer
0.1904
0000
0.1871
1 EOX0S
0.1900
1.000
0.01
le-6
160
SFT offpolicy-random
0,1872
0.1907
0.000
0.1900
0,1922
0.0001
0.01
1e-6
480
offline
0.1/193+
0,1519
0.000
0.1495
50006
0,1518
0.0001
10
1e-6
0
offpolicy-best
0.1830 0000
0.1858
0.00
0.1714
00007
0.1755
0.000
2,5
3e-7
160
SimPO
ranpoli
0,1822 1 runsc
0.1857
1. 0000
0.1735
1 DUECT
0.1740
1.000
10
3e-7
320
SFT offline*
0.1919
0,1976
8002
0.19413
0,1975
0.0033
10
1e-6
160
SIT - offpolicy-best
0.1918 00001
0.1941
0:00
0.1914
00000
0,1936
0.0000
10
3e-7
320
SFT - wwfpollicy-ranniee random
0.1912 1 nomee
0.1989
0000
0.1910 1. COXON
0.1938
0.000s
2,5
3e-7
764
oftiie
0.1830 + 00000
0.1861
0:00
0.1702 = 0000S
0.1747
= 0.0084
le-4
160
offpolicy-hest
0.177T
0,1813
0.1677
0.1723
1e-6
1e:7
640
REBEL
offpolicy-tandom
0.1744 + tuaee / 0.1776 03000
0.1677 DUMMIT
0,1707
0.000
le-4
le-6
320
SFT offline*
0.19141 non0s. 0,1937 0.001
0.1867 = 00000
0,1889
0.0022
1e-6
3e-7
764
SET offpolicy-best
0.1855
0,1884
0.1847
0.1883
1e-4
1e.6
160
SFT + offpolliy--
0.1827 tubac / 0.1859 0000
0,1824 CUERE
0,1860
2.00000
le-4
1e-6
480
offline
0,1855 tunno / 0,1879 0.0000
0,1741 + EXXOO / 0,1835
0.000
3e-4
le-6
480
offpolicy-best
0.1762 no0es 0.1788 0.004
0.1719 = 00013
0,1754
= 0.0003
0.001
le-7
320
QRPO
offpolicy-randonn
0.1772
0.1804
0.1781
0.1787
0.001
1e:7
320
SFT offline*
0.19488 tumne / 0.1972 01000
0.1975
0,2005
2.0004
3e-4
le-6
160
SFT beft
0,1810 0on0:
0,1851 0.000
0.1831
= 00014
0,1867
0.0005
0.001
le-6
320
SFT offpolicy-
0.1759
0.1790
0.1962
0.1994
0.001
1e.6
764
Table 13: Mistral 7B Instruct v0.2, Magpie-Air dataset.
Method
Setting
Reward (valid/test)
LC-Reward (valid/test)
8
LR
Step
0.1574 1 come
0.1598 1 toxoe
0.1572 2: 0.c0s / 0.1598 azccs
Base
SFT
0.1611 5o0ee 0,1633 sonce
0.1610 0.004
0.1633 03004
56-7
764
offline
0.1764 oote 0.1783
0.1768 4.00 / 0.1775 0.00
0.03
3e-7
764
offpolicy-best
0.1803 + eooe / 0.1820 + cuxce
0.1761 acces / 0,1777
+ aser2
0.03
3e-7
160
offpolicy-random
0.1791 ovemn 0.1810 tumco
0.1787
00002 0.1795 asera
0.01
320
DPO
SFT offline*
0.1872 ooxe 0.1901
0,18733
a.00a / 0,1898
0.03
3e-7
320
SFT - bets
0.1869 a: e000
+: 0000 0.1891
0.1844
0.04 / 0.1856
a ore.2
0.03
3e-7
160
SFT ailpolicy-randeen
0.1859 1. ovees
0.1883 bones
0.1865
coos
0.1879 a
0.03
3e-7
480
offline
0.1630
0.1654
0.1630
0.1653
10
1c-7
160
offpolicy- best
0.1678 exmo. /
0.1704 Loner
0.1679 a.0e /
0.1712
arecs
2
1e-7
160
SImPO
offpolicy-random
0.1786 =: 0009 0.1807 +: n000
0.1720 0004
0.1739
0.00
3e-7
160
SFT offline
0.1855
0.1880
0,1853
0,1876
10
1e-7
160
SFT - oept
0.1778 + euoce / 0.1802 + sumce
0.1764
- 0.000
/
0,1786
azocs
10
1c-7
160
SFT > offmolicy-rrndoom
0.1859 =: 00005
0.1884 + 0000)
0.1851 0.06
0.1879
0.0022
10
3e-7
160
offline
0.1727 sunce
0.1734 sunce
0.1789
0.001
0,1821
03002
lc-4
3c-7
764
offpolicy-best
0.1675 e000 /
0.1719 cunce
0.1806 0.004
0,1831
le-4
1c-6
160
REBEL
SFT offline*
offpolicy-random
0.1759 + e0000 0..7741 + tuxce
0.1845 00001
0.1864 00000 0.1771
0.1737
0,1868
0.000 /
0.006
0,1884
0.1764
0.000
0.001
lc-4
1c-4
36-7
3e-7
320
160
SFT - olfpolicy best
0.1757 oome /
1. nomon
0,1811
0,1818
le-4
1e-7
480
SFT + oaapolicy-random
0.1832 # 0000
0.1854 +t Duxca
0.1895
0000
0,1913
azeca
lc-4
3e-7
320
offline
0.1761 1 omeen
0.1782 nrven
0.1784 0004
0.17988
recs
764
offpolicy-best
0.1732
0.1755
0.1731
/ 0,1762
0.003
1c-6
480
QRPO
SFT offline*
offpolicy-randee random
0.1863
0.1657 1. eone
0.1886
0.1695
J. tiooe
sumec
0.1640 0004 / 0.1676 0z00
0.1866
0002
0,1893
0.003
3e-4
3c-7
3e-7
320
764
SFT + befp
0.1822 8000
0.18-19
00004
0.1822 0.000
0,18:19
0.0033
0.003
1c-7
764
SFT + olfpolicy
randooo random
0.1771 a: exn0
0.1800 noon
0.1778
0004 / 0.1810 2: aocss
0.003
1c-6
764
49

---

Table 14: Ilama 8B Tülu 3 SFT, UlraFeedhack dataset.
Method
Setting
Reward (valid/test)
L.C L.C-ewarrd (valid/test)
Step
Base
0.1278 0.1293
0.1279
0.1294
0.000s
SFT
0.1201 Loman
0.1219 a
0.1202
oo0e
0.1219
1.00004
5e-7
476
offline
0.1445 tunec / 0.1452 0.00
0.1445
ouen / 0.1455
0.0000
0.01
le-6
300
offpolicy-best"
0.1450 nome 00.1491
0.000
0.1480 = 0000 0,14493
0.0001
0.01
3e-7
300
DPO
olfpolicy-random
0.1470
0.1485
0.1470
0.1489
0.01
le-6
400
SFT - offline
011439 tome / 0.1444
0.00
0.1438
cumec / 0.1443
1.00003
0.01
le-6
300
SFT - offeolicy-best
0,1442 tume
10.1452 0.000
0.14413 + cooec 00.1451
0.0022
le-6
300
SPT offpolicy-rundom
0.1430
0.1436
0.1430
0.1436
0.0001
0:01
le-6
300
offlnee
0.1516
/ 0.1528
0
0.1512 + come / 0.1535
0.0005
le-6
200
offpolicy-best
0.1525 nooos
0.1539
= 0.000
0.1524 = 6000 / 0.1535
0.0089
le-6
200
SimPO
offolicy-random
0.150T
0.1522
1: oveen.
0.1515
no0s
476
SFT offlin
0.1518
0,1535
0000
0.1515
0.1523
le-6
200
SFT - befp
0,14833+ + conos
0,1500
0.000
0.14175 = 50006
0,1493
0.0002
2
3e-7
476
SPT of offpolicyy-annniee
0.1501 no0ns
0.1508
0005
0.14990 2: 0000.
0.1502 0.000
offline
0.1468
0.1478
0.1468 ouner
0.1473
1.00000
le-4
200
offpolicy-best
1474
0,1487
0.1473
0.1489
le-6
3e-7
200
REBEL
offpolicy-random
0.14178 00001
0.1487 0005
0. 0.1-178- = 6000)
0,1488
0.0004
le-6
le-6
100
SET oflline
0.1448
0.1465
D005
0.1443 1: ooeer
1.0000
SFT offpolicy-best
0.1403
0,1413
0.1404
0.1417
le-1
1e-6
300
SFT + offpolicy-random
0,1413 + Dunoc f 0,1427 0.000
0,14113 = EX00 / 0,1427
0.0084
lo44
le-6
300
offline
0,1454 1 tuase
0.1466
0.000
0.1394 1: cueno / 0,1426 :: ao0es
0.003
le-7
476
offpolicy-best
0.1503
0,1508 0.006
0.1505
0,1514
0.0003
3c-4
3e-7
100
QRPO
oanpolicy-random
0.1502
/ 0.1505 0:06
0.1506
0000
0.1505
0.000
3e-4
3e-7
100
SFT - offline*
0.1497 ruwnr
0.1504 oxce
0.1517
0,1556
0.0077
0.001
3e-7
100
SFT befp
0.1433
0.1448
80002
0.14431
0,1438
0.0022
0.001
1e-6
200
SFT ofmpoliey-rundom
0.1416
0.11429 0.00
0,1432
00007
0,1429
0.0060
0.003
3e-7
100
Table 15: Mistral 7B Instruct v0.2, UltraFocdback datasct.
Method
Setting
Reward (valid/test)
I.C.Reward (valid/test)
8
L.R
Step
Base
0.1335 + 0000
0.1349 + c000
0.1333 0 / 0.13/88 = a.e0
SFT
0.1243
0.1258
0.1237
0.1125 a00ss
5e-7
476
offline
0.1368 6000 / 0.1381 cum00
0.1366
a / 0,1379 0.004
0.01
le-7
476
offpolicy-best
0.1475 a: cooe
0.1484
a: oo00g
0.1456
aeces / 0.1484 o.00
0.03
1c-6
100
offpolicy random
0.1441 oren
0.1458
0.1433
a 0..449
0.03
300
DPO
SFT - offline
0.1402 m006.
0.1417
nonos
0.1410
axces / 0.1413 arees
0.01
1e-7
300
SFT - offpolicy--eeeee best*
0.1469 1 coxeo
0.1480
1. buxon
0.1459
a aoces / 0.1465 acoce
0.03
1e-6
100
SFT offpolicy-random
0.1411 50000
0.14151 50004
0.1430
0.1411
0.01
36-7
476
offline
0.1382
0.1392
0.1364
/ 0.1395
2.5
3e-7
476
offpolicy-best offpolicy-random
0.1133 + sumco / 0.1450 + sumce
0.1443
nccs / 0,14150
02623
10
3e-7
200
SimPO
0.1419 = nooos 0.1431 + 00004
0.1411
noes
0.14332
0.00.
2.5
3c-7
476
SFT offine
0.1415
0.1426
0.14008
0,1416
2
1c-7
400
SFT - bees
0.1460 =: eoone / 0.1472
+: 00005
0.1466
0.009 /
0.1478
=: 0.000
10
3e-7
200
SFT - callpolicy-randeee
0.1435 1. oonee 0.1441 1 booen
0.1445 8oos
0.1447
10
476
offline
0.1364
0.1384
0.1372 0:006
0.1393
0.000
1c-4
16-7
100
offpolicy-hest hest
0.1376 oxmoo / 0.1397 nomo
0.1368 a.00 /
0.13392
1e-4
3e-7
100
REBEL
SFT + oflline
offpolicy-random
0.1386 oonee 0.1395
0.1426 : coxee 0.1444 + cuxon
0.1397
0.1426
0.00a
0.1396
0.1442 azecs
aoecs
lc-4
1c-4
1e-7
1c-6
200
100
SFT - offpolicy-
0.1398 + sumoe / 0.1409 + cunes
0.1398 ncce /
0,14088
azocs
lc-4
lc-7
200
SFT > offpolic
0.1437 = 60000
0.1457 + 00000
0.14455 000s
0.1-666
0.000
lc-4
1e-6
100
offline
0.1413 souce
0.1424 sumce
0.1413
a 0.1422
a3oc3
0.003
1c-6
476
offpolicy-best
0.1455 sxoee 0.1469
60000
0.14660
0 / 0,1-170 0.000
0.003
1c-6
200
QRPO
olfpolicy-rnndom
0.1335 a. come
0.1349 J. nooee
0.13333
oces / 0.1348
2. orec
0.001
1e-6
0
SFT offline
0.1411 60001
0.1421
0.1418
0.14422
0.003
3c-7
400
SFT - ollpolicy best
0.1421 oome
0.1429 nonn
01.1222
neen / 01.1431 orecs
0.003
1e-6
200
SFT + offpolicy-random
0.1243 :: conee
0.1258
cuens
0.1237
0.1254
azecs
0.001
1c-6
0
50

---

Table 16: Llama 8B Tulu SFT. LeetCodeDataset dataset. The datasets contain 10 preference pairs per prompt. generated from 20 reference completions annotated the test-case pass rate reward. QRPO uses the 20 reference completions and their rewards to compute the quantile reward.
Method
Setting
Reward (valid/test)
8
1.R
Step
0,206 cale 0.209 aume
Base
SFT
0.215 coos 0.188 n
5c-7
100
DPO SFT + offpolicy-random
0.317
coos
0.302 0.04
0.01
le-6
80
SimPO SFT offpolicy-random
0.280 6037
0.223 2014
10
3e-7
160
REBEL SFT oafpolicy-ranoom
0.301
0.261
0.01
le-6
160
QRPO
SFT
offpolicy-random
0.316
0.327
0.01
le-6
200
500
DPC,p-0.4 0.49
400
Impliitt (normalized)
Smic, pm 0.37
Implicit reward (normalized)
400
QAPO, p- -0.16
300
300
200
DPO, p 0.664 0.64
100
Smnn, 0.44
QAFO, p -0.06
-2
3 Completion length (normalized)
Completion length (normalized)
(a) Llama Magpic-Air (Figure
Mistral 7B Magpic-Air
250
Implicit (normalized)
RFFe 0.20
Serri, p- 0.36
Implicit reward (normalized)
200
reani 0.331
0.65
200
CRRO
QRPO, 0.33
150
150
100
100
50
50
5
-2
2
4
Completion length (normalized)
Completion length (normalized)
(c) Ilama 8B on UltraFeedback
(d) Mistral 7B on UltraFcedback
in Figure 4 for all of the model and dataset configurations. Each figure shows the length bias of the best checkpoint each algorithm after training the given initial model on the given dataset. Trend lines capture a first order fit of the scatter points for each algorithm and the central point on the line indicates the average normalized completion length for the algorithm. The legend also indicates a Spearman rank correlation.
Figure 15: Length bias
51

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are made clear and numbered in the introduction. Claims is discussed Sections 3 and claims 2, 3, and 4 in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
52

---

Justification:
Each theoretical claim in the main paper references a proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the experimental protocol in Section and further describe its details in Appendix M including reference to codebase.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
53

---

reproducible, they Answer: [Yes]
Justification: We provide our codebase Section M. We used open open-source datasets that we augmented with synthetically generated data. We open-source these datasets with the codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co
gquides/CodeSubni
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. httppppwwwpgpu//p/pppppppeeeeeep
http:///Soouuumssiiioiiiaiaa ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We describe the experimental protocol in Section 4 and further describe its details in Appendix M with a reference to our codebase.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We discuss the statistical significance of our experiments in Appendix M.11.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
54

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss computing resources in Appendix M.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. httpp//pss.cpplllccccccccceceeed
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper addresses fundamental algorithmic gap with no specific application. The impact of any application should be discussed independently when using our method.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
55

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We will release models and synthetically generated data for tasks that have been well-studied before and have limited impact.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets used have been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
56

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We introduce a codebase with enough documentation in Section M.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
57

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in an important, original, non-standard manner.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https /ttps///neuui.ppeeeeee cc/Coonferrnncee/2225///1 for what should or should not be described.
58


Reviews:
[{"review_id": "jVR9v2RQJP", "reviewer": "Reviewer_n4LT", "summary": "This paper presents Quantile Reward Policy Optimization, a novel policy fitting method for fine-tuning large language models  that sidesteps a longstandong obstacle in  alignment: the intractability of the partition function in KL-regularized objectives when using absolute rewards. It introduces an elegant and computationally efficient solution by applying a quantile transformation to the reward distribution, rendering the partition function analytically tractable in closed form. Empirical results demonstrate QRPO’s superiority across chat and code generation tasks when compared to strong baselines like DPO, REBEL, and SimPO.", "questions": "1. Have you explored any other transformations with regards that allow the framework to a multi level preference reward signal?", "limitations": "Due to my limited exposure to applied RL I am unable to analyze the limitations of the empirical results.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "1. From a theoretical standpoint, the most remarkable contribution of this work is the closed-form expression for the partition function Z(x) after applying a quantile transformation to the reward. The tractability of Z(x), which typically renders absolute reward regression intractable, is a longstanding bottleneck in regularized RL. This is an elegant and rigorous contribution that simplifies the RL fine-tuning objective while preserving its core structure.\n\n2. The authors extend the novel theoretical insight and they validate the method comprehensively with experiments on two major alignment datasets and a code generation task, showing that QRPO outperforms state-of-the-art methods under comparable settings.\n\nSince I am from a primarily a theory background, I am unable to evaluate the robustness of the empirical results in a rigorous manner.", "was_revised": true, "final_justification": "I have gone over the replies of the reviewers. They have answered the questions to my satisfaction, and I maintain my positive score.", "rebuttal": "We thank the reviewer for their time and for providing a thoughtful and thorough review, particularly from the theoretical perspective.  \n\n---\n\n### Strengths and Weaknesses\n\n**Longstanding Bottleneck and Broader Impact**\n\nWe appreciate the acknowledgment of our theoretical contributions, especially the closed-form expression for the partition function. As highlighted, this addresses a longstanding bottleneck in the field and was **also described as \"elegant and rigorous\" by Reviewer u3X5.**\n\nAdditionally, Reviewer Ghhc noted the **broader potential impact**:\n\n> “The approach set out in this paper (i.e., converting an arbitrary scalar reward into a bounded, uniform signal whose partition function is a constant) could be reused elsewhere where we face energy-based or KL objectives with nuisance normalizers.”\n\n\n**Empirical Evaluation**\n\nRegarding the empirical evaluation, which the reviewer mentioned was outside their primary area of expertise, we believe the comments from other reviewers offer a helpful, complementary perspective.\n\n* Reviewer Ghhc:\n\n  > \"The experimental work is careful and high-quality, and appears to follow rigorous methodology to appropriately tune baselines, re-use appropriate reward models, and report numbers with uncertainty estimates throughout.\"\n\n* Reviewer u3X5:\n\n  > \"Solid empirical evidence supporting the effectiveness of QRPO as compared to the baselines.\"\n\n---\n\n### Questions\n\n**Multi-Level Preference Reward Signal**\n\nThis is indeed an interesting opportunity. Multi-level (multi-dimensional) rewards could be integrated into QRPO using several conventional strategies (linear projection into a single scalar reward, independent training on each reward, followed by model merging, or training with a linear combination of individual reward losses).\n\nHowever, there is also an opportunity for a novel approach within QRPO. Specifically, we can recover the quantile of a training completion given the reference completions with a non-linear mapping that takes into account all the relative information between the completions.\nI.e., the completions would not be ranked independently, but all would be ranked relative to each other.\nFor example, we can use a carefully prompted LLM-as-judge with all the reference completions together with training completions to obtain quantile rewards.\n\nWe will add this discussion to the paper, which we believe will significantly benefit practitioners in effectively leveraging QRPO and inspire new directions and open questions for future research.\n\n---\n\nWe greatly value the feedback and questions from the reviewer, and we remain open to any additional ideas or recommendations that could further enhance the quality and impact of our work."}, {"review_id": "uh24r2IERj", "reviewer": "Reviewer_u3X5", "summary": "QRPO is a novel alignment algorithm that is simple and works offline, but also allows the utilization of pointwise absolute rewards. The key idea is the use of a quantile transformation to make the partition function analytically tractable. The authors demonstrate consistent improvements over DPO, REBEL, and SimPO on both conversation and coding datasets.", "questions": "See weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-structured and clearly written, explaining the methodology and results effectively.\n- The quantile transformation is elegant and nicely bridges the gap between policy fitting and policy improvement methods. This enables offline methods to effectively leverage absolute reward models. The proof is rigorous. There is solid empirical evidence supporting the effectiveness of QRPO as compared to the baselines.\n\n**Weaknesses**\n- Experiments focus only on 7B-8B models. Given the importance of scale in modern LLM alignment, evaluation on larger models would significantly strengthen the claims.\n- QRPO's efficacy hinges on having a reasonably good reference sampler. If the reference policy is poorly calibrated, quantile estimates may misrank high‑quality outputs. A small experiment varying reference accuracy would clarify this sensitivity.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughtful and encouraging review and address the raised points below.\n\n---\n\n### **W1: Only 7B–8B Experiments**\n\n- We agree with the reviewer that given the potential impact of QRPO, experiments with larger models would make the claims stronger and accelerate confident adoption.\n- However, providing a **thorough scientific experimental protocol** such as the one we followed to formulate our conclusions would have been **computationally prohibitive with larger-scale models**.\n- We opted to put our budget and efforts into such a protocol, which at the 8B scale is already significant and allows comparison with previous papers (notably SimPO), **rather than conduct a more limited number of experiments with larger-scale models**.\n- Our protocol includes diverse datasets and models, and appropriately tuned baselines (over 200 trained models for each baseline), and used over 50k GPU hours, which is already very high for an academic project.\n\n---\n\n### **W2: Calibration of the Reference Policy**\n\n- The reference policy is typically taken to be the initial training model, so the calibration can be seen with respect to the initial training checkpoint.\n- If the training data is **purely offline and strictly much better than the initial checkpoint**, then we can perform **SFT on this data first to reduce the distribution shift** and make the initial checkpoint for QRPO, thus the reference policy. more \"calibrated\" to allow effective training.\n- This is what we have observed in our experiments in Table 6, for example, where models trained on Magpie-Air were **more performant by having a preliminary SFT on the chosen answer** (whose quantile is around 0.91) before QRPO. (Followed by the models trained offline without the preliminary SFT, and then the SFT models—so the performance is not due to SFT alone).\n- When the **data is collected iteratively** from the trained model, this data can have **quantiles that progressively become in the top quantile** of the reference model/initial checkpoint.\n- However, this saturation coincides with **convergence to the optimal policy**. This is because the optimal policy under the quantile reward remains supported (\"calibrated\") within the reference distribution, due to the “best-of-N” nature of this optimal distribution (Appendix M).\n- As a result, the policy cannot easily escape the support of the reference model, and the model is unlikely to produce completions that consistently outperform all reference completions. **Hence, there is no dramatic miscalibration before convergence**.\n- We also validate this in practice; our experiments never exceeded an average quantile reward of 0.95–0.98.\n- Moreover, when gains may start to diminish, observe that the **reference policy can be iteratively replaced by the most recent/performant checkpoint during training**, with new reference completions to estimate the quantile reward, which would be better \"calibrated\". This is a common recipe used with policy fitting methods. For example, iterative DPO was used to train the Llama 3 models.\n- We thank the reviewer for raising this point, and we will add this discussion to the paper to make it easier for practitioners to understand how to use QRPO.\n\n\nWe appreciate the reviewer’s insightful feedback and thoughtful points. We also remain receptive to any further suggestions or recommendations that may help strengthen the quality and impact of our work."}, {"review_id": "4Cis7TmP0A", "reviewer": "Reviewer_A3fZ", "summary": "This paper proposes “Quantile Reward Policy Optimization” (QRPO) -- a new algorithm / framework for optimizing the KL constrained objective of RL for LLMs, by analytically solving for the partition function under a quantile based reward.  This furthur can be used in both offline and online settings which is an advantage over strictly online methods like PPO and RLOO. The authors report results against multiple baselines on the AlpacaEval2 and UltraFeedback dataset showing the method outperforming baselines.", "questions": "- How sensitive is the algorithm to online vs offline data-distribution? \n- Do the authors have a hypothesis about whyQRPO is more robust to length-hacking? \n- I believe RLOO is left out of the reference citations eventhough it is referenced in the first figure\n\nSee weakneses for a list of additional questions / suggestions.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Well motivated method and theoretically intuitive\n- The paper is generally well written and clear -- especially on describing the subtle difference between widely used algorithms. \n\nWeaknesses: \n- As outlined in the limitations,  there aren't any ablations with the use of online data in training beyond Figure 3 which looks at generating data in one timestep, with a single behaviour policy. However, is it described as one of the key novelity points of the algorithm since it bridges the gap between online and offline algorithms. Thus, I think abalations giving insight into the dynamics of an online training setup would strengthen the paper. \n- As transforiming the reward is one of the key insights of the paper, it could benefit from a more extensive discussion of the reward distributions experimented on both for ArmorRM under different datasets, and the LeetCode test case distribution. In practice, how Gaussian were the original rewards? and how uniform were the transformed rewards?", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and appreciate the detailed comments and pertinent questions.\n\nWe address the raised concerns below:\n\n**W1: online ablation**\n- We had preliminary ablations in our early experiments where we added online data to offline data in QRPO training on the Magpie-Air dataset, and we observed a 26% greater reward improvement compared to pure offline training.\n- We thank the reviewer for the suggestion and will include this preliminary online–offline ablation result in the appendix to provide additional empirical support to the broader impact of QRPO.\n- For us, this was rather a consistency check and an expected behavior, as in general for policy fitting methods, online data is expected to help (Guo et. al., 2024).\n- We chose to focus on the offline regime (varying distribution shifts) as this is where the main difficulty arises: where existing policy improvement methods (PPO, etc.) completely fail and where QRPO makes its significant contribution compared to the rest of the policy fitting methods (DPO, etc.) that use pairwise data.\n- In fact, the inability to scale the RL training by leveraging offline data (offline or off-policy from previous iterations) is a key bottleneck in current RL post-training pipelines, for example, requiring the post-training of Grok-4 RL  to consume compute comparable to pre-training, largely due to costly online data sampling.\n- Hence, to \"bridge the gap\" between online and offline algorithms, we focused on validating the ability of QRPO to keep the characteristic of using pointwise rewards from the online-only policy improvement methods and enabling training with offline/off-policy data. \n- Our rigorous experimental protocol to evaluate QPRO for offline/off-policy regimes already required more than 50k GPU hours to complete, which is already very large for an academic lab, and we expect practitioners to expand on the preliminary online-offline ablation results that we had (which require 2 times more resources due to online sampling costs) to further highlight the online and semi-online performance of QRPO.\n\n**W2: reward distribution discussion**\n- QRPO does not rely on the specific distribution of the initial reward. The quantile reward has a uniform distribution independently of the shape of the initial reward distribution (see proof in Appendix E; there are no assumptions on the initial distribution shape).\n- We've verified that quantile rewards indeed follow a uniform distribution for all models, datasets, and tasks. We will add the corresponding plots to the appendix.\n- ArmorRM rewards for the general chat task are nearly Gaussian on both the Magpie-Air and UltraFeedback datasets. However, for some prompts, it is skewed, and this is why it is not correct to assume it to always be Gaussian and use a closed-form expression for the partition function Z(x) of an assumed distribution without applying a quantile transformation.\n- On coding dataset, since the reward is limited to the range from 0 to 1 (test case pass rate), the distribution is often non-Gaussian\n\n**Q1: sensitivity to data distribution**\n- We have discussed the online vs. offline in our answer to **W1**.\n- But what we find also interesting, and have experimented with thoroughly in the scaling ablation (Figure 3), is the sensitivity to offline vs. off-policy (that can be treated as relatively close to online):\n  - In our offline regime, data comes from a stronger model, or could be from human annotators, resulting in high-quality samples. Since these samples typically lie in the right tail of the reference model distribution (used for quantile transformation), the quantile reward estimates were less sensitive to the number of reference completions used for CDF estimation. Hence, with offline data, a small number of reference completions suffices for a good signal (we used 1 on Magpie-Air), but scaling reference completions yields limited gains.\n  - However, in the off-policy regime, however, data is generated from the reference model, so there is no distribution shift between the training data and reference model completions. This makes the noise in quantile reward estimates decrease significantly when increasing the number of reference completions. Hence, with off-policy data, scaling reference data pre-computation leads to larger performance improvements.\n\n**Q2: length bias mitigation**:\n- We hypothesize that the reduced length bias observed in QRPO (and similarly in REBEL) is primarily due to the methods being able to use the scalar ArmoRM reward signal, as this reward model was trained to penalize verbosity and avoid length exploitation.\n- Notably, while DPO and SimPO also rely on the same underlying reward model to generate preference pairs, the conversion of scalar rewards into binary preferences appears to discard part of the anti-verbosity signal, making these methods more susceptible to length-hacking.\n\n**Q3: RLOO citation**:\nThank you for catching this oversight. We will ensure that the appropriate citation for RLOO is included in the revised version of the paper.\n\nWe appreciate the reviewer’s constructive feedback, and we think their questions improved our work. We remain open to any further suggestions.\n\n[1] Guo, Shangmin, et al. \"Direct language model alignment from online ai feedback.\" arXiv preprint arXiv:2402.04792 (2024)."}, {"review_id": "Clj7u5Hlf6", "reviewer": "Reviewer_ipQ5", "summary": "This paper introduces Quantile Reward Policy Optimization (QRPO), a novel approach for offline alignment optimization. The key challenge it addresses is the estimation of the partition function in KL-regularized RL objectives. To circumvent this issue, QRPO leverages a quantile reward transformation, eliminating the need for explicit integration. Empirical results on 8B-scale models demonstrate its effectiveness in both chatting and code generation tasks.", "questions": "- Equation (9) – Uniform Partition Function?\n\nThe partition function in Equation (9) becomes uniform across prompts (independent of *x*), which seems counterintuitive—given that reward distributions vary significantly by prompt, a fixed baseline could lead to gradient imbalance during optimization. Without prompt-dependent normalization, does this formulation fail to properly stabilize training?\n\n- Equation (11) – Necessity of the Proposed Objective?\n\n   - A simpler baseline could minimize MSE: $(r(x, y) - \\log \\pi_{\\theta}(y|x))^2$, leading to a standard RL objective.\n\n   - Why is QRPO’s design necessary? What advantages does it provide over this simpler approach?\n\n- Evaluation & Performance Claims (Table 3)\n\n    - The results appear noisy—sometimes DPO outperforms QRPO.\n\n   - Can you provide a quantitative analysis of QRPO’s SOTA performance and the model behaviors? For example,\n\n       -  Under what conditions does QRPO differ significantly from other methods?\n\n       - What specific behaviors in the responses lead to its improvements?\n\n- Code Generation & Mathematical Reasoning\n\n   - For code generation: What specific problems does QRPO solve that other methods cannot?\n\n  -  For mathematical reasoning tasks:\n\n     - Have you evaluated QRPO on such tasks?\n\n      - Since KL regularization is often unnecessary here, does QRPO still provide benefits?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Advantages**\n- The paper is well-written and easy to follow in most parts.\n\n- The empirical results show promising performance improvements.\n\n**Disadvantages**\n\n- Lack of Motivation & Baseline Comparison\n\n   - There is insufficient justification for why Equation (6) must be explored in an offline setting.\n\n   - Direct offline optimization methods (e.g., policy gradient with KL regularization) could be viable alternatives—why is QRPO necessary?\n\n*Theoretical Gaps*\n\n- The approximation of the partition function introduces bias—does the proposed objective still converge to the optimal policy?\n\n- A theoretical analysis of the approximation error and convergence properties is missing.", "was_revised": true, "final_justification": "The authors have addressed my concerns and the method seems to provide a new formulation for learning the optimal policy.", "rebuttal": "We thank the reviewer for their time and effort in providing detailed feedback on our submission. \n\nBelow, we address each point raised:\n___\n\n## Motivation\n\nWe specifically focused on the offline regime because we believe that a major reason for the cost inefficiency of RL post-training is the **inability to scale by leveraging diverse offline data** (i.e., previously collected online samples or offline datasets). This is currently one of the main bottlenecks in RL post-training pipelines. For example, recent large-scale RL post-training efforts (e.g., Grok-4) required compute budgets **comparable to pre-training**, largely due to the sequential and expensive nature of online data generation, making such pipelines difficult to scale.\n\nTo bridge this gap, there is a strong need for an algorithm that can effectively leverage offline data, while also:\n- Operating directly on pointwise absolute rewards, like the SoTA policy gradient methods, and\n- Maintaining a simple and stable optimization objective (not requiring costly partition function estimation or auxiliary model-based approximations)\n___\n## Why QRPO is Needed\n\n1. The policy improvement algorithms (policy gradient, PPO, GRPO) are the main reason for this online inefficiency and cannot use offline data. It is well known that applying PPO-like methods with offline data leads to divergence.\n2. The current _policy fitting_ (DPO, SimPO, REBEL) algorithms can use offline data, but do not meet the requirements above. They optimize a relative reward signal via pairwise comparisons, which is not a good replacement for pointwise rewards as mentioned in our background section, due to suboptimal updates, probability mass drifts, and limited relative signal with verifiable reward settings.\n\nTo the best of our knowledge, QRPO is the first offline method combining pointwise absolute rewards and stable optimization.\n___\n## Theoretical Aspects\n\n### **No approximation (hence no bias) in** $Z_q(x)$\n\nIn QRPO, the partition function $Z_q(x)$ is computed **analytically and exactly** due to the quantile reward transformation. Thus, QRPO introduces **no approximation or bias**, ensuring a stable and principled optimization without extra modeling overhead.\n\n### **The only stochastic element is the empirical quantile reward** ${\\mathcal R}_q$\n\nFor each prompt $x$ we estimate\n\n$$\n\\hat{\\mathcal R}_ q(x,y)=\\frac{1}{n}\\sum_ {i=1}^{n} \\mathbf 1 \\bigl\\\\{ \\mathcal R(x,y_i)\\le \\mathcal R(x,y) \\bigr\\\\},\n\\qquad y_ i \\sim \\pi_ {\\text{ref}}(\\cdot|x).\n$$\n\n* **Unbiasedness.**\n  $\\mathbb E[\\hat{\\mathcal R}_q(x,y)|x,y]=\\mathcal R_q(x,y)$.\n\n* **Consistency and rate.**\n  By the Glivenko–Cantelli theorem the empirical CDF converges uniformly:\n\n  $$\n  \\sup_r\\bigl|\\hat F_{n}(x,r)-F_{\\text{ref}}(x,r)\\bigr| \\xrightarrow[n\\to\\infty]{a.s.} 0.\n  $$\n\n  The Dvoretzky–Kiefer–Wolfowitz inequality gives a finite-sample bound\n  $ \\Pr \\bigl(|\\hat{\\mathcal R}_q-\\mathcal R_q|>\\varepsilon\\bigr)\\le 2e^{-2n\\varepsilon^2}$.\n\nHence, the only error term we add is **zero-mean “label noise’’ that shrinks with $n$.**\n\n### **Unbiased quantile noise keeps the optimum intact**\n\nThe empirical quantile adds only zero‑mean, additive noise to the true target.\nIn an MSE objective, such noise contributes a constant variance term, leaving the arg‑min, and thus the desired policy, unchanged (standard least‑squares result).\nBecause gradients remain unbiased with finite variance, SGD with the usual diminishing step‑size still converges to that same optimum.\n\nWe will include this clarification in the paper to be more explicit about the QRPO convergence properties.\n___\n## Answers to the questions\n\n### **Q1**\n\nThe partition function is indeed independent of the prompt; however, the normalization **is prompt-dependent**.\nThe normalization in QRPO comes from the quantile transformation that recenters rewards with respect to the performance of the reference policy for **each prompt** separately, allowing $Z$ to be prompt-independent.\n___\n### **Q2**\n\nWe interpret the reviewer’s proposed baseline as $(r(x,y) - \\beta \\log(\\frac{p_ \\theta (x \\mid y)}{p_ {ref} (x \\mid y)}))^2$, since otherwise no KL regularization term would be present.\n\nThis loss can recover the correct **policy gradient** in an **online setting**, however, in the **offline setting**, it leads to biased solutions.\nFor example, consider a toy environment with:\n\n* Actions $a_1, a_2$ with rewards $r_1 = 1, r_2 = 2$\n* Reference policy probabilities $\\pi_\\text{ref}(a_1)=0.8, \\pi_\\text{ref}(a_2)=0.2$\n* Offline dataset uniformly sampling both actions.\n\nOptimizing Equation (1) yields the true optimum:\n\n$$\np^ * (a_1)=0.595,\\quad p^ * (a_2)=0.405\n$$\n\nwhile different squared-loss formulations give:\n\n| Objective                                                           | $p(a_1)$    | $p(a_2)$    |\n| - | - | - |\n| $(r - \\log p)^2$ *(initial loss)*                                   | 0.422     | 0.578     |\n| $(r - \\beta \\log(p/p_\\text{ref}))^2$ *(likely intended variant)*    | 0.542     | 0.458     |\n| **QRPO Loss** | **0.595** | **0.405** |\n___\n### **Q3**\n\n**Results Interpretation**\n\n**1. Metrics Priority**\n\n* Our primary evaluation metric is **AlpacaEval 2**, as it measures downstream performance on a dataset of unseen prompts and uses GPT-4 as a judge, providing a more reliable proxy for human preferences than the training reward model.\n* Within AlpacaEval 2, we emphasize the **Length-Controlled (LC) score**, shown to have the highest correlation with human judgments (Dubois et al., 2024) and to mitigate length bias that many algorithms exploit.\n* ArmoRM rewards (measured on a test set) are generally correlated with LC-AlpacaEval2 but we've observed that they are sometimes **hacked** by trained policies in a non-trivial way. While we manually reviewed outputs to filter obvious hacks, reward scores alone are less reliable.\n* For ArmoRM rewards, we similarly value the LC (Length-Controlled) score more since it mitigates length bias\n\n**2. Significance of Improvements**\n\n* On Llama 8B SFT, QRPO consistently outperforms all baselines. On the high-quality Magpie-Air dataset, the LC-AlpacaEval2 gap between QRPO and SimPO is of the same magnitude as the well-established DPO→SimPO gap, which is widely regarded as a substantial improvement in general chat tasks, confirming that QRPO delivers a significant gain.\n* On the Mistral Instruct model, we likely observe capacity saturation. Despite extensive hyperparameter tuning (same budget across methods), DPO, SimPO, and QRPO achieve **nearly identical scores** on Ultrafeedback, with differences within the noise level. Importantly, our results for DPO and SimPO significantly surpass those originally reported in the SimPO paper (Meng et. al., 2024), indicating we have pushed all methods considerably further than previous studies. Thus, the observed saturation likely reflects that the limits of Mistral Instruct’s performance were reached, rather than a limitation specific to QRPO.\n* On Magpie-Air with Mistral, similar saturation effects appear, with DPO and SimPO exploiting length bias for slight (non-significant) AlpacaEval gains. LC-AlpacaEval scores for QRPO remain higher, consistent with its improved robustness to verbosity bias (Figure 4).\n\n**Qualitative Analysis**\n\nTo robustly identify the specific behaviors that contribute to QRPO’s improvements, a thorough qualitative analysis (e.g., an extensive blinded case study with a large sample of examples) would be required. Such an analysis is beyond the scope of this work, where we focused on quantitative metrics that are known to correlate well with human preferences.  Additionally, since we exhaustively tuned all algorithms, observed differences in responses are subtle and difficult to characterize qualitatively without careful and systematic examination. We would appreciate it if the reviewer could clarify whether we correctly understood this question.\n___\n### **Q4**\n**Qualitative Analysis**\n\nSimilar to what we noted for chat tasks, such characteristics cannot be robustly identified within our scope. We are open to further clarification from the reviewer.\n\n**Mathematical Reasoning and Effect of KL Penalty**\n\n1. Role of KL Regularization\n* Prior work (Liu et al., 2025) observes that a KL penalty stabilizes training and leads to stronger solutions when the reference policy is periodically updated to recent checkpoints.\n* Gao et al. (2022) demonstrate that KL regularization primarily acts like early stopping, without shaping the training trajectory. These results support the view that periodic reference policy resets are sufficient to mitigate potential KL-related issues, making QRPO applicable to reasoning tasks as well.\n\n2. Applicability to Reasoning Tasks\n* We have not tested QRPO datasets because 7–8B models are limited in reasoning ability, making comparisons between methods potentially uninformative.\n* We chose to evaluate QRPO on code generation instead of mathematical reasoning because this is still a highly technical domain demonstrating QRPO's ability to handle tasks requiring structured reasoning and precision, while being easier and more robust to verify, requiring less computational budget (no judge), and more suitable for 7–8B models.\n\n[1] Pal, Arka, et al. \"Smaug: Fixing failure modes of preference optimisation with dpo-positive.\" arXiv preprint arXiv:2402.13228 (2024)\n\n[2] Liu, Mingjie, et al. \"Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.\" arXiv preprint arXiv:2505.24864 (2025).\n\n[3] Gao, Leo, John Schulman, and Jacob Hilton. \"Scaling laws for reward model overoptimization.\" International Conference on Machine Learning. PMLR, 2023."}, {"review_id": "EgnvrttKLX", "reviewer": "Reviewer_Ghhc", "summary": "The paper proposes a LLM alignment algorithm in the 'policy fitting' (i.e. DPO style) class of methods, that uses pointwise absolute rewards and offline data. Quantile Reward Policy Optimization (QRPO) first transforms each raw reward into its percentile within the reference model's reward distribution, yielding a transformed reward on $[0, 1]$. Due to this transform, the KL-regularized RL objective admits a closed-form log-likelihood target and removes noisy Monte-Carlo partition function estimates that certain prior methods suffered from. The authors propose a straightforward supervised regression between this quantile reward and the model's log-probability ratio to the reference policy. The method eliminates the need for a critic, pairwise preferences and depending on the setting the need for a reward model (using instead verifiable rewards, unit test pass rates, single human scores). The authors run experiments on instruction-following, maths and coding tasks, reporting that QRPO matches or surpasses DPO and other baselines with lower length bias. The learning signal can however plateau once the learned policy overtakes the cached reference completions, and reward signal magnitude is effectively discarded.", "questions": "1. Have you quantified how quickly gradients vanish when the learned policy's reward surpasses all reference completions' rewards (i.e. $\\mathcal{R}(x, y') \\le \\mathcal{R}(x, y)\\ \\forall y' \\sim \\pi_{\\text{ref}}(y' \\vert x)$? Adding this detail, or further illustrating how QRPO might be used in more online settings (that is, periodically updating the reference and re-sampling the empirical CDF) would clarify its practicality in long-training regimes or tasks firmly outside the initial reference model's distribution.\n2. How do both RLOO and GRPO compare to QRPO, seeing as they seem to occupy similar design spaces to (they accept point-wise rewards and address gradient variance reduction)? I am not expecting you to re-run experiments with these on account of the additional work, but if you can that would be great, or failing that a short discussion of their characteristics (compute requirements, ability to run offline or online, expected performance, etc) would be a nice addition to the paper.\n3. What is the impact of discarding the scale of the reward? In many environments, the pointwise reward contains magnitude information that can help provide richer learning signals, for instance differentiating a good from a great completion (distance to goal, safety penalties, code performance). As I understand it, the price of the quantile transform used to get a closed-form partition function is that we discard the reward magnitude information and instead only rank it relative to the reference model's completions (i.e. the policy is merely pushed to outrank its own reference model). Could you comment further on the implications of this; for example washing out extreme negatives (to the detriment of safety) or extreme positives (to the detriment of learning speed).", "limitations": "While the two main limitations with the work are brought up (the quantile reward re-shaping the reward distribution, offline only focus), this is done so with a brevity that may not be commensurate with their significance.\n\nWhile a more rigorous investigation of these may be postponed for future work, I do believe it would be valuable to discuss the potential drawbacks of the re-shaped reward distribution and loss of reward magnitude in the main text in some greater depth, dwelling on potential mitigation strategies for practitioners.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Quality\nThe paper contains a good section on preliminary work, with a clean and relatively easy to follow derivation from the KL-regularized RL objective to their closed form regression loss. The auxiliary proofs in the appendix for the uniform CDF and analytic partition function on which the work rests are clean and re-use standard probability results.\n\nThe experimental work is careful and high-quality, and appears to follow rigorous methodology to appropriately tune baselines, re-use appropriate reward models, and report numbers with uncertainty estimates throughout.\n\nGiven the prevalence of 'policy improvement' methods such as GRPO and RLOO, which are prominently mentioned in Table 1, and serve similar purposes as post-training algorithms that use pointwise rewards and work online (and increasingly offline too), I believe these would make valuable additions to the comparisons in the paper. While I recognize the additional work and perhaps scope creep - and hence I won't consider this point strongly in my score - it would be very interesting to see how QRPO compares with GRPO in the Llama 8B LeetCode run reported in Table 2. This seems particularly relevant owing to the similarities one might draw between the computational requirements of both methods: with the group rollouts in GRPO and and $n$ completions $y_{i, j} \\sim \\pi_{\\text{ref}}(\\cdot \\vert x_{i})$ used in QRPO.\n\n# Clarity\n\nThe paper is for the most part clearly written and easy to follow. Figure 2 is somewhat hard to read and would benefit from being typeset properly.\n\n# Significance\n\nThe proposed method does introduce some genuinely useful properties: dropping the pairwise preference data requirement of many previous direct alignment algorithms is beneficial for applications or data pipelines that only measure a pointwise reward signal (thumbs up/down, likert scale), while the offline nature of the algorithm may help to amortize pre-computation costs, and being less prone to length bias is also beneficial. However, there already exist prior alignment algorithms with these qualitative properties for practitioners. Much of the introduction and 3-axis taxonomy is seemingly spent to explain why this work is unique: indeed, offline policy-fitting, with absolute scalar rewards and no auxiliary networks (for estimating $Z$, a critic, or otherwise) *is* useful, but it is only narrowly unique in the alignment space.\n\nHowever, the reported performance is at times only marginally superior to DPO or SimPO which slightly limits the significance of the work. Moreover, the exclusive focus on offline policy fitting comparisons, while in the authors' own words \"*PF methods are losing applicability*\" in the face of progress such as agent reasoning on verifiable reward tasks also limits the significance of the work: while the work addresses the use of absolute reward signals, it seems likely that online data collection will continue to play a significant role in post-training as the field appears to be moving from aligning chatbots to training reasoning-style agents.\n\n# Originality\n\nThe main original contribution in QRPO appears to be the \"quantile reward trick\" (Eqns 7 -- 10), where the quantile transformation makes the exponentiated reward's partition function prompt-independent, which turns the KL-regularized RL optimum into a closed-form regression target.  While there are similar sounding techniques in other areas of ML (e.g. probability-integral / CDF transforms are used in density estimation methods like normalizing flows so the base distribution's log-density has a tractable normalizer, or quantile regression in QR-DQN), the approach in this paper is original and solves a problem in direct alignment algorithms. Moreover, the approach set out in this paper (i.e. converting an arbitrary scalar reward into a bounded, uniform signal whose partition function is a constant) could be re-used elsewhere where we face energy-based or KL objectives with nuisance normalizers.\n\n**Nits:**\n- line 56: \"effective on a scale\" -> \"effectiive at scale\"\n- line 301: \"we choose to compare to DPO\"\n- Paragraph at line 496: repeated sentence\n- Eq 20: d(F_{ref}(r)) -> F_{ref}(r, x)", "was_revised": true, "final_justification": "My initial evaluation of the work contained some misunderstanding about the originality and significance of the work. This was clarified in the discussion, and I consider this work to be a useful and novel offline alignment algorithm for cases where only pointwise rewards are available. The paper is technically sound with a good evaluation in the offline setting. While I do believe that the paper would be improved by addressing a couple of points which the authors consider out of scope and defer to future work (expanding on how and when it is useful to apply a transform on top of the quantile reward, as well as performing an evaluation online to match the claims that this method can work effectively in online settings as well as offline), I also believe that the contributions made as the paper stands are substantial enough to recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for the detailed and constructive feedback, as well as for highlighting the strengths of our work.\n\nBelow, we address each point raised:\n___\n## Vital role of offline RL algorithms for post-training pipelines\nWe specifically focused on the offline regime because we believe that a major reason for the cost inefficiency of online RL post-training is the **inability to scale** by leveraging diverse offline data (i.e., previously collected samples or offline datasets). This is currently one of the **main bottlenecks** in RL post-training pipelines. For example, recent large-scale RL post-training efforts (e.g., Grok-4) required compute budgets **comparable to pre-training**, largely due to the sequential and expensive nature of online data generation. \n\nThus, even though online data collection will likely continue to play a significant role in post-training pipelines, the effective scaling of training requires an offline-capable algorithm that can significantly amortize this data generation cost by reusing the data generated in previous iterations.\n___\n## Policy improvement algorithms cannot be a viable option for offline training\nPolicy improvement methods (e.g., PPO, GRPO, RLOO) are **inherently online** algorithms. While slight off-policy training is possible via importance sampling corrections, this approach introduces high-variance gradient estimates, making optimization unstable at strong off-policiness. More importantly, **true offline** training is **still infeasible** with importance sampling because it requires knowing the sampling probabilities of each data point under the behavior policy, which is typically unknown for offline datasets collected from arbitrary sources.\n\nTherefore, we would be grateful if the reviewer could provide examples for their claim that \"there already exist prior alignment algorithms with these qualitative properties for practitioners\" (in the context of offline pointwise algorithms).\n\nIn addition, could the reviewer elaborate on \"is only narrowly unique in the alignment space\"?\nGiven that it covers an essential gap in stable, offline (or highly off-policy) pointwise methods, which can largely impact the scaling of RL pipelines.\n\n---\n\n## Results interpretation\nIn some cases, performance can seem marginally superior to DPO or SimPO for some metrics, but we elaborate on their significance, noting that we primarily rely on AlpacaEval 2, which evaluates downstream generalization on an unseen dataset with GPT-4 as a judge and is a more robust proxy for human preferences than the reward model. Importantly, we focus on its Length-Controlled (LC) score as it both mitigates length bias and shows the highest correlation with human judgments.\nThen come ArmoRM rewards, which correlate with AlpacaEval2 but can be \"hacked\" by trained policies, making them less reliable. Among average reward scores, LC-ArmoRM scores remain more informative since they reduce verbosity bias.\n\nWith this metric priority in mind,\n- On Llama 8B SFT, QRPO consistently outperforms all baselines. On the high-quality Magpie-Air dataset, the LC-AlpacaEval2 gap between QRPO and SimPO is of the same magnitude as the well-established DPO→SimPO gap, which is widely regarded as a substantial improvement in general chat tasks, confirming that QRPO delivers a significant gain.\n- On the Mistral Instruct model, we likely observe capacity saturation. After extensive hyperparameter tuning, DPO, SimPO, and QRPO achieve **nearly identical scores** on UltraFeedback, with differences within the noise level. Importantly, our results for DPO and SimPO significantly surpass those originally reported in the SimPO paper (Meng et. al., 2024), indicating we have pushed all methods considerably further than previous studies. Thus, the observed saturation likely reflects that the limits of Mistral Instruct’s performance were reached, rather than a limitation specific to QRPO.\n- On Magpie-Air with Mistral, similar saturation effects appear, with DPO and SimPO exploiting length bias for non-significant AlpacaEval gains. LC-AlpacaEval scores for QRPO remain higher, consistent with its improved robustness to verbosity bias (Fig. 4).\n---\n## Answers to the questions\n**Q1**\n\nFirst, note that **gradient vanishing does not occur simply due to quantile reward saturation ($\\mathcal{R}_q(x, y)=1$)**. Even when $\\mathcal{R}_ q=1$, gradients remain non-zero unless the ratio $\\pi_{\\theta}/\\pi_{\\text{ref}}=1/\\beta$ is exactly reached (from plugging $\\mathcal{R}_q=1$ into the optimal policy expression and approximating $Z_q(x)\\approx \\beta e^{1/\\beta}$ for small $\\beta$). Thus, gradient vanishing coincides strictly with policy convergence, not reward saturation.\n\nHowever, widespread quantile reward saturation (many samples achieving $\\mathcal{R}_q=1$) can still reduce the magnitude of the reward signal, potentially slowing training. Clarifying when this can occur:\n* In the **offline setting**, quantile rewards are computed from a static dataset that is crafted by the user. If this dataset contains samples that would saturate the quantile, then a preliminary SFT on this data can be performed to reduce the gap/distribution shift of the initial checkpoint/reference policy for QRPO.\n* In the **online setting**, improved policy outputs may frequently exceed reference completions, causing a quantile reward saturation.\n\nHowever, **this saturation coincides with (or follows) convergence to the optimal policy**. This is because the optimal policy under the quantile reward remains **supported within the reference distribution**, due to the “best-of-N” nature of this optimal distribution (Appendix M).\n\nAs a result, the policy cannot easily escape the support of $\\pi_{\\text{ref}}$, and the model is unlikely to produce completions that consistently outperform all reference completions. Hence, full quantile reward saturation typically does not occur before convergence.\nWe also validate this in practice; our experiments never exceeded an average quantile reward of 0.95–0.98, preserving informative gradients.\n\nMoreover, one can still improve beyond the optimal policy given a reference by either lowering $\\beta$ or performing several iterations \nperiodically updating the reference model to the most recent trained policy and re-estimating the quantile CDF.\nThis is a common recipe used with policy fitting methods, for example, used with DPO for the Llama 3 models.\n___\n**Q2**\n\nWe do not consider RLOO and GRPO to occupy the same design space as QRPO, as neither supports fully offline training, which is a key property of QRPO. Nevertheless, it is useful to compare their characteristics in the online setting.\n\nWe opted for a rigorous experimental protocol that required 50k GPU hours to complete to show the effectiveness of QRPO in offline/off-policy regimes and support its unique contribution.\nAs the reviewer acknowledges, adding a thorough evaluation of online and semi-online regimes is outside our scope and would require 2 times more resources due to online sampling costs. We therefore provide a theoretical comparison:\n1. Theoretical Similarities:\n- QRPO’s gradient in the online case is equivalent to a **policy gradient with a baseline**, making it closely related to GRPO and RLOO.\n- The main difference lies in the reward transformation: QRPO optimizes a quantile-based reward, while GRPO and RLOO use the raw signal.\n- Both approaches can be brought closer by either applying an inverse-CDF transform (Eq. 12) to QRPO’s quantiles to approximate the raw scale or, conversely, by using quantile rewards in GRPO.\n\n2. Differences: Optimal Baseline in QRPO\n- QRPO provides a closed-form baseline, $\\beta \\log{Z_ q}$, which equals the expected reward under the optimal policy:\n$\\beta \\log{Z_q} = \\mathbb{E}_ {y \\sim \\pi^ * (\\cdot \\mid y)} [r(x,y)], \\quad r(x,y) = \\mathcal{R}_ q (x,y) - \\beta \\log{\\frac{\\pi_ \\theta(y \\mid x)}{\\pi_ {ref}(y \\mid x)}}$. This guarantees minimum gradient variance as the learned policy approaches the optimum, while in GRPO and RLOO baseline is typically an estimator with additional variance.\n- This baseline is what allows QRPO to work both online and offline, bridging the policy improvement gradient and the policy fitting gradient.\n\n3. Online/Offline Compute Efficiency:\n - In the offline case, QRPO is unmatched, the quantile rewards and the reference model probabilities can be precomputed, therefore only requiring to have the training policy during training. These are the same requirements as SFT and represent ~3x less memory compared to GRPO (at least online) which needs the reference model, sampler, and training policy.\n- Online:\n  - QRPO can precompute $n$ reference completions per prompt once (unless the reference is updated), reusing them throughout training.\n  - GRPO and RLOO require fresh completions at every gradient step, leading to considerably higher compute costs due to repeated online sampling.\n___\n**Q3**\n\nWe distinguish two distinct interpretations of the question: (i) preserving different reward ranges for different prompts, (ii) extracting the information from outliers.\n- (i): QRPO allows for a reward range preserving by applying the additional transform on top of a quantile reward (as proposed in Eq. 12). This transformation can be prompt-specific, e.g., it can be a multiplication by the std of the reference model rewards for each prompt.\n- (ii): QRPO is indeed insensitive to the outliers with respect to the reference rewards distribution (i.e., it can't distinguish them from normal boundary values). However, one can try to mitigate this issue by picking a special transformation function (eq. 12). For instance, if one needs to account for negative outliers (i.e., safety penalties), one can use a log transformation, thus heavily penalizing any sample with near 0 quantile reward.\n\nFinally, we note that other algorithms based on normalization (e.g., GRPO) also discard the scale information and reduce sensitivity to outliers."}]

Author rebuttal:

Meta-review: This paper presents Quantile Reward Policy Optimization (QRPO), a RL method that aim to better leverage pointwise reward signals. The main challenge is the intractable partition function, and QRPO proposes to apply a quantile transformation to obtain a tractable partition function. The derivations are clear, and the method’s motivation and distinction from related work were clarified during the author–reviewer discussion, with the authors emphasizing robustness benefits of quantile weighting and theoretical guarantees. Reviewers appreciated the solid theoretical foundation but raised concerns about the modest empirical gains and limited experimental scope, which the authors addressed with additional ablations and explanations. Quantile transformation in QRPO is insightful, although estimate the quantile itself may still require "relative" computation among rewards. Overall, I recommend accept for this paper.
Decision: Accept (poster)