Title: Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning fine-tuning

alignment

offline alignment

direct alignment algorithms

large language model
Abstract: Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO.
In contrast, simpler methods that can leverage offline or off-policy data, such as DPO and REBEL, are limited to learning from preference pairs or relative signals.
To bridge this gap, we introduce Quantile Reward Policy Optimization (QRPO), which learns from pointwise absolute rewards while preserving the simplicity and offline applicability of DPO-like methods.
QRPO uses quantile rewards to enable regression to the closed-form solution of the KL-regularized RL objective.
This reward yields an analytically tractable partition function, removing the need for relative signals to cancel this term.
Moreover, QRPO scales with increased compute to estimate quantile rewards, opening a new dimension for pre-computation scaling.
Empirically, QRPO consistently achieves top performance on chat and coding evaluations—reward model scores, AlpacaEval 2, and LeetCode—compared to DPO, REBEL, and SimPO across diverse datasets and 8B-scale models.
Finally, we find that training with robust rewards instead of converting them to preferences induces less length bias.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a LLM alignment algorithm in the 'policy fitting' (i.e. DPO style) class of methods, that uses pointwise absolute rewards and offline data. Quantile Reward Policy Optimization (QRPO) first transforms each raw reward into its percentile within the reference model's reward distribution, yielding a transformed reward on $[0, 1]$. Due to this transform, the KL-regularized RL objective admits a closed-form log-likelihood target and removes noisy Monte-Carlo partition function estimates that certain prior methods suffered from. The authors propose a straightforward supervised regression between this quantile reward and the model's log-probability ratio to the reference policy. The method eliminates the need for a critic, pairwise preferences and depending on the setting the need for a reward model (using instead verifiable rewards, unit test pass rates, single human scores). The authors run experiments on instruction-following, maths and coding tasks, reporting that QRPO matches or surpasses DPO and other baselines with lower length bias. The learning signal can however plateau once the learned policy overtakes the cached reference completions, and reward signal magnitude is effectively discarded.\n\nSTRENGTHS AND WEAKNESSES:\n# Quality\nThe paper contains a good section on preliminary work, with a clean and relatively easy to follow derivation from the KL-regularized RL objective to their closed form regression loss. The auxiliary proofs in the appendix for the uniform CDF and analytic partition function on which the work rests are clean and re-use standard probability results.\n\nThe experimental work is careful and high-quality, and appears to follow rigorous methodology to appropriately tune baselines, re-use appropriate reward models, and report numbers with uncertainty estimates throughout.\n\nGiven the prevalence of 'policy improvement' methods such as GRPO and RLOO, which are prominently mentioned in Table 1, and serve similar purposes as post-training algorithms that use pointwise rewards and work online (and increasingly offline too), I believe these would make valuable additions to the comparisons in the paper. While I recognize the additional work and perhaps scope creep - and hence I won't consider this point strongly in my score - it would be very interesting to see how QRPO compares with GRPO in the Llama 8B LeetCode run reported in Table 2. This seems particularly relevant owing to the similarities one might draw between the computational requirements of both methods: with the group rollouts in GRPO and and $n$ completions $y_{i, j} \\sim \\pi_{\\text{ref}}(\\cdot \\vert x_{i})$ used in QRPO.\n\n# Clarity\n\nThe paper is for the most part clearly written and easy to follow. Figure 2 is somewhat hard to read and would benefit from being typeset properly.\n\n# Significance\n\nThe proposed method does introduce some genuinely useful properties: dropping the pairwise preference data requirement of many previous direct alignment algorithms is beneficial for applications or data pipelines that only measure a pointwise reward signal (thumbs up/down, likert scale), while the offline nature of the algorithm may help to amortize pre-computation costs, and being less prone to length bias is also beneficial. However, there already exist prior alignment algorithms with these qualitative properties for practitioners. Much of the introduction and 3-axis taxonomy is seemingly spent to explain why this work is unique: indeed, offline policy-fitting, with absolute scalar rewards and no auxiliary networks (for estimating $Z$, a critic, or otherwise) *is* useful, but it is only narrowly unique in the alignment space.\n\nHowever, the reported performance is at times only marginally superior to DPO or SimPO which slightly limits the significance of the work. Moreover, the exclusive focus on offline policy fitting comparisons, while in the authors' own words \"*PF methods are losing applicability*\" in the face of progress such as agent reasoning on verifiable reward tasks also limits the significance of the work: while the work addresses the use of absolute reward signals, it seems likely that online data collection will continue to play a significant role in post-training as the field appears to be moving from aligning chatbots to training reasoning-style agents.\n\n# Originality\n\nThe main original contribution in QRPO appears to be the \"quantile reward trick\" (Eqns 7 -- 10), where the quantile transformation makes the exponentiated reward's partition function prompt-independent, which turns the KL-regularized RL optimum into a closed-form regression target.  While there are similar sounding techniques in other areas of ML (e.g. probability-integral / CDF transforms are used in density estimation methods like normalizing flows so the base distribution's log-density has a tractable normalizer, or quantile regression in QR-DQN), the approach in this paper is original and solves a problem in direct alignment algorithms. Moreover, the approach set out in this paper (i.e. converting an arbitrary scalar reward into a bounded, uniform signal whose partition function is a constant) could be re-used elsewhere where we face energy-based or KL objectives with nuisance normalizers.\n\n**Nits:**\n- line 56: \"effective on a scale\" -> \"effectiive at scale\"\n- line 301: \"we choose to compare to DPO\"\n- Paragraph at line 496: repeated sentence\n- Eq 20: d(F_{ref}(r)) -> F_{ref}(r, x)\n\nQUESTIONS:\n1. Have you quantified how quickly gradients vanish when the learned policy's reward surpasses all reference completions' rewards (i.e. $\\mathcal{R}(x, y') \\le \\mathcal{R}(x, y)\\ \\forall y' \\sim \\pi_{\\text{ref}}(y' \\vert x)$? Adding this detail, or further illustrating how QRPO might be used in more online settings (that is, periodically updating the reference and re-sampling the empirical CDF) would clarify its practicality in long-training regimes or tasks firmly outside the initial reference model's distribution.\n2. How do both RLOO and GRPO compare to QRPO, seeing as they seem to occupy similar design spaces to (they accept point-wise rewards and address gradient variance reduction)? I am not expecting you to re-run experiments with these on account of the additional work, but if you can that would be great, or failing that a short discussion of their characteristics (compute requirements, ability to run offline or online, expected performance, etc) would be a nice addition to the paper.\n3. What is the impact of discarding the scale of the reward? In many environments, the pointwise reward contains magnitude information that can help provide richer learning signals, for instance differentiating a good from a great completion (distance to goal, safety penalties, code performance). As I understand it, the price of the quantile transform used to get a closed-form partition function is that we discard the reward magnitude information and instead only rank it relative to the reference model's completions (i.e. the policy is merely pushed to outrank its own reference model). Could you comment further on the implications of this; for example washing out extreme negatives (to the detriment of safety) or extreme positives (to the detriment of learning speed).\n\nLIMITATIONS:\nWhile the two main limitations with the work are brought up (the quantile reward re-shaping the reward distribution, offline only focus), this is done so with a brevity that may not be commensurate with their significance.\n\nWhile a more rigorous investigation of these may be postponed for future work, I do believe it would be valuable to discuss the potential drawbacks of the re-shaped reward distribution and loss of reward magnitude in the main text in some greater depth, dwelling on potential mitigation strategies for practitioners.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Quantile Reward Policy Optimization (QRPO), a novel approach for offline alignment optimization. The key challenge it addresses is the estimation of the partition function in KL-regularized RL objectives. To circumvent this issue, QRPO leverages a quantile reward transformation, eliminating the need for explicit integration. Empirical results on 8B-scale models demonstrate its effectiveness in both chatting and code generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Advantages**\n- The paper is well-written and easy to follow in most parts.\n\n- The empirical results show promising performance improvements.\n\n**Disadvantages**\n\n- Lack of Motivation & Baseline Comparison\n\n   - There is insufficient justification for why Equation (6) must be explored in an offline setting.\n\n   - Direct offline optimization methods (e.g., policy gradient with KL regularization) could be viable alternatives—why is QRPO necessary?\n\n*Theoretical Gaps*\n\n- The approximation of the partition function introduces bias—does the proposed objective still converge to the optimal policy?\n\n- A theoretical analysis of the approximation error and convergence properties is missing.\n\nQUESTIONS:\n- Equation (9) – Uniform Partition Function?\n\nThe partition function in Equation (9) becomes uniform across prompts (independent of *x*), which seems counterintuitive—given that reward distributions vary significantly by prompt, a fixed baseline could lead to gradient imbalance during optimization. Without prompt-dependent normalization, does this formulation fail to properly stabilize training?\n\n- Equation (11) – Necessity of the Proposed Objective?\n\n   - A simpler baseline could minimize MSE: $(r(x, y) - \\log \\pi_{\\theta}(y|x))^2$, leading to a standard RL objective.\n\n   - Why is QRPO’s design necessary? What advantages does it provide over this simpler approach?\n\n- Evaluation & Performance Claims (Table 3)\n\n    - The results appear noisy—sometimes DPO outperforms QRPO.\n\n   - Can you provide a quantitative analysis of QRPO’s SOTA performance and the model behaviors? For example,\n\n       -  Under what conditions does QRPO differ significantly from other methods?\n\n       - What specific behaviors in the responses lead to its improvements?\n\n- Code Generation & Mathematical Reasoning\n\n   - For code generation: What specific problems does QRPO solve that other methods cannot?\n\n  -  For mathematical reasoning tasks:\n\n     - Have you evaluated QRPO on such tasks?\n\n      - Since KL regularization is often unnecessary here, does QRPO still provide benefits?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes “Quantile Reward Policy Optimization” (QRPO) -- a new algorithm / framework for optimizing the KL constrained objective of RL for LLMs, by analytically solving for the partition function under a quantile based reward.  This furthur can be used in both offline and online settings which is an advantage over strictly online methods like PPO and RLOO. The authors report results against multiple baselines on the AlpacaEval2 and UltraFeedback dataset showing the method outperforming baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Well motivated method and theoretically intuitive\n- The paper is generally well written and clear -- especially on describing the subtle difference between widely used algorithms. \n\nWeaknesses: \n- As outlined in the limitations,  there aren't any ablations with the use of online data in training beyond Figure 3 which looks at generating data in one timestep, with a single behaviour policy. However, is it described as one of the key novelity points of the algorithm since it bridges the gap between online and offline algorithms. Thus, I think abalations giving insight into the dynamics of an online training setup would strengthen the paper. \n- As transforiming the reward is one of the key insights of the paper, it could benefit from a more extensive discussion of the reward distributions experimented on both for ArmorRM under different datasets, and the LeetCode test case distribution. In practice, how Gaussian were the original rewards? and how uniform were the transformed rewards?\n\nQUESTIONS:\n- How sensitive is the algorithm to online vs offline data-distribution? \n- Do the authors have a hypothesis about whyQRPO is more robust to length-hacking? \n- I believe RLOO is left out of the reference citations eventhough it is referenced in the first figure\n\nSee weakneses for a list of additional questions / suggestions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
QRPO is a novel alignment algorithm that is simple and works offline, but also allows the utilization of pointwise absolute rewards. The key idea is the use of a quantile transformation to make the partition function analytically tractable. The authors demonstrate consistent improvements over DPO, REBEL, and SimPO on both conversation and coding datasets.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The paper is well-structured and clearly written, explaining the methodology and results effectively.
- The quantile transformation is elegant and nicely bridges the gap between policy fitting and policy improvement methods. This enables offline methods to effectively leverage absolute reward models. The proof is rigorous. There is solid empirical evidence supporting the effectiveness of QRPO as compared to the baselines.

**Weaknesses**
- Experiments focus only on 7B-8B models. Given the importance of scale in modern LLM alignment, evaluation on larger models would significantly strengthen the claims.
- QRPO's efficacy hinges on having a reasonably good reference sampler. If the reference policy is poorly calibrated, quantile estimates may misrank high‑quality outputs. A small experiment varying reference accuracy would clarify this sensitivity.

QUESTIONS:
See weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-23

comments: SUMMARY:
This paper presents Quantile Reward Policy Optimization, a novel policy fitting method for fine-tuning large language models  that sidesteps a longstandong obstacle in  alignment: the intractability of the partition function in KL-regularized objectives when using absolute rewards. It introduces an elegant and computationally efficient solution by applying a quantile transformation to the reward distribution, rendering the partition function analytically tractable in closed form. Empirical results demonstrate QRPO’s superiority across chat and code generation tasks when compared to strong baselines like DPO, REBEL, and SimPO.

STRENGTHS AND WEAKNESSES:
1. From a theoretical standpoint, the most remarkable contribution of this work is the closed-form expression for the partition function Z(x) after applying a quantile transformation to the reward. The tractability of Z(x), which typically renders absolute reward regression intractable, is a longstanding bottleneck in regularized RL. This is an elegant and rigorous contribution that simplifies the RL fine-tuning objective while preserving its core structure.

2. The authors extend the novel theoretical insight and they validate the method comprehensively with experiments on two major alignment datasets and a code generation task, showing that QRPO outperforms state-of-the-art methods under comparable settings.

Since I am from a primarily a theory background, I am unable to evaluate the robustness of the empirical results in a rigorous manner.

QUESTIONS:
1. Have you explored any other transformations with regards that allow the framework to a multi level preference reward signal?

LIMITATIONS:
Due to my limited exposure to applied RL I am unable to analyze the limitations of the empirical results.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-23