Title: Beyond Expectations: Quantile-Guided Alignment for Risk-Calibrated Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Alignment

Large language models
Abstract: Large language models can generate rare but catastrophic outputs, such as harmful conversations or insecure code. Existing Reinforcement Learning from Human Feedback (RLHF) typically maximizes average reward, leaving high-risk tail events insufficiently controlled. We introduce Quantile‑Guided Alignment (QA), a framework that allows users to specify desired improvements at any quantile—individually or across multiple reward dimensions—thus shifting the distribution of outputs with finer control toward safer, more desirable outcomes. The method extends standard RLHF via an augmented reward formulation that enforces quantile constraints. Experiments on conversation and code‐generation tasks show that quantile alignment significantly enhances quality at targeted tails while maintaining overall performance. The results position QA as a principled route to risk‑calibrated language models with tail‑focused alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nQuantile-Guided (QA) alignment aims to improve a language model’s performance at any desired quantile over the reference language model. The authors formally define this as a KL-regularized optimization problem under quantile constraints. They transform the problem into a convex optimization over dual variables, which they solve using Monte Carlo estimation. These dual variables are then used to construct a KL-regularized reward function, which is optimized via PPO. The approach is evaluated on the Anthropic HH dataset and the HumanEval dataset for code generation, using the OPT-1.3B and CodeGen-350M models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Quantile alignment could be valuable in scenarios where we aim to generally improve generation quality while ensuring safety constraints are not violated.\n- The proposed quantile-based reward function is inherently useful, particularly in multi-reward settings, as it is robust to the scale of reward values.\n- The paper is largely well-written, and both the main idea and experimental setup are easy to follow.\n\nWeaknesses\n\n- There is often an inherent trade-off between quality and safety metrics, as also observed in the paper’s experiments (lines 219–320), where jointly improving helpfulness and harmlessness is not always feasible. A more realistic use case for the proposed method might be to focus on improving helpfulness while enforcing safety guardrails (e.g., ensuring the lower tail does not worsen, as motivated in the introduction), rather than aiming for significant gains in harmlessness as in the current experimental setup. In this context, several relevant papers—such as [Risk-Averse Fine-Tuning of Large Language Models](https://proceedings.neurips.cc/paper_files/paper/2024/file/c182ec594f38926b7fcb827635b9a8f4-Paper-Conference.pdf), [Enhancing LLM Safety via Constrained Direct Preference Optimization](https://arxiv.org/pdf/2403.02475), and [SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety](https://arxiv.org/pdf/2505.20065)—are not discussed or compared against. It is possible that QA is unnecessarily complex (involving a two-stage online alignment) for these scenarios and that similar goals could be achieved more efficiently, but additional experiments would be needed to assess this.\n- The computational efficiency of the approach is underexplored. While the authors note in line 161 that a few thousand samples “typically” suffice for stable estimates of $g(\\lambda)$, they do not specify how stability is determined, how sample requirements vary across tasks, or how much this first stage of estimating $\\lambda$ increases total training time.\n- The practical implications of “Continuous Quantile Alignment” remain unclear. Unless I am mistaken, it appears to be a theoretical extension that is not evaluated in the experiments section, and clarification from the authors would be helpful.\n\nQUESTIONS:\nPlease refer to the weaknesses section for more context.\n- Could the authors provide a comparison with related work under a more realistic setup (e.g., improving helpfulness while constraining the lower tail of harmfulness)?\n- Could the authors provide an analysis of the computational efficiency of the approach across different tasks and models?\n- Could the authors clarify how continuous quantile alignment could be applied in practice?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper proposes to extend RLHF with quantile constraints with the goal of eliminating tail behaviors that are not eliminated by standard RLHF. To this end, the authors propose a framework to solve the constrained optimization problem using the method of Lagrange Multipliers and rely on strong duality properties. 

The effectiveness of the proposed approach is shown in the context of a safety and a code generation task (using two different small scale models < 1.3B parameters) with minimal decrease in responses diversity and coherence.

STRENGTHS AND WEAKNESSES:
**Strengths**:
- Well written and organized
- Well motivated
- Motivated theoretically and backed by empirical experiments in code generation and helpfulness/harmlessness datasets

**Weaknesses**:
- **Limited generalizability of the empirical findings**: experiments consider 2 models of smaller size (<1.3B), and only 2 datasets.
- **Few insights concerning the generations of the models when trained using the proposed method and the baselines**. 
- Missing error bars to Figures 1(c) and 2(c) which could be insightful to understand whether differences are significant or not.
- Missing efficiency comparisons with baselines
- The authors mention decoding-based approaches to steering/controlling model's behaviors but do not cite any work. The authors  should consider summarizing the trade-offs between the decoding-based and training-based and cite either a survey that comments on steering/controllability approaches or cite a few proeminent papers.

QUESTIONS:
Typos:
- line 63: "build" --> "builds"
- line 80: $\mathcal{P}$ is defined but not introduced in Equation (1) 
- line 84: "from It" --> "It"

LIMITATIONS:
yes, in appendix
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors point out that standard RLHF optimizes only average reward, leaving rare but critical failure modes unmitigated. They argue that safety and security-sensitive applications require explicit control over the fraction of outputs meeting user-specified thresholds. \n\nTo address this, they introduce Quantile-Guided Alignment, which transforms each quantile requirement into a linear expectation constraint using indicator-based rewards. They go on to show that minimizing divergence from the base model under these constraints yields a convex optimization with a finite set of dual multipliers. \n\nTo implement this practice, they sample outputs to estimate quantile thresholds and indicator values, perform gradient ascent on the dual to obtain multipliers, and then fine-tune the model with a standard RLHF solver using a composite reward defined by the weighted indicator functions.\n\nThe authors evaluate their approach on conversational and code-generation benchmarks. In dialogue experiments, they target quantile lifts on Helpfulness and Harmlessness metrics. In code experiments, they target Simplicity and Security metrics. They use a fine tuned GPT-2 model with a value head for helpful, harmless evals and an LLM as a judge framework with a gpt 4o model for their simplicity, security evals. Their results demonstrate substantial improvements in lower-tail performance, reducing the frequency of undesirable outputs, while preserving fluency, diversity, and coherence. Comparisons against multi-objective RL baselines show that Quantile-Guided Alignment achieves more stable simultaneous gains at specified quantile targets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n* The paper presents a rigorous theoretical foundation, proving convexity of the primal and deriving a finite-dimensional Lagrangian dual that yields an exponential-family solution.\n* The authors carefully connect to standard RLHF, showing that their framework subsumes RLHF as the single-quantile special case, which adds credibility and clarity to the contribution.\n* The exposition is well-structured, beginning with the simplest single-quantile formulation and incrementally extending to multi-value and continuous quantile settings.\n* Mathematical derivations are laid out step by step, with clear notation and intuition offered alongside proofs, which makes the paper accessible to readers familiar with RLHF.\n* Addressing tail-risk in large-language models is a pressing challenge, especially for safety-critical applications, and the quantile-guided approach offers a principled way to limit rare but severe failures.\n* The idea of enforcing quantile constraints via indicator-based expectation constraints is novel in the context of language-model alignment.\n* Extending to a continuum of quantiles with an integral dual multiplier function is an innovative generalization that goes beyond existing tail-risk or CVaR methods.\n\n### Weaknesses\n* While quantile constraints are novel here, related work in risk-sensitive RL (e.g., CVaR optimization) is not discussed.\n* The evaluation for code generation experiment depends on GPT-4o as an automated judge for code safety and simplicity, which can be highly sensitive to prompt phrasing. Without a prompt template (e.g., chain-of-thought examples to guide the range of values output by the judge), it is difficult to assess the reported gains.\n\nQUESTIONS:\n* How does Quantile‐Guided Alignment compare empirically to other tail‐risk or CVaR‐based RL approaches on the same benchmarks?\n* What is the additional computation cost in terms of runtime for QA compared to vanilla RLHF?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}