Beyond Expectations: Quantile-Guided Alignment for Risk-Calibrated Language Models
Authors: Xinran Wang
Jin Du
Azal Ahmad Khan
Qi Le
Enmao Diao
Jiawei Zhou
Jie Ding
Ali Anwar
Venue: NeurIPS 2025 spotlight ()
Keywords: Alignment
Large language models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Quantile-Guided Alignment (QA) is a framework for multi-dimensional quantile alignment that reduces catastrophic risks in language models by imposing constraints on reward quantiles across multiple performance dimensions.

Abstract: Large language models can generate rare but catastrophic outputs, such as harmful conversations or insecure code. Existing Reinforcement Learning from Human Feedback (RLHF) typically maximizes average reward, leaving high-risk tail events insufficiently controlled. We introduce Quantile‑Guided Alignment (QA), a framework that allows users to specify desired improvements at any quantile—individually or across multiple reward dimensions—thus shifting the distribution of outputs with finer control toward safer, more desirable outcomes. The method extends standard RLHF via an augmented reward formulation that enforces quantile constraints. Experiments on conversation and code‐generation tasks show that quantile alignment significantly enhances quality at targeted tails while maintaining overall performance. The results position QA as a principled route to risk‑calibrated language models with tail‑focused alignment.

Beyond Expectations: Quantile-Guided Alignment for Risk-Calibrater Language Models
Xinran Wang', Jin Du²; Azal Khan', Qi Le', Enmao Diao, Jiawei Zhou², Jie Ding", Ali Anwar "Department of Computer Science and Engineering, University of Minnesota "MorphMind Al Department of Applied Mathematics & Statistics, Stony Brook University "School of Statistics, University of Minnesota (wang8740, khan1069, 1e000288, dingj, aanwar @umn. edu j jn@morphmind. ai di aaa.eemhhottaalll.mmmmmammmm. com jiawei zhou. 1@stonybrook.ed.
Abstract
Large language models can generate rare but catastrophic outputs, such as harm- ful conversations or insecure code. Existing Reinforcement Learning from Hu- man Feedback (RLHF) typically maximizes average reward, leaving high-risk tail events insufficiently controlled. We introduce Quantile-Guided Alignment (QA), framework that allows users to specify desired improvements at any quantil--individulll or across multiple reward dimensions thus shifting the distribution of outputs with finer control toward safer, more desirable outcomes. The method extends standard RLHF via an augmented reward formulation that enforces quantile constraints. Experiments conversation and onde-generation tasks show that quantile alignment significantly enhances quality at targeted tails while maintaining overall performance. The results position QA as principled route to risk-calibrated language models with tail-focused alignment.
1 Introduction
Large language models (LLMs) have achieved remarkable results in language understanding D+4, code generation 6), and agentic decision-making tasks 7 However, they also pose safety chal- lenges when small fraction of their outputs can be harmful catastrophic. Standard Reinforcement Learning from Human Feedback (RLHF) S 9 optimizes expected reward to improve average performance, but it may still allow rare, sigh-impact failures to persist. In safety-critical scenarios, ranging from Al personal assistants [10] to security- sensiti- code generation [TT) optimizing only the mean behavior is insufficient;
must also address specific quantiles of the distribution (e.g., its extreme tails) where dangerous errors occur.
For instance, consider personal assistant handling content-sensitive consultations or code- generation model that sometimes produces unreliable snippets. Although standard RLHF may reduce errors overall, it can fail to eliminate occasional yet severe missteps. This motivates quantile alignment, which explicitly enforces constraints on fraction of outputs that violate certain thresh- olds, effectively controlling the distributional tails (or any other critical quantiles) of relevant reward metrics. For example, requiring "90% of scheduling decisions meet a minimum standard" ensures the assistant rarely overlooks urgent tasks; similarly, imposing "99% of code must remain below given harmfulness score" mitigates security exploits or unauthorized system calls.
We formulate quantile alignment as constrained KL-regularization objective that minimizes the KL divergence from a reference model subject to user-defined quantile constraints, thereby limiting how many outputs fall below or above-critin values. We prove that the resulting optimization is convex
'Contibuueed to this work while at the University of Minnesota.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and admits a Lagrangian dual, whose solution reweights the original model distribution by linear combination of indicator-based quantile rewards. This structure closely parallels RLHF, replacing the usual scalar reward with aaantiee-ddaptd terms. In practice, we solve the dual numerically using Monte Carlo approximations, enabling an efficient and flexible way to regulate model outputs across multiple quantiles and metrics. The main contributions of the paper are twofold:
Quantile-Guided Alignment (QA) Framework. We introduce cianttlleguiddd alignment, a princi- pled way to impose distribution-tail constraints on multiple reward dimensions (e.g. harmlessness, helpfulness). Rather than maximising the mean reward, QA guarantees that user-specified fraction of outputs satisfies strict thresholds, thereby mitigating rare but critical failures. Each quantile constraint is encoded as an indicator- bassed reward, leading to a convex KL-regularized objective with primal- dual solution whose dimensionality equals the number of constraints.
RLHF- Coop-Coopteible Implementation and Empirical Validation. Because the dual problem is dinensonlasiinnl,, QA drops straight into the standard RLHF tool-chain (PPO, TRL., etc.) after sub- stituting the composite reward. Experiments on conversational and code- code-generation benchmarks show that QA sharply improves tail-risk metrics-e.2 reducing the worst 5% of harmful outputs- preserving diversity, coherence, and perplexity.
Related work
Reinforcement Learning from Human Feedback (RLHF). Language model alignment strategies can be categorized into training-basee and decoding-bases approaches. Training- methods adjust model parameters through fine-tuning guided by human feedback, whereas decoding- based methods constrain or steer the models outputs at inference time without retraining. Within training- based solutions, a prominent technique is RLHF [2 14] that involves two main steps. First, reward model is trained to map each candidate output to scalar that reflects human preferences. Typically, dataset of comparisons (I, Ylose Vwin) is collected from human annotators, where is prompt and two different responses Ylose and 1/win receive preference labels indicating Ywin is preferred over Vlose- Then, one uses the reward model r to maximize the expectation of generated rewards subject to a Kullback-Leibler (KI.) regularization. As a result, RL.FF nudges the model distribution toward higher-reward (i.e. more human (reean-rreferre) outputs.
Multi-dimension Preference Alignment. Multi-objective reinforcement learning (MORL) [15] 17 has been used as a popular approach to address multi-dimensional preference alignment. often build on the classical RLHF setup 12 18), using linear scalarization to combine reward signals or data sources [1.
to construct a single RLHF objective. For example, the attempts approximate the Pareto frontier of models by fine-tuning separate models for each reward function, then blend these specialized models by interpolating their model weights. In principle, this approximates the ensemble of models that would have emerged from training on every linear combination r -E==1 X r. directly. The recent MAP approach [21] provides alternative perspective: instead of requiring single scalar reward, it enforces constraints on the expected values of different reward functions relative to user-specified targets.
19
18
20
recent work Rewarded Soup 19
AT Safety, Generative Al models are susceptible to adversarial manipulations such as backdoor attacks (22255) and jailbreak attacks 266 27 which can lead to severe issues including hallucinations or security breaches [28] 229). These stealthy exploits often arise from deceptive or poisoned training data, compromising model safety even after deployment. In response, efforts in AI safety include inference-time methods such as prompt engineering (30] or detection-basee filters BT) and training- time adjustments via RLHF alignment schemes that incorporate robustness into reward functions [14
Background
3.1 Preliminaries on RLHF
The RLHF framework aligns language model po with human preferences by solving:
Ep Ep(r(a,v)] - -3 DxL (pkl(p(112)) (p(-1=)) Po }
max p
(I)
where Po denotes the reference distribution that corresponds to the original model, P denotes the class of all distributions, p is the distribution that represents the aligned model, a reward function that quantifies the preference level of any given pair of prompt and generation y, DxL measures the

---

KL-divergence, and 3 > 0 is regularization hyperparameter. The expectation is taken over ~ D (prompt set) and y x ~ p(- x) (conditional generation), briefly written Ep. from It be shown that the solution of (1) can be written in the form of ply x) poly 2) exp((8">ra, y)) (3( where C(3) is normalizing constant for p(- x) to be valid probability density function. While this formulation can effectively raise average performance of large language models, it does
inherently control quantiles, especially tail events that lead to rare but potentially damaging outputs.
3.2 Preliminaries Quantile Constraints
Consider random variable Z and its 7-th quantile Q2( By definition, Q+(Z) satisfies P(Z S Qr (Z)) - T, where the probability PP, is defined under æ, y ~ p. For a reward function D and y q(- [x) for some generative model q induce random variable r(z, y). The T-quantile of r(z, y) is defined by: (r.p(r) - inf(c (r(x,, y) S c) 2 r). We aim to impose constraints on this quantile, for instance (r.((r) 2 c. Rewriting this, we have:
(r)(r) 2 c
pp(r(2,))) S s) T E,I(((e.)) s
Rearranging this, we obtain the equivalent requirement
B - T(r(,,)) <c] 2 0,
(2)
where 1() c is indicator. Thus, we define an indicator-based quantile reward:
(x,y) + Pr.c(r(x, v)) - [[r(x,)) y) < <c},
which can be regarded composite function Pr.c that maps
pair (z, to
a reward. According to Inequality (2) imposing the expectation of (r(2 y)) to be nonnegative under the aligned distribution g equivalently ensures (r. 2 This construction translates
quantile constraints into inequalities that involve only linear functional in
Quantile Alignment (QA): Formulation, Theory, and Algorithms
We consider the most general setting of aligning pretrained distribution to satisfy multiple quantile constraints on multiple reward functions. For clarity, we first present the single human value (i.e. single reward function) scenario where each reward has multiple quantile thresholds, and then generalize to multiple reward functions.
4.1 Single- Value Multi- Quati-Quantile Constraints
Let r(x, y) be single scalar reward function. We want enforce multiple constraints of the form
(r(,, 2 Cj Quy (r(z,
[m].
where each constraint j stipulates that at most fraction Ty of samples fall below threshold, which the -yquuntee of the original distribution po. That is, we lift the Ty-th quantile of to the level of the ky-th quantile. Equivalently,
(p( v) < <,,)) ST, = Ep (r(z, 20,
where we define
Pryme, (r(x, v)) - Ifr(2,0) <c
(3)
We collect all such constraints in the objective:
min PEP E, (pL( (x) po x))]
subject to Eplpt,n, (r(z, v))l 2 o, je j
(4)
where we define the Indicator-based quantile reward for each constraint j
Pry.ag (r(z, v)) Tr (r(
(5)
3

---

Remark 4.1 (Connection to the standard RLHF). Recall from the standard RLHF framework that imposing a scalar reward r(z. y) leads reweighted distribution pyy x) & Po x) exp (8 r(z, y)), where plays the role of a temperature that balances between the original model pu / x) and the reward r(æ, y). As the next theorem shows, quantile constraints also produce an exponential reweighting, but the "reward" now comprises a sum of indicator- based terms:
ply x) Po x)) exp Σ Ng PTy. ( (r(z:
where each active constraint introduces learned multiplier > 0. The objective in (4) a conver program in the space of distributions in po. and each constraint j introduces dual multiplier X3 2 0 whose values are determined by its constraints. Hence, although the QA-aligned distribution retains form, Pry.n; differs from the usual reward r whose corresponding
the same exponential-famie multiplier 3 is tuned as a hyperparameter.
Theorem 4.2 (Representation of the Multi-Quantii Solution), The optimization problem 4 is convex in p. There exists a unique m-dimensional vector X NR. E [m]] 2 O such that the optimal solution is
py x) Poly at) exp Em= Xs Prs.ng (r(2,v)))
C(22
where ((X) is the normalizing constant, and X [AI...., Am] is determined by
M( Kyg j E [m]]) arg max log ](
2
Remark 4.3 (Generality of Quantile Alignment). Theorem 4.2 implies that any distribution satisfying the specified quantile targets can be written in the exponential-famlly form defined by QA. Conversely, for any aligned distribution p produced by other methods, the pair po- P corresponds to a feasible set of pairs {Ty, Ki)) ). which can be an infinite set of continuously-value quantiles (as elaborated in Section 5. This shows QA's generality. Notably, the standard RLHF objective can be regarded as special case of QA.
4.2 Numerically Solving the QA Problem
To solve the QA problem efficiently, first outline the high-level logic: while the original for- mulation involves optimizing the infinite-dimensional distribution p. we can transform it into a lower-dimensional convex optimization problem over the dual variables X. This is because the problem in (4) is convex, and standard convex analysis techniques such as strong duality apply. This reformulation enables tractable optimization of X by leveraging Monte Carlo estimation. Furthermore, by Theorem 4.2 the role of À manifests in an exponential reweighting, which can be interpreted as solving a single RLHF problem where the reward is a weighted sum of indicator-based quantile rewards. This enables us to leverage existing RLHF solvers, such as the Proximal Policy Optimization (PPO) algorithm implemented in the TRL package [32].
One may wonder why, from the original QA formulation, it appears that we are optimizing a nonconvex problem when p represents large-scale language model. The key insight is that if we do not treat the optimization as occurring in the model parameter space, but rather over a generic probability density, we can reframe the problem convex optimization over X, whose dimension equals to the number of quantile constraints.
To solve the dual problem, we first rewrite C(A) as expectation under the original model po:
C(A) = Eyypo((11)) exp Σ
Xy Privs (r(x, u))
(6)
1=1
This allows us to estimate the dual objective via Monte Carlo sampling. Specifically, let (1x, 9))) be i..dd samples drawn from Po( y). Notably, this sample can be used for all various alignment compute the indicator-based quantile rewards: Pry.z, (r(2e, ve)) =
targets. For each sample, T I(r(ze, ve) < Cj}, where éj is calculated as the ky-th quantile of (r(we, we), E [m]]. Using these samples, we approximate the dual objective as:
((A) A - log
Σ exp
X Pr,in (r(22.81))

---

Remark 4.4 (Concavity and Convergence). By Theorem 4.2
log C()) is concave in A. Since the sample average special expectation, g( preserves concavity, ensuring that standard gradient
ascent methods converge the global maximum of (A). Once we solve for X", we obtain R(x, y) - Σ5=1 Xÿ Pr.., (r( y)). According to Remark 4.1 R can be treated as an effective reward model in the standard RLHF with inverse temperature 3 allowing us to directly apply PPO solvers.
The accuracy of this numerical solution depends on the number of Monte Carlo samples In practice, a few thousand samples typically suffice for stable estimates of g()). uper-specified thresholds are infeasible, the procedure detects it via divergence in the dual or violation of positivity constraints. In such cases, we can automatically adjust the thresholds via line search or alternative strategies.
Algorithmic Steps. Next, we summarize the procedure for solving the QA problem numerically.
1. Sampling. Draw samples {I ve) from po-
2. Compute Based Rewards. For each L, evaluate Pry.ng (r(ze, ye)) for each Ty, Ky.
3. Dual Update. Initialize an m-dimensional X (0) z 0 and perform gradient ascent:
A (++1) < (> (0) + V ())))
until convergence, where (+)+ denotes projection onto the nonnegative orthant, and rj 0 is the step size. If diverges, we decide the constraints are infeasible.
4. Construct QA Reward. Once we obtain the dual solution A", compute the effective reward:
R(I,
Σ X3 (r(,nn(((2,))) v)). j=1
5. Optimize based on the QA reward. Treat R(x, y) the reward function in the standard RLHF setting with 3 = and apply a PPO solver to update from Po to p.
Remark 4.5. Since the primal-dual method only requires forward passes under p, it does not involve backpropagation through model parameters. The runtime and memory complexity scale O m)) where n the number of MC samples and m is the number of constraints. This setup remains computationally feasible even for large models, as the same set of MC samples can be reused across different quantile constraints without retraining the base model.
4.3
Multi- Multi-Value, Multiple-Quantin Alignment
The QA framework readily generalizes to multiple reward functions rı (x. v)..... may have multiple quantile constraints. That is, for each reward function rs, we impose the constraint
... & (ar, y). and each
Qr...P .P (F,(I, v)) 2 Qu..y.Po (rr(1, v)). [m,]], [K],
where Tijj represents the quantile threshold for and Ki,j represents the corresponding threshold for Po- This ensures that at most a fraction Ti,y of generated samples fall below the Kg, qqaantie of the original model Po-
Following the single-reward case, define the multi-value indicator-based quantile rewards:
Pr (r.(x. v)) Tu. I{ (z cij) }.
where c.y is the empirical K quantile of r. rz(2 (2,,0)) y) under Po- These constraints then incorporated into the KL-regularized objective:
min PEP
Ep (pL((p(: x) | x))]
s.t. Epppr. (r1(z,8))] 2 0.j E [m=], E [K].
The only difference from the single-reward setting is that the exponent in the optimal solution now becomes R(x, y) A Σ E/= A (r;(2, y))- Computationally. the optimization dimensionality is proportional to the total number of constraints mi + m2 - + mg The numerical solution follows the primal-dual Monte Carlo approach outlined in Section 4.2 with each reward function ri (2, y) contributing its own set constraints. Thus, the QA framework provides a principled and efficient mechanism to enforce multiple quantile constraints across reward functions.
5

---

Continuous Quantile Alignment
Thus far, we have formulated quantile alignment as a discrete optimization problem, enforeing constraints at specific quantile levels. However, it is both theoretically intriguing and practically helpful to study oversight across a continuum of quantile levels. To generalize our approach, we now extend our framework to enforce continuous quantile constraints, shaping the entire distribution of reward values rather than a finite subset quantiles.
5.1 Uniform Distribution Enhancement through Continuous Constraints
For notational simplicity, we focus on the single value represented by reward function There is no essential difference in generalizing to multiple values, as we discussed in Subsection 4.3 Previously, we considered finite set of quantile constraints at levels {- ensuring that specific quantiles of the reward distribution under meet or exceed corresponding reference values, We now impose quantile constraint every level
[0, 1], requiring (T)(r)) 2 c(T), where c(T) 2 Qr.po (r) is target quantile function defined for all 7 E [0, 1]. This ensures that the quantile curve of r under g remains above c(r) for all т, uniformly lifting the entire reward distribution to match a desired profile. Rewriting this in expectation form, we obtain:
(8) where the indicator-based quantile reward is defined as (r(x,,, y)) - v) < c(T)). This formulation ensures that instead of controlling individual quantiles, we impose constraints over an entire continuum, creating a smooth and robust enhancement of the reward distribution. This leads to an dimeni-dd constrained optimization problem:
(r)p 2 O, E
min Ep DKL (p(- x)) I| Po( [))]
[p[p+(r)] 2 O, V [0,1].
pEP
Unlike the discrete-quantile case, where the number of constraints is m, we now have an infinite set of constraints one for each To handle this, we introduce nonnegative Lagrange multiplier function {()) indexed by т, leading to the following Lagrangian- -based reweighting.
Continuous Re-Rlaulrrd Solution. With a similar argument as in Theorem 4.2] the aligned distribution also takes an exponential reweighting form, but now with continuous integral:
Px x Po(y x) exp So A(T) p. pr(r1, d)
C(2)
where the normalizing term is:
C(A) A Exyop exppppopp/// lf X(T) (r((,,)))
smoothly incorporates the influence all quantile constraints through the weighting of X(r).
Dual Problem and Connection the Discrete Case. Following the primal-dual logic in the discrete setting (cf. Theorem 4.2),, the dual problem optimizes over the function X(T), leading to: maxx(7) log C()).. This generalizes the discrete quantile dual objective, now integrating over an entire range of r. The function \(7) plays a similar role the discrete multipliers Aj.
While the continuous formulation theoretically imposes constraints at every quantile level, in practice, approximate the integral over 7 by discretizing it into finite set of representative quantile levels. This reduces the problem to solving a discrete QA problem with a finer resolution. The numerical solution follows the same Monte Carlo-based primal-dual method described in Section 4.2 treating the continuous constraints as an additional layer of sampling.
5.2 Infinitesimal Enhancement Analysis and Algorithm
We now study how small perturbations in the target quantile function cx (T) affect both the optimal dual variable X(r) and the aligned distribution p.. This analysis reveals how the system reacts to incremental changes in oversight targets, providing a sensitivity measure for quantile alignment.
Perturbation of the Target Quantile Function. Consider a baseline quantile function c. (7) that
satisfies the constraints: Ep. (r)] Vr [0, 1], where (r(r) L|| cx(r) is the

---

lai QQ flotfer He pfiineess
(b) QQ Plac for Harmlessness
Baseline Performance Netrics
Helpfulness
Hemmlessness
Original Model
Algned Model quanttie
Alggedd Mode: Quant ies
3.0
Aligned
2.5
Mesn
0.5
-2
0.0
Diversit
Canerenca
Perplexay
Original Model Quantless
Original Model Quanties
Figure 1: Effect of Quantile Alignment Helpfulness and Other Metrics. (a) QQ plot comparing Helpfulness quantiles before and after alignment. Points above the red 45-degree line indicate increased Helpfulness scores the aligned model. (b) QQ plot for Harmlessness, showing tradeoff where some responses become less harmless. (c) Comparison of baseline performance metries, showing minimal change in Diversity and Coherence, with slight improvement in Perplexity: Note that alignment constraints were imposed only for He Helpfulness, with direct constraints set for Harmlessness.
indicator-based quantile reward. The aligned distribution p. is obtained with dual solution A. (T). Now, perturb the quantile function by a tiny shift 8(T): c(T) = c, (x(T) + ((T). This leads to the solution (px, X(T)). where dA(T) X(T) (T) is the first-order change in the dual variable.
Definition of Operator V. The effect of perturbing (T( propagates through X(r) and ultimately shifts the distribution p.. The key object governing this interaction the linear operator V, which captures the response of different quantile constraints to changes in X(T):
(V (V / [ppp
(-(()) (+ 62(F) di.
Intuitively, models the dependency between quantile constraints across different r-values, deter- mining which shifts in A(r) induce correlated responses.
Theorem 5.1 (First- Order Sensitivity of Quantile Alignment). Let (px, X.) be the baseline solution for the quantile function e. (T), satisfying:
Ep. [p)(r) 2 0, Wr E [0, 1).
Consider a perturbation 8(T) such that c(T) = CT( + 5(T). Define the first-order changes in the dual variable and the aligned distribution as:
8X(T) = ((T) - \u(T), óp = Px - P..
Then, if the original constraint is active, namely Ep. (r)) = 0, the perturbed dual variable satisfies
the linear integral equation: (V ())(T) _ Ep. p (r) (r))) where p-((r) -Ir < c. (T)]. af the constraint is strictly satisfied, we have (a(T) = O. and small perturbations in c.(T) do not affect X(T) until the constraint becomes active. Furthermore, the updated distribution px is
+ 0(16))),,, where given by: px x) = (y x)[1+ + (r( - -.. (p.
|0 - suprejo.1] (8X(T)]-
In practice, E [0, 1] is discretized into a finite set {m)), and V reduces to the matrix form: Vig Ep. Pp: (r) Pr, (r)].
Experimental Study
We evaluate quantile alignment on conversational and code-genee gedeeeeerrttion tasks, where model outputs range from benign to risky behaviors. Experiments were conducted single Nvidia A 1100 GPU.
Models. We apply our alignment procedure to two models: the OPT-1.3B 33 for conversational tasks and CODEGEN-350M (34] for code generation. We experiment with smaller models in the corresponding model families due to GPU constraints.
Data. For the conversational task, we use prompts from the Anthropic Harmless dataset 35 which contains human requests formatted between "Human:" and "Assistant:". This dataset serves

---

[a] CQ Plat for Helpfuiness
1b) QQ Pott for Harmlessness
Ic] Baseline Performance Mctrics
3.5
0.5
Helpfuiness
Harm ieseness
Original Model
Nigned Mode Quantiis
0.0
y
Alggned Mods Quanties
3.0
Alggned
-0.5
2.5
-1.0
2.0
-1.5
Mean
-2.0
1.0
-2.
0.5
-3.0
0.0
-3
-2
Diversity
Canarencc
Perplexity
Original Model Quantiles
Original Model Quantiees
Figure 2: Effect of Quantile Alignment on Helpfulness and Harmlessness. (a) QQ plot comparing Helpfulness quantiles before and after alignment. (b) QQ plot for Harmlessness. (c) Comparison of
baseline performance metrics. Unlike Experiment constraints were imposed for both Helpfulness and Harmlegsness.
Camparison Quantile Alignment (OA) with Baselines
9-5%
q- 10%
Q-25%
9-75
Original
Original
Original
Original
Q-Quandle of Harmlessnes newards
Q4
QA MORL
QA
QA MORL
MORL
MORL
1-
X
Xa
-3.5
-3.0
-2.5
-2.0
-1.5 -1.0
-0.5
0.0
q-Quantile of Helpfuiness Rewards
Figure 3: Comparison of QA and MORL on Conversational Alignment. The plot shows the r-quantiles of Helpfulness and Harmlessness rewards for 5%, 10%, 25%, 75%, comparing the original model (cross markers). QA-aligned model (diamonds), and MORL (dots for 20 randomly sampled À values).
benchmark for evaluating alignment with safe and cooperative responses. For the we employ the HUMANEVAL dataset [5]. a standard benchmark that consists Python programming tasks. Each prompt in HUMANEVAL specifies a function signature and doestring describing the expected behavior. Since HUMANEVAL primarily consists of relatively simple coding tasks, we extend it with custom prompt set to increase diversity and difficulty.
task,
Custom Code-Generation Prompts. We curate 200 additional prompts spanning eight categories of software quality: file access, network calls, security risks, maintainability, execution time, data integrity, scalability, and documentation quality. These prompts simulate a range of real- real-world coding concerns, including adversarial cases. Each custom prompt randomly assigned a quality level (standard, low, very low, edge case, worst case) and control statement that explicitly directs the model toward suboptimal coding practices. Examples include: "Do not handle edge cases or errors." "Write in way that has little error handling," and "Nake variable names confusing or mon-descriptive"" The prompts were generated programmatically using GPT-4 to maintain structured, instruction-based format. The final dataset combines the original HumanEyal tasks with these curated prompts to comprehensively assess model alignment.
Alignment Values and Evaluation Metrics. We evaluate four key alignment values: Harml essness and Helpfulness for conversational tasks, and Simplicity and Security for code generation. For Harmlessnes and Helpfulness, we use two GPT-2 models with value heads fine-tuned on human-ana annotated preferences [36]. providing scalar ratings that indicate how well generated responses align with harmless and helpful behavior. For Simplicity and Security, we employ an automated evaluation using OpenAl's API. A GPT4o-based reviewer model is prompted with a definition of each attribute- Siaiiee-iimpiiiitty measures how minimal, maintainable. and Pythonic the code is, avoiding unnecessary complexity or redundancy, while Security assesses the code's 's resilience against vulnerabilities, including secure function calls, input sanitization, and careful data handling, The model rates each attribute on 0-1 continuous scale.
8

---

Additionally, for each alignment task, we assess three baseline performance metrics. 1) Diversity: Quantifies lexical variety in model outputs, computed as the proportion of unique n-grams (n 2, 3, 4) and aggregated into a composite diversity score [37]. 2) Coherence: Evaluates semantic consistency within generated text using supervised SimCSE BERT-based model for sentence embedding similarity [38] 3) Perplexityy Measures how predictable a generated response is under a language model, serving as a proxy for fluency.
Experiment I: Single- Value Quantile Enhancement. We evaluate our quantile alignment approach by enforcing constraints to systematically improve Helpfulness. Specifically, we align the model's helpfulness scores using the following quantile pairs (Ty, ): (1%, 5%). (5%. 10%). )). (10%, 50%). (50%, 60%), (60%, 70%), (70%, 80%), (80%, 90%). (90%, 95%), (95%. 99%). This means, for instance, that responses previously scoring at the 1% percentile in helpfulness are elevated match the 5% percentile level, and so forth. By lifting multiple quantiles, we enforce strict improvement across the entire distribution.
Figure a visualizes this shift using a quantile-quantin (QQ) plot, where the x-axis represents the quantile values from the original model, and the y-axis represents the corresponding quantiles from the aligned model. Each blue point represents the 7-quantile of the rewards of a particular human value under both models, with sampled regular intervals (5%, 10%, .... 95%). Points above the red 45-degree line indicate that the aligned model achieves higher rewards at that quantile compared to the original model. As expected, the results confirm that our approach effectively improves Helpfulness across the entire distribution.
We follow the numerical optimization procedure outlined in Section 4.2 and apply PPO-based fine- tuning to obtain the aligned model. Figure bb) illustrates the impact on Harmlessness, which, as noted in prior work (14), exhibits tradeoff with Helpfulness. The QQ plot reveals noticeable decline in
scores across quantiles, indicating that improvements in Helpfulness come at the cost of reduced Harmlessness Meanwhile, Dig shows that baseline performance metrics, including Diversity and Coherence, remain largely stable. Interestingly, Perplexity decreases slightly, suggesting that alignment may also contribute to improved response fluency.
Multi- Value Quantile Enhancement. Building the previous experiment, we now introduce an additional constraint to enhance Barmlessness, specifically lifting its 5% - 50% quantile, while maintaining the existing constraints Helpfulness. This aims to aggressively im- prove Harmlessness essness while preserving Helpfulness as much as possible. Figure[bb confirms that Harmlessness is indeed elevated across quantiles. However, Figure 2 a) shows that Helpfulness exhibits some degradation in certain regions, suggesting a tradeoff between the two values. Although our numerical solver guarantees feasible dual solution, the observed discrepancy in Helpfulness may stem from Monte Carlo approximation errors, where the estimated À deviates from its true population counterpart, combined with inherent stochasticity in the PPO fine-tuning process, As seen in Figure [cc), baseline metrics-including Diversity and Coherence-reman relatively stable, while Perplexity again shows slight improvement.
Experiment 2:
Experiment 3: Comparison of QA with MORL Baseline. We compare Quantile Alignment (QA) with Multi-Objecttve Reinforcement Learning (MORL) [15H77), which optimizes expected rewards by sampling tradeoff weights between objectives. Specifically, in the MORL setting, the dual weight vector À is generated as À = 8. u, where s is uniformly sampled from (0, 6) and u is sampled from the probability simplex, representing random tradeoff preferences. Figure 3 compares quantile performance across different alignment methods at T = 5%, 10%, 25%, and 75% for Helpfulness and Harmlessness The QA-aligned model consistently moves towards the upper-right quadrant, indicating simultaneous improvement in both values relative to the original model. In contrast, MORL without principled optimization strategy often sacrifices one objective in favor of the other, leading to greater instability in alignment outcomes.
7 Conclusion
We have presented Quantile- -uidded Alignment (QA), principled extension of RLHF that regulates quantiles of reward distributions. Casting the problem as a convex KL-regularized program yields a finite-dimensional dual; a Monte Carlo estimate of this dual directly constructs an overall reward function, which can then be optimized with standard alignment tooling. Complete proofs and additional code-generatin results are provided in the Appendix.
9

---

Acknowledgements
The work of Xinran Wang was supported in part by the 3M Science and Technology Graduate Fellowship and the Doctoral Dissertation Fellowship. The work of Qi Le and Azal Ahmad Khan was supported by the Amazon Machine Learning System Fellowship. The work of Jie Ding was supported in part by the National Science Foundation CAREER Program under grant number 2338506. The work of Ali Anwar was supported by the Samsung Global Research Outreach Award and the National Science Foundation Privacy-Preserviie Data Sharing in Practice (PDaSP) program under grant number 2452817.
10

---

References
[1] A. Wang, Y. Pruksachatkun, N. Nangia, Singh, Michael, F. Hill, O. Levy, and Bow- man, "Superglue: A stickier benchmark for general-purpose language understanding systems." Advances in neural information processing systems, vol. 32, 2019.
[2] S. Gehrmann, Bhattacharjee, A. Mahendiran, A. Wang, A. Papangelis, A. Madaan, A. Micmillan-major, A. Shvets, A. Upadhyay, B. Bohnet el al., "Gemv2: Multilingual nig benchmarking in a single line of code," in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022, pp. 266-281.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell el al., "Language models are few-shot learners," Advances neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[4] J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young el al., "Scaling language models: Methods, analysis & insights from training gopher," arXiv preprint arXiv:2112. 11446.14466600 2021.
[5] M. Chen, J. Tworek, H. Jun, Q. Yuan et al., "Evaluating large language models trained on code." 2021.
[6] B. Roziere, Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, Liu, R. Sauvestre, T. Remez et al., "Code llama: Open foundation models for code," arXiv preprint arXiv:2308. /2900 2023.
[7] Tian, A. Luo, Du, Xian, R. Specht, G. Wang. X. Bi, J. Zhou, A. Kundu, Srinivasa et al., "An outlook on the opportunities and challenges of multi agent ai systems," arXiv preprint arXiv:2505.18337, 2025.
[8] Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
[9] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," Advances neural information processing systems, vol. 35, pp. 27 730-27 744, 2022.
[10] M. C. Rillig and A. Kasirzadeh, "Ai personal assistants and sustainability: Risks and opportuni- ties," Environmental Science & Technology, vol. 58, no. 17, pp. 7237-7239,200
[11] R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, "How secure is code generated by chatgpt?" in 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2023, pp. 2445-2451.
[12] S. Griffith, K. Subramanian, J. Scholz, C. L. Isbell, and A. L.. Thomaz, "Policy shaping: Integrating human feedback with reinforcement learning," Advances in neural information processing systems, vol. 26, 2013.
[13] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman, "Deep reinforcement learning from policy-dependent human feedback," arXiv preprint arXiv: 1902.009024 2019.
[14] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, and T. Henighan, "Training a helpful and harmless assistant with reinforcement learning from human feedback," arXiv preprint arXhv:2004.05862, 2022.
[15] L. Barrett and S. Narayanan, "Learning all optimal policies with multiple criteria," in Proceed- ings of the 25th international conference on Machine learning, 2008, pp. 41-47.
[16] K. Li, T. Zhang, and R. Wang, "Deep reinforcement learning for multiobjective optimization," IEEE transactions cybernetics, vol. 51, no. 6, pp. 1103-3114, 2020.
[17] Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P. Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Hajishirzi, "Fine-grained human feedback gives better rewards language model training." Advances in Neural Information Processing Systems, vol. 36, 2024.
11

---

[18] J. Dai, X. Pan. R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang, "Safe RLHF: Safe reinforcement learning from human feedback," " in The Twelfth International Conference on Learning Representations, 2024.
[19] A. Rame, G. Couairon, C. Dancette, J.-B. Gaya, M. Shukor, L. Soulier, and M. Cord, "Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards," shirth-seventh Conference on Neural Information Processing Systems, 2023.
[20] P. Dognin, J. Rios, R. Luss, I. Padhi, M. D. Riemer, M. Liu, P. Sattigeri, M. Nagireddy, K. R. Varshney, and D. Bouneffouf, "Contextual moral value alignment through context-based aggregation," arXiv preprint arXiv:2403. /2805, 2024.
[21] X. Wang, Q. Le, A. Ahmed, E. Diao, Y. Zhou, N. Baracaldo, J. Ding, and A. Anwar, "MAP: multi-human-valut alignment palette," International Conference on Learning Representations, 2025.
[22] X. Xian, G. Wang. J. Srinivasa, A. Kundu, X. Bi, M. Hong, and J. Ding, "Understanding backdoor attacks through the adaptability hypothesis," in International Conference on Machine Learning. PMLR, 2023, pp. 37 972-377 976.
[23] "A unified detection framework for inference-stage backdoor defenses," Advances in Neural Information Processing Systems, vol. 36, pp. 7867-7894, 2023.
[24] G. Wang, X. Xian, J. Srinivasa, A. Kundu, X. Bi, M. Hong, and J. Ding, "Demystifying poisoning backdoor attacks from a statistical perspective," International Conference on Learning Representations, 2024.
[25] J. Ding, Generative Al: Principles and Practices Lecture Notes, University of Minnesota, 2024, accessed: Nov 27, 2024. [Online]. Available: https://genai-coorse course. jiing.eediiigg.rg...... www.d//nn..rgggffettiiieeeeeeee html
[26] Z. Niu, H. Ren, X. Gao, G. Hua, and R. Jin, "Jailbreaking attack against multimodal large language model," arXiv preprint ar 2rXiv:240..02:000 2024.
[27] X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, "Visual adversarial examples jailbreak aligned large language models," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, 2024, pp. 21 527-21 536.
[28] Z. Ji, N. Lee, R. Frieske, T. Yu, Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung. "Survey of hallucination in natural language generation," ACM Computing Surveys, vol. 55, no. 12, pp. 1-38, 2023.
[29] Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Zhang, "A survey on large language model (Ilm) security and privacy: The good, the bad, and the ugly," Coifh--onffleecee Computing, 100211, 2024.
[30] Y. Xie, J. Yi, J. Shao, Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, "Defending chatgpt against jailbreak attack via self-reminders." Nature Machine Intelligence, vol. 5, no. 12, pp. 2023.
[31] X. Wang, E. Diao, Q. Le, J. Ding, and A. Anwar, "AID: Adaptive integration of detectors for safe ai with large language models." in Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025.
[32] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q Gallouédee, "Trl: Transformer reinforcement learning." trl. 2020.
[33] S. Zhang, S. Roller, N. Goyal, M. Artetxe. M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Out, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, Sridhar, T. Wang, and L. Zettlemoyer, "Opt: Open pre-trained transformer language models," 2022.
[34] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, "Codegen: An open large language model for code with multi-turn program synthesis," ICLR, 2023.
12

---

[35] Anthropic. "HH-RLHF Data," 2024, accessed on July 5, 2024. [Online]. Available: httpss//wwggiggiiio...//ccoooopp wwwp:opp-nn//uupp--eoee cemnee--bertte bast-uncasedd
[36] R. Yang, X. Pan, Luo, S. Qiu, H. Zhong, D. Yu, and Chen, "Rewards-in-context Multi- objective alignment of foundation models with dynamic preference adjustment," International Conference on Machine Learning, 2024.
[37] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," International Conference Learning Representations, 2020.
[38] T. Gao, X. Yao, and D. Chen, "Simcse: Simple contrastive learning of sentence embeddings." Conference on Empirical Methods in Natural Language Processing, 2021.
13

---

A Experimental Results for Code Generation
We apply quantile alignment to the code generation task, focusing Simplicity and Security, aiming to lift lower quantiles while preserving baseline performance.
In the first experiment, we align only Simplicity progressively elevating its lower quantiles. Specifically, the 1%, 5,, and 10% quantiles lifted to the level of the 60% quantile in the original distribution. Mid-to-high quantiles are also incrementally adjusted: the 50% quantile is raised to the 70% level, the 60% the 80%, and so forth, up to aligning the 90% quantile with the 99%. This ensures more uniformly improved distribution of Simplicity across generated code.
Figure presents the results. The QQ plot for Simplicity confirms that lower quantiles have been successfully lifted while maintaining a well-structured distribution. Although Security was not explicitly aligned, its tail performance slightly improved, suggesting degree of synergy rather than strict tradeoff between the two attributes. Baseline performance metrics, including Diversity, Coherence, and Perplexityy ty, remain stable.
In the second experiment, we shift our focus to aligning Security, enforcing constraints similar to the previous setup. The 1%, 5%, and 10% quantiles raised to the level of the 60% quantile in the original distribution, while mid-to-high quantiles are lifted progressively, with the 50% quantile reaching the 70% level, the 60% to 80%, and the 90% quantile 99%. Figure presents preeets the results. As expected, Security improves its distribution especially in the mid-range quantiles. Interestingly, despite not being explicitly aligned, Simplicity also exhibits noticeable gains, reinforcing the synergy observed earlier. This suggests that enforcing structured improvements in one alignment attribute can positively influence related attributes, rather than introducing strict tradeoffs. Once again, baseline performance metrics remain largely stable, confirming that quantile alignment does not disrupt overall model fluency.
Code Generation Example 1. To further illustrate the impact of quantile alignment, we compare model generations before and after alignment for specific task. The following case was sampled at approximately the 10% quantile of Simplicity from the original model's Simplicity ciptyyyiy reward distribution. The aligned model output is drawn from our earlier experiment designed to elevate S Simplicity quantiles.
Original Prompt
**aak**:: Write Python program to measure the execution time of function.
**Purpose**; This script will help in benchmarking the performance of Python function. It will measure the time it took to run the function by leveraging Python's time.
Ignore security concerns. The generated code should be of worst-case quality. Here is the Python code:
(al 00 for Security
(b) 00 Plet Smplicity
(d] Baseline Performance Metrics
Algnndd Model Quantiee
0.8
y
Security
Aligned Model Quandiee
a.5
Simplicity
1.2
Original Model
Aligned Model
1.0
0.3
Mrem
0.8
0.4-
0.1
0.2
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.0
0.3
0.2
0.3
0.4
0.5
0.0
Diversity
Cohereence
Perplexity
Original Modil Quantiles
Original Modell Quantiles
Figure 4: Effect of quantile alignment on Simplicity in eode generation. QQ plot comparing Security quantiles before after alignment. Points above the 45-degree line indicate improved Security scores the aligned model. (b) QQ plot for Simplicityy showing successful shift in the lower quantiles towards higher values. (c) Baseline performance metrics remain stable, with no degradation in Diversity or Coherence, and minor improvement in Perplexity.

---

Original Model Output
Note: This will take while to X timeit -r python This will take Long time. Use 'x' as placeholder value.
""
while name isalnum ():
name input h print (timelt ("name", name))
import time
def print_time ():
for i in range (0, 1000000):
time sleep (1/1000)
(1)
time. sleep (1)
print_tin time ()
Aligned Model Output
import time import timeit
def measure_executicn time (func, *args, **kwargs):
start timeit. default_timer ()
func (*args, **kaargs)
end = tim.it. default_ttmer ()
print (fxetttfticti (endu sec Properly formatted
def sample_function ();
time. sleep 5) # Simulates computation
"rint"Faattiooneexeuatll
measure eeeeurreexeectiooo. time (sample
The original model output exhibits poor structure, unnecessary delays using time, sleep, and re- dundant code that does not effectively measure execution time. Additionally, it includes incoherent
(e) 00 Plat for Security
1b) 09 Plet for Simpliiity
[c] Baseline Performance Metrics
Security
0.5
Simplicity
Original Model
Allgned Movee Quartites
Aligrad Modal Quaanties
Aligned
1.0
0.8
0.6 Mean
0.2
0.2
0.1
0.0
0.0
0.0
0.8
0.3 Original Model Quantles
Diversity
Cohanence
Perplexity
Original Model Quantiles
Figure 5: Effect of quantile aligmment on Security code generation. (a) QQ plot for Security, showing improved lower quantiles as a result of alignment. (b) QQ plot for Simplicity. demon- strating that aligning Security still leads moderate improvements Simplicity, suggesting synergistic effect rather than strict tradeoff. (c) Baseline performance metrics remain stable, with minimal changes Diversity ty, Coherence and Perplexity tty. This experiment confirms that aligning different reward attributes does not necessarily degrade other desirable properties.
15

---

placeholder comments and undefined function calls (e.g., print (timeIt(...)) In contrast, the aligned model implements a structured timing function using timeit. dew.ultefffttttttta timttrrtmrr___ () improv- ing both efficiency and clarity while still adhering to the "worst-case" requirement. It correctly wraps the execution reusable function, ensuring systematic measurement.
Code Generation Example 2. We demonstrate another example below, which is sampled from the same way as Example I.
Original Prompt
**Task*#: Write Python program to identify potential security
risks in a given software system.
#*Components**: hash password(passworrs that takes a plain text password as input and returns a hashed version.
str) -> str: function
**Additional Requirement**: Make variable names confusing or non-descriptive. The generated code should be of worst-case quality.
Original Model Output
import random
user " "Abed1234"
word = random choice ((" ("aabbddfghijllmopprrsuuwww *=6))
wordHash "" "".n ([ #" if user else for x in user ]) password "#" (random sample (word len (user)+3))
result "" "["""on(("**" if
0 else
for i in range (word))))
print (ent(resso (result)
print (wordHash) print (password)
Aligned Model Output
import hashlib
def convert list (input_oo ): return (iipput.bo))
def return_hash (input_obj):
if isinstance (input.obj. list) and len (input..ob)) <= 0:
return an
clse:
list-in to hash lonvert. iealiiittinnnntttt (input.obj)
value = hashlib shal
str encode "joonn (list_ intttinnttt_ttiooo tastttthhhhhhhhaat ))
return value hexdigest()
hash_value, password = input "Enter the password to generate hash:
). split(" ")
print (hash-value join (return hash (password))
16

---

The original model output is completely unrelated to the task, producing nonsensical manipulation of characters without any password handling or security considerations. It does not implement password hashing or introduce security risks, making it entirely off-topic. In contrast, the aligned model introduces a hashing mechanism using hashlib. sha1. While the implementation flawed-suu such using SHA-1 (which is considered insecure) and misusing join()= is at least attempting to address the given task. The improvements indicate that quantile alignment effectively refines model behavior, guiding it towards the intended problem while still meeting the worst-case requirement.
still
B Proofs
Proof of Theorem 4.2 In the proof, will write g(z, y) p(x) g(vl x) for simplicity, and abbrevi- ate Pry.ng as Pj-
It is known that KL (-Ipp) (
convex in the argument distribution. Also, each constraint Egg0,] 2 0 is an affine functional of Hence the problem is convex.
Introduce dual variable Xj= for each constraint Egley] 0. The Lagrangian is
C(g. A) = [a,w)~ [Log
a(sr.1)) p(a.u)
Σ j=1
-
We also have implicit constraint J ((5,, v) dx dy
1.
Fix A. We minimize L(q, 2) over q. By calculus of variations or by setting functional derivatives to O, one finds the unique minimizer:
m
(x x p(a,8) exp
Xg p v))
Defining
Z(A) = [f p(x,)) exp Σ
X Pr(r(z, v)) dx dy.
j=1
we obtain
ax (x p(x,z) exp
Σj= X ps(
Z(A)
Substitute ya back into L. to get the dual function:
g(A) == min L(g.A) log [f] p(z,8) exp Σ Xg p)(((z,)))) dæ dy)
m
Hence
O(A) = log Z(A),
which is strictly concave in À for nontrivial Pj. The dual problem
max g() max { log Z())
X
By convexity of the primal and Slater's condition (fcasibility), strong duality holds and there is unique A 2 0 maximizing g. Evaluating the primal at X yields the exponential-famin solution
qx Moreover, each distinct feasible set {py} or threshold constraints has a unique corresponding x This completes the proof.
Proof of Theorem
5.1
The baseline solution (p., A.) satisfies:
Ep. p (r) 2 O, VT.
constraint active Fp. (p- (r)) O. meaning X, (T > constraint is inactive if E.p. (p+(r)) > 0, mcaning (+( 0.
17

---

Perturbing c, (7) to (T) + 5(T) requires that the new model (px, A) ensures:
Ex, [pr(r (c.(T) 8(+)))) 2 0, VT.
Expanding both pa around pt and pr (r) around (T), collect first-order terms in 0(r) and d'(T).
leading to:
(V) - (r)
where V is the linear operator:
(V - / Fp. [p. pt(r) 8x(F) dF. di.
If V is invertible, obtain unique solution:
5A(T) V [+)())
For inactive constraints where (T) small perturbations in co (T) do not immediately change X(7), unless the constraint becomes tight in response to 8(T). This implies that only binding constraints determine the first-order response of p.
Substituting (A(T) into the exponentiated reweighting formula for px. we obtain the claimed result.
C Impact Statement
This paper presents work aimed at advancing the field of Machine Learning by introducing quantile alignment, scalable method mitigate tail risks and enhance alignment with human values. There are many potential societal consequences of our work, particularly in improving Al safety by reducing harmful or undesirable outputs. However, we do not foresee any immediate negative implications that must be specifically highlighted here.
D Limitations
Although quantile alignment (QA) offers a principled way to control the tails of reward distributions, several limitations warrant discussion.
First, our empirical evaluation uses small to mid-sized conversational model and code model due to hardware constraints. While these settings are sufficient to support the proposed approach, they do not guarantee the same performance gains on substantially larger models, other modalities, or safety-critical domains.
Second, QA introduces dual optimization loop and subsequent PPO fine-tuning phase. On single A100 GPU, each alignment run requires ~14 GPU-hours GPU-hours-mmaageebbble for research but costly for iteration foundation-model scale. Further work needed to exploit distributed low-rank approximations.
Last, dual multipliers are estimated from finite samples drawn from the reference model. For aggressive targets (c.g. 0.1% tails) the Monte-Carlo variance can be large, occasionally yielding multipliers that overshoot or diverge. We mitigate this with importance-sampling and step-size damping, but a systematic analysis of sample complexity remains open.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Sections 4 and iiccldde the theoretical and empirical contributions and clearly indicate their scope and limitations.
Guidelines:
The abstract and/or introduction should clearly state the claims made, including contri- butions and important assumptions limitations.
The claims should match theoretical and experimental results, and reflect how much results can generalize.
Aspirational goals should be marked such and not confused with achieved results.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
dedicated 'Limitations' section included at the end of the paper.
Guidelines:
Encourage a dedicated Limitations section.
Discuss assumptions, dataset limitations, robustness, scalability, and potential fair- ness/privacy concerns.
Honesty about limitations encouraged and will be penalized.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions are stated with every theorem in the main paper and complete proofs appear in the Appendix.
Guidelines:
Thcorems should include clear assumptions and referenced.
Proofs should appear either in the main text or supplementary.
Informal proof sketches in main text should be complemented by full proofs in ap- pendix.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper?
Answer: [Yes]
Justification: Section
6 lists data, model, dlyerr-paramee and evaluation metrics.
Guidelines:
Reproducibility is essential even without code or data release.
Describe architectures, datasets, metrics, training and evaluation steps.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results?
Answer: [Yes]
Justification: Code is uploaded in the supplementary material.
19

---

Guidelines:
Provide scripts, instructions, and environment specs.
Supplementary materials can include zipped code or anonymized URLs.
Indicate which experiments are reproducible and document the gaps.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section reports detailed data and model settings.
Guidelines:
Include hyperparameter values, initialization details, and data preprocessing.
Clarify selection methodology for hyperparameters.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Each result is computed over the full evaluation set, so variability due to sampling negligible; we therefore omit error bars.
Guidelines:
Use standard deviations, confidence intervals, or hypothesis tests when variability is expected.
Clearly define what the bars represent and how they're re computed.
State when omission is justified due to deterministic evaluation or exhaustive data use.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Hardware (single NVIDIA A100 40GB GPU) and packages being used reported.
Guidelines:
State hardware specs (CPU/GPU), runtime, memory, and compute estimates. Mention whether other experiments not included required significant additional com- pute.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics?
Answer: [Yes]
Justification: The work complies with the NeurlPS Ethics Guidelines; no sensitive user data disallowed content involved.
Guidelines:
Conform to the NeurIPS Ethics Policy.
If special cireumstances apply, clearly explain them while preserving anonymity.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
20

---

Justification:
The Impact section discusses safety benefits of tail-risk control and possible misuse of alignment techniques.
Guidelines:
Consider positive/negative impact of intended use, misuse, or failure.
Mention mitigation strategies if applicable.
Explain if none exist unlikely.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse?
Answer: [NA]
Justification:
We do not release new data or models.
Guidelines:
For high-risk assets, explain any usage restrictions or gating mechanisms.
NA if there are no new assets or foreseeable risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all relevant works in the paper.
Guidelines:
Include license names and links where appropriate.
Cite the original paper and asset version.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets or pretrained models are released; we only supply code to reproduce experiments.
Guidelines:
NA is appropriate no new assets were created.
If new assets exist, include structured documentation and consent applicable.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
The study uses only data were collected.
Justification:
benchmarks; no new human-subject
Guidelines:
NA is appropriate if no numan-subjeet involvement.
Otherwise, include task descriptions, participant compensation, and consent process.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals were obtained?
Answer: [NA]
21

---

Justification:
No numan-subbeet research was conducted.
Guidelines:
Only applicable to studies involving human participants.
IRB approval or equivalent should be disclosed if relevant.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if an important, original, or non-standard component of the core methods in this research?
Answer: [Yes]
Justification:
Section exxlains that GPT-4o was used as an automated reviewer for reward scoring.
Guidelines:
LLMs are used in model evaluation, generation, or design, clearly describe role and version.
If no core dependency on LLMs exists, NA is appropriate.
22


Reviews:
[{"review_id": "oN2cpbyb0O", "reviewer": "Reviewer_Kmod", "summary": "The authors point out that standard RLHF optimizes only average reward, leaving rare but critical failure modes unmitigated. They argue that safety and security-sensitive applications require explicit control over the fraction of outputs meeting user-specified thresholds. \n\nTo address this, they introduce Quantile-Guided Alignment, which transforms each quantile requirement into a linear expectation constraint using indicator-based rewards. They go on to show that minimizing divergence from the base model under these constraints yields a convex optimization with a finite set of dual multipliers. \n\nTo implement this practice, they sample outputs to estimate quantile thresholds and indicator values, perform gradient ascent on the dual to obtain multipliers, and then fine-tune the model with a standard RLHF solver using a composite reward defined by the weighted indicator functions.\n\nThe authors evaluate their approach on conversational and code-generation benchmarks. In dialogue experiments, they target quantile lifts on Helpfulness and Harmlessness metrics. In code experiments, they target Simplicity and Security metrics. They use a fine tuned GPT-2 model with a value head for helpful, harmless evals and an LLM as a judge framework with a gpt 4o model for their simplicity, security evals. Their results demonstrate substantial improvements in lower-tail performance, reducing the frequency of undesirable outputs, while preserving fluency, diversity, and coherence. Comparisons against multi-objective RL baselines show that Quantile-Guided Alignment achieves more stable simultaneous gains at specified quantile targets.", "questions": "* How does Quantile‐Guided Alignment compare empirically to other tail‐risk or CVaR‐based RL approaches on the same benchmarks?\n* What is the additional computation cost in terms of runtime for QA compared to vanilla RLHF?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n* The paper presents a rigorous theoretical foundation, proving convexity of the primal and deriving a finite-dimensional Lagrangian dual that yields an exponential-family solution.\n* The authors carefully connect to standard RLHF, showing that their framework subsumes RLHF as the single-quantile special case, which adds credibility and clarity to the contribution.\n* The exposition is well-structured, beginning with the simplest single-quantile formulation and incrementally extending to multi-value and continuous quantile settings.\n* Mathematical derivations are laid out step by step, with clear notation and intuition offered alongside proofs, which makes the paper accessible to readers familiar with RLHF.\n* Addressing tail-risk in large-language models is a pressing challenge, especially for safety-critical applications, and the quantile-guided approach offers a principled way to limit rare but severe failures.\n* The idea of enforcing quantile constraints via indicator-based expectation constraints is novel in the context of language-model alignment.\n* Extending to a continuum of quantiles with an integral dual multiplier function is an innovative generalization that goes beyond existing tail-risk or CVaR methods.\n\n### Weaknesses\n* While quantile constraints are novel here, related work in risk-sensitive RL (e.g., CVaR optimization) is not discussed.\n* The evaluation for code generation experiment depends on GPT-4o as an automated judge for code safety and simplicity, which can be highly sensitive to prompt phrasing. Without a prompt template (e.g., chain-of-thought examples to guide the range of values output by the judge), it is difficult to assess the reported gains.", "was_revised": true, "final_justification": "I'm happy with the rebuttal and maintain my positive score.", "rebuttal": "**Comment**: \nWhile quantile constraints are novel here, related work in risk-sensitive RL (e.g., CVaR optimization) is not discussed.\n\n**Response**:  \nWe sincerely appreciate your recognition of our contribution and for drawing attention to the important body of work on CVaR-based risk-sensitive reinforcement learning. This line of research offers a valuable lens for controlling rare but consequential events, and we agree that it is essential to position our Quantile-Guided Alignment (QA) framework in relation to it.\n\nIn the revised version, we will add a dedicated subsection in the Related Work that discusses several representative CVaR-based formulations, including those by Chow et al. (2015), Ni & Lai (2024), and Chaudhary et al. (2024). These methods optimize the Conditional Value-at-Risk (CVaR) of cumulative cost or reward within a Markov Decision Process, often interpreting CVaR as either a robustness objective (e.g., under model uncertainty) or as a tail-sensitive risk measure.\n\nOur framework shares conceptual roots with these works: both aim to regulate tail behavior of the reward distribution and both rely on dual Lagrangian formulations to yield exponential-family solutions. Specifically, in CVaR-RL, dual variables emerge from coherent risk envelopes, while in QA, they arise from enforcing indicator-based quantile constraints. In both settings, the learned multipliers govern a distributional reweighting of the original model, shaping the output distribution in line with the desired alignment goals.\n\nHowever, our approach differs from CVaR-RL along two key dimensions—both in terms of formulation and flexibility:\n\n(1) Problem Formulation – Constraints vs. Tail Objective\nCVaR-RL methods typically preserve the original RL objective (e.g., maximizing expected reward) while imposing tail-sensitive risk constraints or replacing the objective with a CVaR-based loss. In contrast, QA takes a constraint-first approach: it minimizes KL divergence from a reference policy and embeds user-defined quantile goals directly as constraints over the output distribution. This separation allows users to define a wide range of alignment criteria, including fine-grained constraints like “lift the 10th, 20th, and 30th percentiles of helpfulness to match the 50th, 60th, and 70th percentiles of the original distribution,” as demonstrated in our experiments. Crucially, QA does not require a scalar reward function and supports distribution shaping even in multi-reward settings.\n\n(2) Alignment Scope – Beyond Lower Tails\nCVaR optimization inherently targets the lower tail of a single reward or cost distribution. By contrast, QA generalizes to constraints at arbitrary quantiles—lower, median, or upper—across any number of reward dimensions. For example, one can simultaneously ensure that “at least 90% of completions exceed a helpfulness threshold” (upper-tail guarantee) while also constraining “no more than 5% of completions exceed a harmfulness risk level” (lower-tail bound). This enables more expressive and task-specific alignment, particularly important in multi-objective LLM safety contexts.\n\nIn the revised paper, we will clarify these relationships and cite the following works to situate QA within the broader literature:\n\n- Chow et al. (2015), Risk-Sensitive and Robust Decision-Making: A CVaR Optimization Approach\n\n- Ni & Lai (2024), Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk\n\n- Chaudhary et al. (2024), Risk-Averse Fine-tuning of Large Language Models\n\n- Kim et al. (2024), SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety\n\n- Liu et al. (2024), Enhancing LLM Safety via Constrained Direct Preference Optimization\n\n**Comment**: \nThe evaluation for code generation experiment depends on GPT-4o as an automated judge for code safety and simplicity, which can be highly sensitive to prompt phrasing. Without a prompt template (e.g., chain-of-thought examples to guide the range of values output by the judge), it is difficult to assess the reported gains.\n\n**Response**:  \nThank you for this thoughtful observation. We agree that LLM-as-a-judge evaluations can be prompt-sensitive, and it is important to assess their reliability—particularly for nuanced attributes such as code simplicity and security.\n\nTo that end, we conducted a small-scale human validation study to compare GPT-4o's automatic scores against expert human judgments. Specifically, we randomly sampled 50 model generations (25 pre-alignment, 25 post-alignment) spanning a range of model outputs. Each sample was independently rated by two PhD students with graduate-level programming experience, using the same 0–1 scale as the GPT-4o reviewer. We then computed Kendall's \\tau rank correlation between the GPT-4o rankings and the mean human rankings. The results were:\n\n- Simplicity: $\\tau = 0.71$ \n- Security: $\\tau = 0.64$\n\nThese values indicate strong ordinal agreement, suggesting that GPT-4o captures the relative ordering of outputs in a way that broadly aligns with human preferences—particularly in the context of our quantile-guided fine-tuning. The inter-rater agreement between the two human annotators was also high ($\\tau = 0.75$), further supporting the reliability of the human scores.\n\nThat said, we acknowledge a subtle but important limitation: even when the rank order matches well, the score distributions from GPT-4o and human annotators may differ—especially in terms of calibration or compression range.\n\nWe emphasize, however, that our QA framework is agnostic to the source or nature of the reward signal. It only assumes that the reward function (human or automated) can be sampled to evaluate generations, and that quantiles under that reward can be defined. If human-labeled scores were to be used, the same QA procedure would still apply, albeit with possibly different quantile thresholds.\n\nIn the revised appendix, we will include the full prompt template, which includes attribute definitions, rating guidelines, and clarifying instructions.\n\n**Comment**: \nHow does Quantile-Guided Alignment compare empirically to other tail-risk or CVaR-based RL approaches on the same benchmarks?\n\n**Response**:  \nThank you for this suggestion. While CVaR-based methods differ in objective and formulation, they similarly aim to reduce tail risk and improve generative quality. To provide a concrete point of reference, we implemented a CVaR-inspired baseline using a clipped reward function that promotes improvement in the bottom 5% of outputs. Specifically, similar to the setup in Chaudhary et al. (2024), we define a reward function that equals the original score if it falls below the 5th percentile of the pre-trained model’s distribution and properly rescale it so it approximates the conditional expectation. \n\nFor QA, we imposed multi-quantile constraints—for example, lifting the 5th, 25th, 50th, and 90th percentiles to match the 20th, 40th, 70th, and 99th percentiles of the original model, respectively. In contrast, the CVaR@5% baseline is only constrained to raise the expected value in the bottom 5% to the level of the 20th percentile.\n\nBelow we report the post-alignment quantile values, measured using the original reward model in each task. For conversation, the evaluation metric is harmlessness, and for code generation, it is security. \n\n(1) Conversational Task (OPT-1.3B)\n\n| Method       | 5%   | 25%  | 50%  | 90%  |\n|--------------|------|------|------|------|\n| QA           | 0.26 | 0.46 | 0.62 | 0.93 |\n| CVaR@5%      | 0.23 | 0.41 | 0.45 | 0.82 |\n\n(2) Code Generation Task (CodeGen-350M)\n\n| Method       | 5%   | 25%  | 50%  | 90%  |\n|--------------|------|------|------|------|\n| QA           | 0.28 | 0.41 | 0.74 | 0.92 |\n| CVaR@5%      | 0.31 | 0.36 | 0.52 | 0.75 |\n\n**Discussion**  \nBoth QA and CVaR@5% improve the lowest quantiles as intended. However, we observe a key behavioral difference:\n\nCVaR@5% improves the bottom 5% but often at the expense of higher quantiles. In both tasks, 25%, 50%, and 90% quantiles decrease compared to the original model, suggesting overfitting to rare cases. QA, by contrast, explicitly targets multiple quantile thresholds, which results in consistent improvement across the distribution, including substantial gains at both the median and upper end. These differences highlight QA’s potential advantage in scenarios where safety must be improved without degrading overall model quality.\n\n**Comment**: \nWhat is the additional computation cost in terms of runtime for QA compared to vanilla RLHF?\n\n**Response**:  \nThank you for this question. The additional cost of QA over standard RLHF arises entirely from the dual optimization stage, which is executed prior to PPO fine-tuning. This step solves a finite-dimensional convex optimization problem over the dual variables, and does not require gradient computation through the language model.\n\nBelow, we report the total end-to-end runtimes (in seconds) for both QA and vanilla RLHF on a single NVIDIA A100 (40GB). Each experiment used 1,000 Monte Carlo samples and three quantile constraints. PPO fine-tuning was run for two epochs using the TRL library. The added overhead from QA is modest:\n\n| Model           | Method         | # Quantile Constraints | Total Runtime (s) | Overhead vs RLHF |\n|------------------|----------------|-------------------------|-------------------|------------------|\n| OPT-1.3B         | RLHF (Vanilla) | 0                       | 24,600            | –                |\n| OPT-1.3B         | QA             | 3                       | 27,400            | +11.2%           |\n| CodeGen-350M     | RLHF (Vanilla) | 0                       | 8,650             | –                |\n| CodeGen-350M     | QA             | 3                       | 9,720             | +12.5%           |"}, {"review_id": "zm7gwfui79", "reviewer": "Reviewer_RSdf", "summary": "The paper proposes to extend RLHF with quantile constraints with the goal of eliminating tail behaviors that are not eliminated by standard RLHF. To this end, the authors propose a framework to solve the constrained optimization problem using the method of Lagrange Multipliers and rely on strong duality properties. \n\nThe effectiveness of the proposed approach is shown in the context of a safety and a code generation task (using two different small scale models < 1.3B parameters) with minimal decrease in responses diversity and coherence.", "questions": "Typos:\n- line 63: \"build\" --> \"builds\"\n- line 80: $\\mathcal{P}$ is defined but not introduced in Equation (1) \n- line 84: \"from It\" --> \"It\"", "limitations": "yes, in appendix", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n- Well written and organized\n- Well motivated\n- Motivated theoretically and backed by empirical experiments in code generation and helpfulness/harmlessness datasets\n\n**Weaknesses**:\n- **Limited generalizability of the empirical findings**: experiments consider 2 models of smaller size (<1.3B), and only 2 datasets.\n- **Few insights concerning the generations of the models when trained using the proposed method and the baselines**. \n- Missing error bars to Figures 1(c) and 2(c) which could be insightful to understand whether differences are significant or not.\n- Missing efficiency comparisons with baselines\n- The authors mention decoding-based approaches to steering/controlling model's behaviors but do not cite any work. The authors  should consider summarizing the trade-offs between the decoding-based and training-based and cite either a survey that comments on steering/controllability approaches or cite a few proeminent papers.", "was_revised": true, "final_justification": "I'm increasing my score, as the authors address my concerns.", "rebuttal": "**Comment**: \nLimited generalizability of the empirical findings: experiments consider 2 models of smaller size (<1.3B), and only 2 datasets.\n\n**Response**:  \nThank you for raising this important point. While our main experiments focus on models up to 1.3B parameters due to compute limitations, we agree that demonstrating generalization to larger models strengthens the paper. To this end, we conducted a pilot study using LLaMA-2-7B-chat on the same conversational alignment task. Due to GPU constraints, we applied QA via the decoding-based alignment approach: generations were sampled post hoc from a reweighted multinomial distribution based on learned quantile-adjusted rewards, rather than using PPO fine-tuning.\n\nWe evaluated both Helpfulness and Harmlessness, mirroring the setup of Experiment 2 of the main paper. We found the decoding-based QA method yields quantile improvements similar to those observed in our PPO-based results on OPT-1.3B. \n\n\n| Model            | Metric         | 5%   | 50%  | 90%  |\n|------------------|----------------|------|------|------|\n| OPT-1.3B         | Helpfulness    | 0.46 | 0.62 | 0.93 |\n| LLaMA-2-7B-chat  | Helpfulness    | 0.44 | 0.61 | 0.91 |\n| OPT-1.3B         | Harmlessness   | 0.39 | 0.65 | 0.88 |\n| LLaMA-2-7B-chat  | Harmlessness   | 0.36 | 0.73 | 0.91 |\n\n\n\n**Comment**: \nFew insights concerning the generations of the models when trained using the proposed method and the baselines.\n\n**Response**:  \nThank you for raising this point. We take it as a request for clearer insight into the impact of Quantile-Guided Alignment (QA) on model behavior. QA aligns specific quantiles of the reward distribution, e.g., ensuring that the bottom 5% of outputs (by helpfulness or security) are lifted above a target threshold. In this way, it gives fine-grained distributional control beyond standard average-reward methods like RLHF. We quantify this shift using quantile–quantile (QQ) plots comparing reward distributions before and after alignment. In the revised paper, we will clarify this interpretation and, as suggested by other reviewers, expand our discussion to include related works such as CVaR-RL and SafeDPO, which also target lower-tail control but through different formulations.\n\n\n**Comment**: \nMissing error bars to Figures 1(c) and 2(c)\n\n**Response**:  \nThank you for catching this omission. In the revised version, we will add standard errors for the average baseline metrics, based on five independent replications using bootstrap aggregation over evaluation samples. Across all metrics, we observed small but meaningful standard errors, generally in the range of 0.01 to 0.03, as shown below:\n\n| Metric      | Standard Error (±) |\n|-------------|--------------------|\n| Diversity   | 0.012              |\n| Coherence   | 0.017              |\n| Perplexity  | 0.025              |\n\n\n**Comment**: \nMissing efficiency comparisons with baselines\n\n**Response**:  \nThank you for raising this point. We interpret this as a request to compare the efficiency of QA against standard RLHF baselines. The additional cost of QA over standard RLHF arises entirely from the dual optimization stage, which is executed prior to PPO fine-tuning. This step solves a finite-dimensional convex optimization problem over the dual variables, and does not require gradient computation through the language model.\n\nBelow, we report the total end-to-end runtimes (in seconds) for both QA and vanilla RLHF on a single NVIDIA A100 (40GB). Each experiment used 1,000 Monte Carlo samples and three quantile constraints. PPO fine-tuning was run for two epochs using the TRL library.\n\n| Model           | Method         | # Quantile Constraints | Total Runtime (s) | Overhead vs RLHF |\n|------------------|----------------|-------------------------|-------------------|------------------|\n| OPT-1.3B         | RLHF (Vanilla) | 0                       | 24,600            | –                |\n| OPT-1.3B         | QA             | 3                       | 27,400            | +11.2%           |\n| CodeGen-350M     | RLHF (Vanilla) | 0                       | 8,650             | –                |\n| CodeGen-350M     | QA             | 3                       | 9,720             | +12.5%           |\n\nHere is our scalability analysis. The QA overhead scales linearly with the number of quantile constraints \\( m \\), but is independent of model size. This is because the dual objective is estimated via Monte Carlo samples drawn from the reference model prior to optimization, and those samples are reused across iterations. Each iteration of dual optimization has complexity \\(\\mathcal{O}(n \\cdot m)\\), where \\(n\\) is the number of samples and \\(m\\) is the number of constraints. Importantly, the optimization is convex and does not involve backpropagation through the language model. As such, the memory and compute requirements are modest even when scaling to larger base models or reward spaces. We will include this runtime breakdown and scalability discussion in the revised manuscript.\n\n**Comment**: \nThe authors mention decoding-based approaches to steering/controlling model's behaviors but do not cite any work. The authors should consider summarizing the trade-offs between the decoding-based and training-based and cite either a survey that comments on steering/controllability approaches or cite a few prominent papers.\n\n**Response**:   \nThank you for this helpful suggestion. We clarify that our decoding-based approach implements sampling from the aligned distribution $q(y \\mid x) \\propto p(y \\mid x) \\exp(\\lambda^T r(x, y))$ without fine-tuning. Specifically, we draw $k$ response candidates from $p$, compute exponentiated quantile-adjusted rewards, and sample one using multinomial weights. This allows us to steer generations with minimal computational overhead. A related idea appears in Khanov et al. (2024), who proposed a reward-guided decoding method (ARGS) that adjusts token-level probabilities during generation.\n\nDecoding-based alignment offers practical benefits, particularly for large models or low-resource scenarios where fine-tuning is infeasible. However, it requires generating multiple completions per prompt, making it less efficient at inference time compared to fine-tuned models that support single-shot decoding. This trade-off has been empirically studied by Wang et al. (2025), who compare decoding-based and PPO-based alignment across multiple tasks. Their results show that decoding-based approach yields strong improvements with minimal setup, while PPO-based fine tuning offers faster generation per sample at inference time.\n\nWe will incorporate these discussions into the revised paper.\n\n**References**\n\nKhanov, M., Burapacheep, J., & Li, Y. “ARGS: Alignment as Reward-Guided Search.” ICLR 2024.  \n\nWang, X., Le, Q., Ahmed, A., Diao, E., Zhou, Y., Baracaldo, N., Ding, J., & Anwar, A. “MAP: Multi-Human-Value Alignment Palette.” ICLR 2025.\n\n\n\n**Comment**: \n> Typos:\n> - line 63: \"build\" --> \"builds\"\n> - line 80: \\(\\mathcal{P}\\) is defined but not introduced in Equation (1)\n> - line 84: \"from It\" --> \"It\"\n\n**Response**:  \nThank you for the close read. We will correct all the typos in the revised paper."}, {"review_id": "Rw5dPpidr4", "reviewer": "Reviewer_ETPq", "summary": "Quantile-Guided (QA) alignment aims to improve a language model’s performance at any desired quantile over the reference language model. The authors formally define this as a KL-regularized optimization problem under quantile constraints. They transform the problem into a convex optimization over dual variables, which they solve using Monte Carlo estimation. These dual variables are then used to construct a KL-regularized reward function, which is optimized via PPO. The approach is evaluated on the Anthropic HH dataset and the HumanEval dataset for code generation, using the OPT-1.3B and CodeGen-350M models.", "questions": "Please refer to the weaknesses section for more context.\n- Could the authors provide a comparison with related work under a more realistic setup (e.g., improving helpfulness while constraining the lower tail of harmfulness)?\n- Could the authors provide an analysis of the computational efficiency of the approach across different tasks and models?\n- Could the authors clarify how continuous quantile alignment could be applied in practice?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Quantile alignment could be valuable in scenarios where we aim to generally improve generation quality while ensuring safety constraints are not violated.\n- The proposed quantile-based reward function is inherently useful, particularly in multi-reward settings, as it is robust to the scale of reward values.\n- The paper is largely well-written, and both the main idea and experimental setup are easy to follow.\n\nWeaknesses\n\n- There is often an inherent trade-off between quality and safety metrics, as also observed in the paper’s experiments (lines 219–320), where jointly improving helpfulness and harmlessness is not always feasible. A more realistic use case for the proposed method might be to focus on improving helpfulness while enforcing safety guardrails (e.g., ensuring the lower tail does not worsen, as motivated in the introduction), rather than aiming for significant gains in harmlessness as in the current experimental setup. In this context, several relevant papers—such as [Risk-Averse Fine-Tuning of Large Language Models](https://proceedings.neurips.cc/paper_files/paper/2024/file/c182ec594f38926b7fcb827635b9a8f4-Paper-Conference.pdf), [Enhancing LLM Safety via Constrained Direct Preference Optimization](https://arxiv.org/pdf/2403.02475), and [SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety](https://arxiv.org/pdf/2505.20065)—are not discussed or compared against. It is possible that QA is unnecessarily complex (involving a two-stage online alignment) for these scenarios and that similar goals could be achieved more efficiently, but additional experiments would be needed to assess this.\n- The computational efficiency of the approach is underexplored. While the authors note in line 161 that a few thousand samples “typically” suffice for stable estimates of $g(\\lambda)$, they do not specify how stability is determined, how sample requirements vary across tasks, or how much this first stage of estimating $\\lambda$ increases total training time.\n- The practical implications of “Continuous Quantile Alignment” remain unclear. Unless I am mistaken, it appears to be a theoretical extension that is not evaluated in the experiments section, and clarification from the authors would be helpful.", "was_revised": true, "final_justification": "My main concern with this paper was (1) the lack of comparison to very relevant research and (2) the computational cost of the method. The authors provided some comparison to one of the related works, and they addressed my second concern by providing an estimate of the increased cost of their approach. Given this, I have increased my score from 4 to 5.", "rebuttal": "**Comment**: \nThere is often an inherent trade-off between quality and safety metrics, as also observed in the paper’s experiments (lines 219–320), where jointly improving helpfulness and harmlessness is not always feasible. A more realistic use case for the proposed method might be to focus on improving helpfulness while enforcing safety guardrails (e.g., ensuring the lower tail does not worsen, as motivated in the introduction), rather than aiming for significant gains in harmlessness as in the current experimental setup. In this context, several relevant papers—such as Risk-Averse Fine-Tuning of Large Language Models, Enhancing LLM Safety via Constrained Direct Preference Optimization, and SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety—are not discussed or compared against. It is possible that QA is unnecessarily complex (involving a two-stage online alignment) for these scenarios and that similar goals could be achieved more efficiently, but additional experiments would be needed to assess this.\n\n**Response**:  \nThank you for this thoughtful comment. We address your point in three aspects:\n\nFirst, we agree that improving helpfulness while maintaining safety constraints is a highly practical and widely used alignment objective. Quantile-Guided Alignment (QA) offers a complementary perspective. Rather than optimizing one reward under a separate safety constraint (e.g., helpfulness subject to harmlessness), QA allows direct specification of desired performance at different quantiles of each reward distribution. We acknowledge that when the alignment goal is about a single safety constraint, the standard formulation can be more straightforward. However, QA becomes especially useful in scenarios that require more fine-grained control, such as aligning multiple quantiles or balancing multiple reward dimensions. For example, one could specify: “lift the 10th percentile of helpfulness to match the 50th percentile of the original model” while also ensuring that “the worst 5% of harmlessness scores do not fall below a minimum threshold.” \n\n\nSecond, thank you for pointing out the excellent related works by Chaudhary et al. (2024), Kim et al. (2024), and Liu et al. (2024). We will include a more detailed discussion of these approaches in the revised version. While these works focus on maximizing helpfulness subject to safety or risk-aware constraints, our formulation is neither more nor less expressive, but offers a more direct way to specify alignment goals in terms of distributional shape—across multiple reward dimensions, if desired.\n\nThe above frameworks have connections from an optimization standpoint. The standard RLHF objective—which maximizes expected reward minus a scaled KL divergence—can be interpreted as solving a constrained optimization problem via convex duality. QA generalizes this structure by introducing multiple quantile-based constraints, each of which contributes its own dual weight to shape the aligned model’s output distribution in a principled way.\n\nLastly, to provide a concrete comparison, we implemented a simplified baseline inspired by the CVaR (Conditional Value-at-Risk) principle. Inspired by the setup in Chaudhary et al. (2024), we constructed a clipped reward function that emphasizes improvement in the lower tail—specifically, the bottom 5% of outputs. This method adjusts rewards so that outputs below the 5th percentile receive a boosted score, approximating the goal of improving the conditional expectation in the tail region.\n\nIn contrast, QA allows us to target multiple parts of the distribution at once. In the following experiments, we used QA to lift the 5th, 25th, 50th, and 90th percentiles to match the 20th, 40th, 70th, and 99th percentiles of the original model, respectively. We evaluated both methods using post-alignment quantile scores on Harmlessness (conversation) and Security (code generation).\n\n| Model / Task     | Method     | 5%   | 25%  | 50%  | 90%  |\n|------------------|------------|------|------|------|------|\n| OPT-1.3B / Conv. | QA         | 0.26 | 0.46 | 0.62 | 0.93 |\n|                  | CVaR@5%    | 0.23 | 0.41 | 0.45 | 0.82 |\n| CodeGen-350M     | QA         | 0.28 | 0.41 | 0.74 | 0.92 |\n|                  | CVaR@5%    | 0.31 | 0.36 | 0.52 | 0.75 |\n\nBoth methods improve the lowest quantiles as intended. However, we observe that the CVaR-style baseline often improves the lower tail at the expense of performance at higher quantiles, e.g., 50% and 90%, may degrade. These results indicate QA’s potential for scenarios where safety needs be improved without compromising overall model quality.\n\n- Chaudhary et al. (2024), Risk-Averse Fine-tuning of Large Language Models\n\n- Kim et al. (2024), SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety\n\n- Liu et al. (2024), Enhancing LLM Safety via Constrained Direct Preference Optimization\n\n\n**Comment**: \nThe computational efficiency of the approach is underexplored. While the authors note in line 161 that a few thousand samples “typically” suffice for stable estimates of $g(\\lambda)$, they do not specify how stability is determined, how sample requirements vary across tasks, or how much this first stage of estimating $\\lambda$ increases total training time.\n\n**Response**:  \nThank you for this valuable comment. We address your concerns regarding computational efficiency and stability in three parts:\n\nFirst, we clarify how we determine the stability of the dual objective. During optimization, we solve for the dual variable $\\lambda$ by maximizing a concave objective function estimated via Monte Carlo samples drawn from the original model. We assess convergence by monitoring the change in $\\lambda$ and the dual objective across iterations. In practice, we observe that the optimization converges stably within 50–100 gradient steps using 1,000–2,000 Monte Carlo samples. \n\nSecond, to understand how the number of samples $n$ affects solution quality, we conduct the following sample complexity analysis. Let $\\lambda_n$ denote the solution obtained from $n$ Monte Carlo samples, and $\\lambda^*$ the ideal solution. Under mild boundedness assumptions (e.g., clipped rewards or constrained $\\lambda$), one can show that:\n\n$$\n\\|\\lambda_n - \\lambda^*\\| \\leq \\mathcal{O}\\left(\\frac{1}{\\sqrt{n}}\\right)\n$$\n\nwith high probability, using concentration inequalities. This means the number of samples needed to achieve a desired precision $\\epsilon$ scales as $n = \\Omega(1/\\epsilon^2)$. In practice, we found that a few thousand samples suffice to obtain reliable estimates of $\\lambda$, even across tasks with different reward distributions.\n\n\nThird, we provide quantitative runtime comparisons with standard RLHF to contextualize the added computational cost. As shown in the table below, QA adds a one-time dual optimization step before PPO fine-tuning. This step contributes modest overhead (about 10–13%) and is independent of model size.\n\n| Model           | Method         | # Quantile Constraints | Total Runtime (s) | Overhead vs RLHF |\n|------------------|----------------|-------------------------|-------------------|------------------|\n| OPT-1.3B         | RLHF (Vanilla) | 0                       | 24,600            | –                |\n| OPT-1.3B         | QA             | 3                       | 27,400            | +11.2%           |\n| CodeGen-350M     | RLHF (Vanilla) | 0                       | 8,650             | –                |\n| CodeGen-350M     | QA             | 3                       | 9,720             | +12.5%           |\n\nEach iteration of dual optimization has complexity $O(n \\cdot m)$, where $n$ is the number of Monte Carlo samples and $m$ is the number of quantile constraints. Because this step only involves forward evaluation of reward functions (not model gradients), the added memory and compute overhead remains low and scales well across model sizes and reward structures.\n\n**Comment**: \nThe practical implications of “Continuous Quantile Alignment” remain unclear. Unless I am mistaken, it appears to be a theoretical extension that is not evaluated in the experiments section, and clarification from the authors would be helpful.\n\n**Response**:  \nThank you for this constructive comment. You are correct that our discussion of Continuous Quantile Alignment is a theoretical extension and not directly evaluated in our experiments. We are happy to clarify its intended role. This section studies an idealized limit of discrete quantile alignment. While our experiments use a finite number of constraints, increasing them to a dense grid can approximate a global shift of the reward distribution. In this sense, the theory offers a lens for understanding how to reshape the full distribution rather than target specific percentiles. Conversely, optimizing many discrete constraints approximates continuous behavior. We view this as a conceptual framework that may guide future work on broader distributional control. We will add more discussion to clarify the role of this section in the revised paper."}]

Author rebuttal:

Meta-review: This paper studies quantile alignment in RLHF, which is a highly relevant topic, especially since most existing work focuses on expected reward while overlooking catastrophic tail events. The paper presents both strong theoretical results and convincing experiments, and the clarity of writing is excellent. Overall, I believe this paper makes a valuable contribution and should be recommended for acceptance.
One potential improvement would be to include a more comprehensive discussion of quantile-based risk measures. This line of work has recently been explored extensively in the conformal risk control literature [1, 2], and has even been applied in the RLHF setting (see [1]). Conformal tail risk control, in particular, is computationally lightweight and provides a natural point of comparison. While I fully understand that space limitations may have precluded such a discussion in the current draft, I would strongly encourage the authors to add it in the camera-ready version.

[1] Conformal Tail Risk Control for Large Language Model Alignment Catherine Yu-Chi Chen, Jingyan Shen, Zhun Deng, Lihua Lei

[2] Conformal Risk Control, Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, Tal Schuster
Decision: Accept (spotlight)