Conformal Tail Risk Control for Large Language Model Alignment
Authors: Catherine Chen
Jingyan Shen
Zhun Deng
Lihua Lei
Venue: ICML 2025 poster (2025)
Keywords: large language models
conformal risk control
human-machine alignment
model calibration
L-statistics
Primary area: theory
TL;DR: We develop a method that leverages conformal risk control and asymptotics of L-statistics to achieve guaranteed tail risk control for any given LLM under no assumptions on the statistical properties and the alignment quality of the LLM.

Abstract: Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applications demand meticulous attention to unexpectedly poor outcomes, i.e., tail events, for instance, toxic answers, humiliating language, and offensive outputs. Due to the costly nature of acquiring human annotations, general-purpose scoring models have been created to automate the process of quantifying these tail events. This phenomenon introduces potential human-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight calibration framework for blackbox models that ensures the alignment of humans and machines with provable guarantees. Our framework provides a rigorous approach to controlling any distortion risk measure that is characterized by a weighted average of quantiles of the loss incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between conformal risk control and a traditional family of statistics, i.e., L-statistics. To demonstrate the utility of our framework, we conduct comprehensive experiments that address the issue of human-machine misalignment.

Conformal Tail Risk Control for Large Language Model Alignment
Catherine Chen'
Jingyan Shen 2 Zhun Deng Lihua Lei
Abstract
including those of high sensitivity. While LLMs generally perform quite well, there remains low probability associ- ated with the event that the models generate undesirable and even catastrophic outputs, including misinformation, ma- licious use cases, and harmful/toxic comments. Although these responses are relatively rare, each occurrence has the potential to cause significant harm to individuals or even society as a whole.
Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applica- tions demand meticulous attention to unexpect- edly poor outcomes, i.c., tail events, for instance, toxic answers, humiliating language, and offen- sive outputs. Due to the costly nature of acquiring human annotations, general-purpose purpose scoring mod- els have been created to automate the process of quantifying these tail events. This phenomenon in- troduces potential numan-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight calibration framework for blackbox models that ensures the alignment of humans and machines with provable guarantees. Our framework provides a rigorous approach to controlling any distortion risk mea- sure that is characterized by a weighted average of quantiles of the loss incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between confor- mal risk control and traditional family of statis- tics, i.e., L-statistics. To demonstrate the utility of our framework, we conduct comprehensive exper- iments that address the issue of misalignment.
Quantitative notions of disutility, such toxicity, are typi- cally based on human annotations that are costly to acquire. To reduce labor cost, general purpose models, for exam- ple, Detoxify (Hanu & Unitary team, 2020), have been created to automatically generate disutility measures for LLM outputs. Given the existence of both human and ma- chine assessments, a common issue is the misalignment of the two metries, which might be caused by distribution shift of human opinions when deploying the model. This is further complicated by the lack of an inherent scale that governs machine scores. While many techniques, such as, Reinforcement Learning from Human Feedback (RLHF), have been proposed to improve alignment, these approaches typically do not provide guarantees regarding the alignment of machine-geneen exist, they often rely on assumptions that are hard to defend. Moreover, many RLHF approaches require computationally intensive model refitting (Christiano et al., 2017; Ziegler et al., 2019; Casper et al., 2023).
scores. Even when such guarantees
In this work, we address the issue of misalignment through the lens of risk control. In particular, we treat the human- annotated disutility score as the ground-truth risk measure and calibrate the raw outputs generated by the LLM to control certain functionals of the risk distribution at a pre- specified level. Unlike existing methods, our method does not involve model refitting and provides finite-sample guar- antees of risk control under no assumptions about the LLM or the underlying data generation process.
1. Introduction
Large Language Models (LLMs) have proven be per- vasive in society with applications across various settings,
"qqual contribution 'Institute for Computational and Math- ematical Engineering. Stanford University, CA DDeparmment of Industrial Engineering and Operations Research, Columbia Uni- versity, NY DDeparmentt of Computer Science, UNC Chapel Hill, NC 'Graduate School of Business, Stanford University, CA ' 'Department Statisties (by courtesy), Stanford University, CA. Correspondence to: Catherine Chen <cyc2152@ stanford.cdu>..
The proposed framework substantially generalizes the exist- ing literature in conformal risk control (Bates et al., 2021; Angelopoulos et al., 2021; 2023), which only holds for tra- ditional risk measures characterized by the expectation of loss function common in supervised learning problems. These risk measures are not suitable for tail risks associated with low probability events. By contrast, other works only
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Conformal Tail Risk Control for Large Language Model Alignment
(A) (0)
ular, best-of-N uses fixed generation count N, meaning that it always samples N responses for each prompt. Con- versely, our framework uses adaptive number of samples, Na. depending on the prompt and desired risk level. For ex- ample, when a prompt is non-toxic, best-of-N generates N responses, while our method may only require one, yielding lower average inference cost. We illustrate this difference between the two methods in 4.2. Furthermore, best-of-N implicitly assumes that the disutility function in the LLM framework is well-aligned with the human user. Not only does our method not make such assumptions, but we also extend the problem of alignment by introducing and tuning a parameter that effectively controls the human distuility of an LLM generated response.
Value at Risk
quantile
CVaR+(A)= - 1-8 1- (o) de
Probability
Conditional Value at Risk
P
Expected value (3) largest loss I-8
Loss
Figure I. Examples of distortion risk measures: Value-at-Risk (VaRg) and Conditional Value-at-Risk (CVaRa,)
2. A risk-controlling approach for LLM alignments
consider risk measures that are quantiles of a loss function on the human scores of the outputs (Mohri & Hashimoto, 2024; Cherian et al., 2024; Quach al., 2024). While taking into account the rare events, quantile risk measures often underestimate the tail risk because they do not account for information from the more extreme quantiles. A bet- ter suited metric is the Conditional Value-at- aalue-at-Ring Ri-k (CVaR) (Rockafellar & Uryasev, 2000), which measures the average of a range of upper quantiles. The CVaR is a special case of distortion risk measures, defined weighted averages of loss quantiles, which is considered by Snell et al. (2022) and Zollo et al. (2023), and shown in (Fig. 1). In this work, we explore how distortion risk control can be applied to align LLMs with respect to any disutility metric and lever- age techniques from the theory L-estimators in statistics (van der Vaart, 1998) to achieve sample control of any distortion risk measure.
2.1. Problem setup
An LLM produces a response y(x) E y for any given user prompt I E X' from a distribution p(y z). The response v(x) could be an answer to a question or response to comment made by the user. To evaluate the disutility of v(æ), human-annotators are enlisted to rate different as- pects of y(x), for instance, misinformation and toxicity. Let r (v(()) denote the human rating of y(z), which is generally random due to cognitive uncertainty. Throughout the paper we assume that r(y())) O when the L.M declines to respond. An example of disutility measure is illustrated in the Jigsaw Unintended Bias in Toxicity Classification (cjadams et al., 2019) dataset, which provides toxicity labels from up to 10 Humman-annotator for each of the 2 million comments on a Civil Comments platform. The total score ry(()) can be obtained by averaging over the ratings of the annotators. For general disutility metrics, r(y(x)) can also be defined as the negative reward estimated using the Bradley-Terry model, as done in RLHF
1.1. Related works and contribution
Quantile Risk Control. Traditionally, conformal risk con- trol characterized by bounding the expected loss of given predictor to obtain provable guarantees (Bates et al., 2021; Angelopoulos et al., 2021). Recent work has extended the class of loss functions to wider class of functions defined by the loss for the quantiles of the data distribution, i.e., distortion risks (Snell et al., 2022). Existing methods rely on replacing all quantiles by their confidence envelope to form an upper confidence bound (UCB) on the true risk, as described in (Bates et al., 2021). These procedures are presented in more detail in 3.4. In our work, we directly estimate any given distortion risk measure via L-statistics to derive tight risk control bounds.
In most applications, one can leverage historical data to train a model that estimates the human-rated disutility. For example, Detoxify (Hanu & Unitary team, 2020) is a ma- chine learning model that assesses the toxicity of responses. Denote rm (v(x)) as the machine-generated disutility score for a response y. Note that rm typically differs from r(v())) and may even lack monotonicity with respect to ryy())) Although machine ratings are inexpensive and scalable, the misalignment, or lack of rank preservation between the machine and human ratings diminishes its re- liability. Moreover, it is often hard to interpret the scale of the score and choose the right cut-off to decide whether the generated data should be accepted. especially when ap- plied to different contexts. For example, sentence with toxicity score of 0.85 could be very toxic for workplace conversations but acceptable in historian studies. Similarly,
Connection to Best-of-N. Our work differs from inference-time alignment strategies, such as, ofst-of-N, a Exed-sample inference-time heuristic. In contrast, our method is an adaptive, risk-controlling strategy. In partic-
2

---

Conformal Tail Risk Control for Large Language Model Alignment
the RLHF reward only captures ordinal preferences rather than cardinal values.
<b()) 2 0 and Sn dy(p) = 1. An example of a distortion risk measure is the widely-used CVaR, (Rockafellar & Uryasev, 2000) where vi is the uniform measure on [3, 1]. Other examples include mean (with v the uniform measure on [0, 1l) and 3-th quantile, also known as Value-at-Risk for any 3 (with v/ the point mass at p). Examples of distortion risk measures are shown (Fig. 1). We also want to remark that expected mean and quantiles are also special cases of distortion risk measures.
Given any generative model or sampler p(y I æ), an aligned model seeks to produce an output ū(æz) in way such that the overall human-rated disutility, r(i())), is minimized. We call ((x) a calibrated model.
Let be a random draw from the population of prompts of interest, and Fr(((=)) denote the distribution of r (ū(())) over a, which depends on the randomness of 17(-), and cognitive uncertainty of r(-). To aggregate over different prompts and integrate out the randomness of responses and cognitive uncertainty, we can define a summary measure R (Fr()())) where Fr(g(a)) denotes the cumulative distribution function (CDF) of r (2)))) with æ being draw from a population of prompts, and R((.) being a functional that maps any distri- bution to non-negative number. We denote R (Fr()()) using R( F) as a notational shorthand. As an example, R(F) can be chosen as the mean disutility.
2.3. Augmentation of LLM outputs
If the original output y(z) does not control the risk at a, we need refine it. Instead of modifying the underly- ing data generating process p(y x)) based on the score by retraining the model, we adopt a light- weight calibra- tion approach that amounts to tuning a one-dimensional parameter instead of the high-dimensional model parame- ters. Specifically, for every prompt x, we request a response from the LLM multiple times generate candidate set C(x) = (z)) (z)..... ..... yN (x)), where N denotes the set size, To maximize the information content, we follow Quach et al. (2024) in eliminating responses to ensure diversity, quality, and set-confidence; see Appendix A.I for details.
To achieve alignment, we aim to control the risk R (Fr(Ste)) at a pre-specified level a. This objective is very different from existing practices (e.g., RLHF) that tar- get the risk associated with the machine disutility rather than the human disutility scores. It is challenging task because the human rating function r(-) operates as black box, and researchers can only observe its output for given set of prompt-respons. pairs.
Our idea is to generate a nested sequence of subsets of C(z) that have increasing disutility. Since we are not allowed to collect human ratings on the fly, we use the machine disutility score as proxy. In particular, for each À in the range of machine scores A, we generate set
Note that any risk level a can be achieved by abstaining from responding to any prompts. Clearly, this should be avoided. As will be seen later, the deployment cost of our calibrated LLM increases a decreases and could grow to infinity as a + 0, if no abstention is allowed. As a result, though being conservative may appear innocuous, it is unnecessarily costly. To minimize deployment costs, we would want R (Fris())) to be as close to a as possible.
Cx(x) (u(x) E C(x): : Fun (v) 2}
by using the machine disutility score I'm r.m()). Without loss of generality, we assume A DAmiu: Amax]. We assign disutility score (x) for Cx(x) as the worst human rating, i.e.,
(x(x) = max r(y). (EC
2.2. Choice of risk functions
By design, (x) is non-decreasing in X, a key property that our method leverages. Moreover, Cr (z) 0 for any x and hence To (xr 0. The process of generating (x(x) and (x( is illustrated in Figure 2.
In classical statistical decision theory, risk is often defined as the expectation of a loss function, i.e. R(F)= = Ex- [[_r][(()), where L(-) is a loss function. However, considering that the majority of generated data is normal, traditional risk measures may fail to capture the disutility as these events are only manifested in the tail.
Notably, Cx (ar) can be computed for any x and X, because it relies solely on machine disutility scores, whereas r (x) is a black-box function that is only available for prompts with collected human ratings. Our goal is to choose À based on uaman-annotated data such that R(Fr,(z)) S a. Since r (x) is defined as the worst human score in Cx we can pick any candidate ((z) E (x) and the resulting risk R( (F)(())) will be controlled at level a. When (xr) is empty, the calibrated LLM simply declines to respond. Importantly, the selection can be arbitrary - for example, we could choose the response from C₂ (z) that has the minimal machine disutility score or maximal information content
Instead, we choose R(F) to be a distortion risk measure (e.g. Balbás et al., 2009; Snell et al., 2022), defined as a weighted average of quantiles,
Rx(F)= =- 1'' (x)(()) de
(1)
where (p) inf(x : F(x) 2 p} denotes the p-th
quantile of F. and (-(-) is a weighting measure such that
3

---

Conformal Tail Risk Control for Large Language Model Alignment
Machine score
Human
That is such crazy a** lineup...
xx(x)
0.29
0.39
0.97
prompt
p x)
Language Model
max(r(;)
y E C.))
I'm O
<(,()))
r(p)
that's a lot of sh*t to have going on at once. hope the promoters are making killing...
0.75
0.97
A
even craziest thing that's happened to teumn year...
0.63
0.39
B
I'm not sure it's good idea or not but definitely unique one...
0.19
0.29
C
C
good one. hope show will huge success...
0.35
0.09
D
D
D
response y;(x)
C (x)
(x)
C (x)
2 0.4
4 0.65
A = 0.8
Figure Illustration of the process to generate C(x), (x)), and (x)).. We sample N responses Vt, i 1,..., N from L.M ply zz). Each response is associated with machine disutility score T' (x))), and human-rated disutility score (z)) To construct (2)). we keep the responses such that the machine toxicity score satisfies rm (x)( & X, for each 2g E A. Finally, we compute the induced score (2) by taking the maximum human disutility score of each Cs, (a).
measured by another metric. To summarize, we reduce the task from retraining a calibrated L.M j(z) with high- dimensional model parameters to searching for a univariate parameter X.
studied in the context of conformal risk control (Bates et al., 2021; Angelopoulos et al., 2021).
3.2. L-statistics
L-statistics refers to the class of estimators expressed as linear combinations of order statistics originating from (Mosteller, 1946). Notable examples include the sam- ple quantile, the trimmed mean, and the winsorized mean (Tukey, 1962). Fixing À E A, let FA,(1) S TA.(2) S ... (x,(n) denote the ordered statistics of (rx (Z,,)) Furthermore, let Fx and FA,A be the true and empirical distributions of (rx(z))...., (*,n)) respectively. For given distortion risk measure, the plug-in estimator that re- places the true distribution by the empirical distribution Fn,
3. Distortion risk control via L-statisties
3.1. Theoretical setting
Suppose we sample prompts 2i....,In i..dd from distribution. For each prompt, we generate the candidate set C(x;) as described in Section 2.3, then recruit human raters to score responses in the set. Following the pro- cedure Section 2.3, we can obtain dataset D that in- cludes (Ir,, (r:():) À A}) for i 1,..., n. We assume that the machine disutility score model is pretrained and independent of our dataset D. Then the data points (z,, (x:)) À A}) remain i.i.d. Throughout the rest of the section we denote generic draw from the prompt distribution by x.
Ras(A) = Rey (Fx.)))))= (F.X()))())))
(2)
By definition, F = TA(()) ()) for any p E 1-1
; thus
For each A, use the shorthand notation (A) for (Frx())) where Rg is the distortion risk measure defined in (1) with user-chosen weight measure vb. Our goal is to learn Ò from D such that
it can be written L-statistic
(A)
Σ
16
TA.())-
n
PP (P (Å) S 21
Theorem 22.3 of van der Vaart (1998) describes the asymp- totic normality of L-statistics. Given this result, we show that Ry (A) is an asymptotically normal estimator of R for any fixed E A and we can find a consistent variance es- timator. The proof of Theorem 3.1 is presented in Appendix C.
for some pre-specified (a. 8). Above, Po accounts for the randomness in D. The parameter represents the target risk level and 1 - 5 corresponds to the confidence level. This formulation aligns with the standard objective probably approximately correct (PAC) learning and has also been

---

Conformal Tail Risk Control for Large Language Model Alignment
Theorem 3.1. Assume (x) E (a, b] almost surely for some
VW
UCB R,t
a < xc, Fx is continuous and strictly increas- ing. Further, assume that <b(y) Se x (z)dz for some V that is bounded and continuous at Fx(r) for Lebesgue almost-every r. Then,
True Rick R,(0)
Rakk (89
Desired Rack
- 4 1).
0(2)
where
62(A)= / / W(LLA (xxx()) di, with
Figure 3. Illustration of A. We choose A as last À such that the (asymptotic) upper confidence bound R (A) falls below a.
Pn,x being the empirical distribution of rx(a1)..... ,. (x(an) and
where
215 21- O(A),
Dx(r, (r, AP) Pnx(r) Equivalently,
and 21-8 is the (1-6)-th quantile of the standard normal dis- tribution, as illustrated in Fig. 3. In practice, we discretize A to avoid checking the condition R (A) S a for infinitely many points. Our procedure is outlined in Algorithm and illustrated in Fig. 2.
0²(
2
j n
i IAJ
ij n²
(3)
n2a
72
By Theorem 3.1, (A(( is asymptotic (1 8) UCB. AI- though Bates et al. (2021) focus on traditional risk measures that are expressed as expected loss, we can easily extend their result (Theorem 6) to distortion risk measures.
We assume the boundedness of rx(;;) for simplicity. It is reasonable assumption for ratings which are often designed to be bounded. It can be relaxed with more involved proof techniques (Gardiner & Kumar Sen, 1979; Stigler, 1974).
Theorem 3.3. Let Å be defined in (4). Assume Rag (A) is continuous and strictly increasing. In the same setting as Theorem 3.1
Among all distortion risk measures, CVaR, is the most interpretable and widely-used metric. For CVaR3, we can find a much simpler variance estimator.
lim inf IPD (Rog aa)
25
(5)
Corollary 3.2. For CVaRg with vp()) max(p 3,0)/(1- 3). Theorem 3.1 holds with
As consequence. for any selection mechanism that picks j(x) from (z)
lim PD (Rys (Fr(t)))) S a) 2 - 8. R=+00
(A = B)2
Var
Remarkably, this result does not involve any assumption on the underlying LLM p(y x)) or machine disutility score model I'm (z). In particular, allows the machine ratings to be arbitrarily misaligned with human ratings.
where Var denotes the sample variance.
Another important example is VaR,s, While the VaR is not distortion risk measure with a differentiable sh, we develop parallel theory in Appendix D based the asymptotic the- ory of empirical quantiles. Unlike CVaR3, the asymptotic variance depends on the density F of r; and hence harder to estimate. We apply the bootstrap technique (Efron & Tibshirani, 1994) instead to estimate d² (X)..
3.4. Alternative (conservative) approaches for distortion risk control
Another strategy to construct pointwise UCBs for Ra) is replace all quantiles by their confidence envelopes. Specifically, if we have statistics { (A) p [0, 1]} such that
3.3. Conformal distortion risk control via L-statistics
(A), P(gp(A) S O (A), Vp E [0, 1)) 2 - -,
(6)
By design, R& (A) is monotonic in X. This allows us to apply the method of Bates et al. (2021) to achieve risk control by inverting pointwise upper confidence bound (UCB). Specifically, we choose
then fo o, (p) is valid (1 ()- UCB for (A). Following Snell et al. (2022), we consider two confidence envelopes based on the Dvoretzky- Kiefer- Wolfowitz (DKW) inequality and Berk-Jones (BJ) statistics, respectively. For simplicity, we assume Fx continuous.
u max (A E A: RT(A) Sa, AA S S)),
(4)
5

---

Conformal Tail Risk Control for Large Language Model Alignment
Algorithm I Conformal distortion risk control
tinuous. It can be computed in polynomial time (Moscovich & Nadler, 2017). Let 8; -G (88). Then
Input: machine- machine-scoring model r'.() (-), discrete subset of its range A, humann-nnotated scores r(-), set prompts
X' (ax))=== target level o, tolerance level 6, weighting
P(g., (A) - TX, Vi - n
-
function pb.
This yields an upper confidence envelope 9p (A) (),()) for any p (8) For p > Sn,
function DRC(X A, (,,
4p (A) Amax-
for v; E X' do
+ CANDIDATESETT(E) {See Algorithm 2}
While both DKW and BJ approaches yield sinite-sample valid UCBs for (A), they are conservative because they do not target the specific choice of sh. In fact, (6) implies
for À A do
Cx(x;) =
for yk E C(1;) do
that Jo at (A)dk((p) is uniform UCB across all weight functions, i...,
if rmk < X then
(x,), + Cx(x;) U {u
end if
P Ryp (2) S (. O5 (A)dv()) VA 2 1 - 5.
(x;) + max(r(u)) Us Cx(x))
end for end for
Therefore, the actual coverage (i.e., probability that Rg (A) is less than or equal to the above UCB) is typically much higher than - 5. By contrast, the UCB given by the L- statistic tailored to (A) and the coverage converges to ô as n OC.
end for
for E do
(rx,11)----.
(a)(a)) ;
(A) + Σi= {vb (1) - (=)) TA.69
(A))) + Eq. (3)
(A) + (A) 2.--8. VR
a(
(UCB)
3.5. Deployment of the calibrated model
end for
Recall that any choice of 5(x) E Cx(x)) controls the risk (Fs())))) For new prompt æ, the most cost-effective approach is to sample candidate responses yı: 32, from the underlying LLM ply xx) until the first time the machine disutility score is below X. Suppose each sample incurs unit of computational cost. Given prompt xr the sampling cost follows geometric distribution with rate PP(m (v) < À ar, λ). Therefore, on average, for given prompt x expect the number of samples needed to generate a response with disutility score less than Ả is
À + max
( A: (X)
return Å
DKW inequality. any Ả E A and
The DKW inequality implies that, for
P sup PER
(r Fx(F) (2) < 22--2mm -2ne
2
Letting 6n.6 \log(//))/21 and r = (A) for pe e [0,
we obtain that, with probability at least d.
N₂ P(rm(v) d x)
(A) - pl S 6n.d, Vp E [0,1] 1].
This demonstrates that the number of responses generated by our method, CDRC-L, is adaptive to the given prompt x, unlike best-of-N. In other words, the (unconditional) expected cost is
This implies an upper confidence envelope as at (A) (R)( where kn.p = [n(p kn,s)]. Here, if kn.p n, we set (k..p...)) Amax, the upper bound of A.
Berk-Jones Statistics. As pointed out by Snell et al. (2022) and Bates et al. (2023), the DKW inequality is overly conservative for p close and 1. The BJ statistic uses the fact that (A) Beta(i, n + 1), where Beta denotes Beta-distribution. Let Bin-i++1 denote the cumu- lative distribution function of Beta(i, i + 1). The BJ statistic is then defined as
Cost(A) EF,r
P(rm(U) < x xc, 2)
Suppose we have hold-out of prompts DD' that is in- dependent of the dataset D defined in Section 3.1, Cost (N) can be estimated by
1 D
1
Mt max (F
P( (v) < N x,
15i5a
ZED'
Let 8y be the 5-th quantile of M.. Clearly, 8s does not de- pend on F, because (r(x))) ~ Un, 1)) when F, is con-
where the probability is estimated by the Monte-Carlo method.
6

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized CVaRs, B 0.5
Realized CVaRa, B = 0.75
Realized CVaRg, / = 0.9
0.35
CDRC-8]
0.35
CDRC-8)
CDRG-8)
CDRC-DKW
CDRC-ORW
CDRC-DKW
0.30
CDRC-L
0.30
CDRC-L
CDRCL
a
0.25
CVaR
CVaRa
0.25
CVaR
CVaRg
0.2
CVaR
0.20
0.15
0.1
0.15
0.10
0.10
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
a
0
a
Average Sampling Cost, 0.5
Average Sampling Cost,
0.75
Average Sampling Cost, =
12
CDRC-0)
CDRC-0)
CDRC-U)
CDRC-DKW
7-
CDRC-DKW
CORC-DIW
4-
CDRC-L
CDRC-L
10
CDRC-L
cost
cost
cost
5
3-
4-
6-
3
0.15
0.2
0.25
0.35
0.15
0.2
0.25
0.35
0.15
0.25
0.35
Figure 4. LLLMMA-7B: Realized CVaR, vs. (row and average sampling vs. (row 2) for Spearman correlation between human and machine toxicity scores at p 0.57 evaluated on held-out dataset with [D] = 6000. The confidence band is computed by taking the mean estimate plus/minus one standard error estimated from the results across independent experiments. Each subplot the respective rows illustrates different setting of 3 E {0.5, 0.75, 0.9}. From the panels in the first row, we observe that our method, CDRC-L (orange), an improvement to CDRC-DKW (green) and CDRC-BJ (blue). as it able to achieve risk control (shown in the black dotted line) while being less conservative than both baseline methods. Evident from the panels in the second row, our method is more cost efficient and least. conservative in generating risk-controlled LLM response than DKW or BJ.
4. Experiments
Toxicity scores. To apply our method, we need a human toxicity score function r(-) and machine toxicity score function r.)(.). Human-annotated data can be costly and time-consuming acquire. To evaluate our method, we create cheap semi-synthetic benchmark using an existing machine scoring model the "human annotator," and biased model as the "machine assessor." Specifically, we use the Detoxify model (Hanu & Unitary team, 2020) for r(.) and retrain the Detoxify model for 7.m(.) on a biased subset of the Jigsaw Unintended Bias Toxicity Classi- fication dataset (cjadams et al., 2019) that consists of the e% most and least toxic instances. The goal is design I'm (-) with varying degrees of misalignment from r(-). This allows us study the effect of misalignment on Á and hence the cost of tail risk control. In particular, quantify the misalig mnisalggmmntt between human and machine scoring mod- els by the Spearman correlation coefficient p between the scores across all candidate responses. In our experiment, we train three models for Tm (.) with c% E (15% 30%, 70%}. The Spearman correlation coefficients are 0.57, 0.68, 0.78, respectively. More details about these models can be found in Appendix A.2.1.
In this section, we perform experiments to investigate the issue of conformal distortion risk control method to mitigate toxicity of --geerrrrteed reseponses. This is critical application, as toxic outputs may cause severely negative impacts on impressionable populations, moreover, propagate across wide audiences, leading to misinformation and harm.
misaligmment by implementing our
4.1. Experimental setup
Datasets and models. We randomly draw 10K prompts from the REALTOXICITY
dataset (Gehman al., generate 40 re-
2020) For each selected prompt sponses y (I;) using the LLaMA2-7B model (Touvron et al., 2023). Given the initial responses, we apply the sequential algorithm described in Algorithm 2 to construct the candidate response sets C(a;),, ensuring the quality of the selected responses. Specifically, we use perplexity (PPL) evaluate response quality, ROUGE-L assess similarity between responses, and restrict the maximum set size to 32 as stopping criterion. More details can be found in Appendix A.

---

Conformal Tail Risk Control for Large Language Model Alignment
01
0.15
02
0.20
0, 0.25
04 0.30
0s 0.35
Cost, B
0.5
Cost, B 0.75
vs. Cost, f 0.9
4.0
3.5
5
3.0
4
2.5
3-
2.0
2
0.57
0.68
0.78
0.57
0.68
0.78
0.57
0.68 0.78
p
p
Figure 5. LLaMA2-7B: Spearman correlation average sampling cost for CVaR, with [D] 6000.
Choices of parameters. We consider both CVaRg and VaR, control with E {0.5, 0.75, We fix the fidence parameter - 5 0.95. To determine rea- sonable target level we compute the empirical CVaR, on human scores of all candidate responses with q E (1%, 5%, 10%, 15%, 20%}: see Appendix A.3. This sug- gests range of reasonable target levels. In particular, we consider a E (0.15, 0.2, 0.25, 0.3, 0.35).
cussed in Section 3.4. As result, incurs the smallest deployment cost among all three methods. Moreover, as 3 increases, the advantage of CDRC-L is more prominent. Although CDRC-BJ improves upon CDRC-DKW due to the tighter bounds for extreme quantiles, it still underperforms CRDC-L. On the other hand, the gap between the target and realized CVaRR, stays nearly constant for CDRC-L across different values B, suggesting that L-statistics are adap- tive to different choices of th. Results for p 0.68, and p 0.78 presented in Appendix B.1, showing similar patterns.
Evaluation. We randomly split the prompts, using [D] E (50, 100, 200, 1000, 6000} to determine the optimal thresh- old Å and the remaining method, after selecting X, we deploy the calibrated model on the held-out dataset. We then evaluate the realized CVaR, and VaR,s 3 of human scores and estimate the sampling cost following (7). We apply all three versions of conformal dis- tortion risk control based on L-statistics, DKW inequality, and BJ statistics. We refer to them as CDRC-L, CDRC- DKW, and CDRC-BJ, respectively, where CDRC stands for conformal distortion risk control. For CDRC-L and CDRC- DKW, we repeat for 15 times, and for CDRC-BJ, we repeat for 3 times due to computational complexity to compute 86-
held-out test dataset. For each
We present the results for VaRg control with E {0.57. 0.68, 0.78) in Appendix D. The comparison between CDRC-L, CDRC-DKW, and CDRC-BJ are qualitatively similar to CVaR, control. Notably, the realized VaR, of CDRC-L is extremely close to the target level with low sam- pling cost, demonstrating that it is not conservative while remaining the most cost effect among all methods.
Cost and misalignment. Next, examine how the de- ployment cost varies with the misalignment between human and machine ratings. Fig. 5 demonstrates that, for all set- tings of 3, as the Spearman correlation coefficient increases, the cost of generating a conttolle LLM response drops. This confirms our intuition that better-aligned ma- chine ratings reduces the cost of calibration. Similar trends are observed for VaR, control, shown in Appendix D.
Additional models. We conduct additional experiments using the LLaMA3.2-3B, and LLaMA3.1-88 models. Fol- lowing the same implementation details described, we obtain results that are highlighted in Appendix E.
4.2. Results
Performance in small samples. Human annotations are the primary contributors the calibration cost in our frame- work. Nevertheless, we demonstrate that our method retains statistical efficiency even with smaller calibration sets, such as, |D] (50, 100, 200, 1000}
Realized risk and average cost analysis. Fig. 4 shows the realized C, of human scores and the average sam- pling cost with 0.57 on the held-out dataset as functions of a for E (0.5, 0.75, 0.9] with (D] = 6000. The panels in the first row shows that all methods control the risk at the target level and CDRC-L is least conservative, as dis-
From Fig. 6, it is evident that CDRC-L is consistently the least conservative among all methods, while still maintain-
8

---

Conformal Tail Risk Control for Large Language Model Alignment
Best-of-N
Mean C + standard deviation
Realized CVaRs vs. Calibration Set Size, P= 0.5, 0.25
0.25
Best-of-3
0.2427 + 0.0068
Best-of-5
0.2015 + 0.007
Table 1. Beta CVaR, Human Scores with standard deviation trials.
CVaR,
0.15
0.10
CDRC-8)
The results in Table I demonstrate that our method achieves comparable C control with lower average inference cost than best-of-N (see Fig. 4). partieular, on average, for 8 0.5, best-of-3 controls CVaR,, approximately level a 0.25, and CDRC-L can achieve the same level of control by generating less than LLM responses. Further- more, average, for 8 0.5, best-of-5 controls CVaR,, at approximately level a 0.20, while CDRC-L can achieve the same level of control by generating less than 3.5 re- sponses.
0.05
CDRC-DKW
CDRC-L
Target CVaR,
200
400
600 Calibration Set Sibe
800
1000
Figure 6. LLaMA2-7B: Realized CVaR, vs. calibration set size for 3 = 0.5, = 0.25, p = 0.57. The confidence bands constructed by taking the mean estimate + standard error estimated from the results across 15 independent experiments.
5. Discussion
Average Sampling Cost Calibration Set Size. f 0.5, 0.25
CDAC-8j
We close the paper by discussing few potential generaliza- tions and extensions of our method.
CDRC-DKW CDRC-I
Average Samp ng Cost.
8
Theorem 3.3 only works for the population of prompts that has the same distribution as the data the method operates on. In the presence of distribution shifts, we can potentially adapt the reweighting technique for traditional expected risk measures (Tibshiraní al., 2019; Lei & Candès, 2021; Candès et al., 2023) correct for bias for L-statistics.
7-
5
4
200
Due to the rise of RLHF, preference data has become in- creasingly prevalent. While the above method only works with direct human ratings, there is a potential of adapta- tion to handle preference data, which exhibits U-statistic structure.
Calibration
Figure 7. LLAMMA2-7B: Average sampling cost for CVaR, con- trol calibration set size for 3 0.5, 0.25, P 0.57. The confidence bands are constructed by taking the mean estimate I standard error estimated from the results across 15 indepen- dent experiments.
ing the guaranteed C control. Furthermore, from Fig. 7, observe that CDRC-L is consistently the most cost effective among all methods. In particular, it is significantly less expensive in comparison to other methods in settings with small calibration set sizes. We show qualitatively equiv- alent results for VaR, in Appendix D.
Comparison to best-of-N. We compared the two meth- ods under controlled setup: setting 3 = 0.5, using N E 3,5 5 for best-of-N, and calibration set size n 1000. Following the procedure in 4.1, we evaluated the realized C of buman-annotatee toxicity scores using the re- sponse with the lowest machine score from each group of N samples:

---

Conformal Tail Risk Control for Large Language Model Alignment
Impact Statement
toxicity classification, 2019. URL hwwph:ht////kttc /kaglp, com honm//commetitiios///i/iiee htw.:/tiioe..iiissseeumnneeennee bias in www.iiiityyclassffccca...i..
This paper presents work whose goal to advance the field of machine learning by ensuring safe deployment and align- ment large language models (LLMs).. There are many potential societal consequences of our work, for instance, addressing the potential misalignment between human and machines, recalibrating machines such that they are better aligned with humans, and controlling the tail risks asso- ciated with language models. this project, we did not use directly use data from humans. There are not negative impacts which feel must be specifically highlighted here.
Csörgö, M. and Révész, P. Strong approximations of the quantile process. The Annals of Statistics, pp. 882-894, 1978.
Efron, B. and Tibshirani, J. An introduction the boot- strap. Chapman and Hall/CRC, 1994.
Gardiner, C. and Kumar Sen, Asymptotic normality of variance estimator of linear combination of function of order statisties. Zeitschrift für Wahhschheinlcchketsstteene und verwandte Gebiete, 50(2):205 221 1979.
References
Angelopoulos, A. N., Bates, S., Candès, J., Jordan, M. I., and Lei, L. Learn then test: Calibrating pre- dictive algorithms achieve risk control. CoRR, a0.02110.01152, 2021. BRL.httss:/axxxtv.o abs/2110. 01052.
Gehman, S., Gururangan, S., Sap, M., Choi, Y., and
Smith, N. A. Realtoxicitypromps:: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 1162,, 2020.
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. Conformal risk control, 2023. URL https: //arxiv.org..oooe..........o.o.p abs/ 2208.. 02814.
Hanu, L. and Unitary team. Detoxify. https//giiuub.commmmiiraaaaaoooe 2020.
Github.
Hoeffding, W. Masstabinvariante korrelationstheorie
Balbás, A., Garrido, J., and Mayoral, S. Properties of dis- tortion risk measures. Methodology and Computing in Applied Probability, 11(3):35--399 2009.
Schriften des Muthematischen Instituts und Instituts fur Angewandte Mathematik der Universitat Berlin, 5:181- 233, 1940.
Bates, S., Angelopoulos, A., Lei, L., Malik, J., and Jordan, M. I. Distribution-free, risk-controlling prediction sets, 2021.
Kiefer, J. Deviations between the sample quantile process and the sample df. Non-parametric Techniques in Statis- tical Inference, pp. 299-319, 1970.
Bates, S., Candès, E., Lei, L., Romano, Y., and Sesia, M. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149- -49-178. 2023.
Lei, L. and Candès, E. J. Conformal inference of counter- factuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, 83(5):91 1199)91119388 1993, 2021.
Candès, E., Lei, L., and Ren, Z. Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85()):24-45, 2023.
Mohri, C. and Hashimoto, T. Language models with conformal factuality guarantees, 2024. URL https:
Casper, S., Zhang, S., Holtzman, A., Kreutzer, J., Zhu, C., Ranzato, M., and Stiennon, N. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023.
/ //arxiv. wwg. abs 220..
Moscovich, A. and Nadler, B. Fast calculation boundary crossing probabilities for poisson processes. Statistics Probability Letters, 123:177-182, 2017.
Cherian, J. J., Gibbs, I., and Candès, E. Large language model validity via enhanced conformal prediction meth- ods, 2024. URL https: arxiv. org/ abs / 2406. 09714.
Mosteller, On some useful" inefficient" statistics. Annals of Mathematical Statisites, 17:377-400 1946.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. arXiv preprint arXiv: (10.:7770.334.1 2017.
Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. Conformal language modeling, 2024. URL https: httt:s::/aaxxiv.rrtt 2306. 10193..
cjadams, andinversion. D. B., Sorensen, J., Dixon, L., Vasserman, L., and nithum. Jigsaw unintended bias in
Rockafellar, R. T. and Uryasev, S. Optimization condi- tional value-at-risk, 2000.
10

---

Conformal Tail Risk Control for Large Language Model Alignment
Snell, C., Zollo, T. P., Deng, Z., Pitassi, T., and Zemel, R. Quantile risk control: A flexible framework for bound- ing the probability of high-loss predictions, 2022. URL https: /tappii//xxxv...//////ae//
arxiv. org/ abs 2212 13629.
Stigler, S. M. Linear functions of order statistics with smooth weight functions. The Annals of Statistics, pp- 676-693, 1974.
Tibshirani, R. J., Foygel Barber, R., Candes, E., and Ramdas, A. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 002228230..09288, 2023.
Tukey, The future of data analysis. In Breakthroughs in Statistics: Methodology and Distribution, pp. 408-452 Springer, 1962.
van der Vaart, A. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: (50iv:990.008800 08593, 2019.
Zollo, T. P., Morrill, T., Deng, Z., Snell, J. C., Pitassi, T., and Zemel, R. Prompt risk control: A rigorous framework for responsible deployment large language models, 2023.

---

Conformal Tail Risk Control for Large Language Model Alignment
Supplementary Materials
the supplementary materials, we provide implementation details, additional experimental results, and formalized proofs. Code repository can be found https: //gituub. com/ www.//eeeeaggeee//RRgg
Notation
Table 2. Summary of notation used in the paper.
Symbol
Meaning
X
User prompt space.
y
LLM output space.
User prompt drawn from X.
(xx)
LLM response given user prompt X. Calibrated L.M response given a user prompt E X.
((x)
p( 1
Data generating process of LLM.
((x)
Set of uncalibrated responses generated by an LLM, where [C(x)] = N for given prompt æ. Set of responses y generated by L.M satisfying I' (y) < for given prompt x.
(x)
r(y)
Human-generater disutility score of response y(z).
Ti (v)
bechine-generated disutility score of response yar)..
Tx(z)
Disutility score for (x)x) the worst human rating, i.e. (x maxyec,(s) r(y). Cumulative distribution function (CDF) of r(((z) over
FrG())
R(Fr()()))
Summary measure of Fe(g(=)) where R(-)) is a functional that maps any distribution to non-negative number. Denote as R(F) as a shorthand.
Range of machine scores with A [Amin: Amax]
D
Dataset of (x,, (rx( X E A}) for n i.idd prompts 21...., Fn. Distortion risk measure, where (-() is a weighting measure such that xp)) 2 0, and So dv(p) = 1. Denote as (A (A) as a shorthand.
Ry (FF()))))
Fx
True distribution of (rx(x1), r> (Ia)).
FA.
Empirical distribution of (rx(z)).... Tx (2x)).
(A)
Empirical distortion risk measure of (rx (x1), (xn)) with
R+ R = F.:(())
Prespecified risk control level. Prespecified confidence level.
8
A. Implementation Details
A.1. Algorithm to generate the candidate set C
Following the original implementation of LLAMA-2--7 --MMA----B----E model, we set the generation temperature at 0.8 and the top-p parameter at 0.95. We generate candidate response for our DRC framework using Algorithm following Quach et al. (2024). To ensure quality and diversity of generated candidates, we filter responses by retaining only those with Perpexity less than 2.61 while ensuring that the ROUGE-L scores between samples in the candidate set is not greater than 0.26. Table 3 presents the percentiles of these metrics across all generated responses. Fig. 8 outlines the cardinality of the sets generated by various combinations of
Table Percentiles of different hyperparameters.
Percentile
50
75
80
85
90
95
ROUGE-L
0.26
0.39
0.43
0.47
0.52
0.59
Perplexity
2.61
3.07
3.18
3.31
3.48
3.76
12

---

Conformal Tail Risk Control for Large Language Model Alignment
Algorithm 2 Generation of the candidate set C(z)
Input: input prompt x, set-based confidence function F, text similarity function S, sample quality estimator Q, fixed threshold configuration 7 = (2:72: 12:13). sampling budget kmax: with conditional output Po(y | x)) from a generative model.
function CANDIDATESHT( F, kmax)
C= 0
for k 1 to kmaz do
W/ y py u) (Sample from generative model]
if @zz, y1)) < 71 and ( 35)
My E EC} > then
CCC { (Quality and similarity control)
end if
if F(C)= 2 73 then FreakkSet-baaae confidence guarantee) end if
end for return C
1..3.070.25.
13.18.8 0.26
13.70.
3000
1500
3000
Pasuury
1000
Preqpeecyy
1.500
Ereepancyy
Trequeeeyy
1000
000
800
500
1000
3000
1
1
< 00:0 19
/
1
1
Size
Size
Sze
S2e
Size
Figure 8. Size of generated sets under different combinations of hyperparameters. (m, 72, 73) refers to (preplexity, similarity, stopping threshold), respectively.
A.2. Detoxify model for r(-) and (-)
We use the original Detoxify model as proxy for human annotator scores, then finetune this base model with various sample sizes, learning rate of 0.0001, a batch size of 16, and a weight decay of 3 x 10-6 on single Nvidia A40 GPU, The Adam optimizer was employed with a = 0.9, 3 = 0.999, and = 1 We exclusively use the Detoxify framework to evaluate the text generated by our model without the prompts.
A. SEMI-SYNTHETIC EXERCISE RESULTS
To study the effect of human-machine misalignment, we create a semi-synthetic data set of human and machine geeerrtee toxicity scores. We use the Detoxify (Hanu & Unitary team, 2020) model our "human" annotator, and Detoxify model finetuned on biased subsample of toxic instances our "machine" annotator. For details about implementation, Section 4. Table 4 outlines ROC-AUC (Area Under ROC Curve) comparison between the original Detoxify model, and the finetuned model, which illustrates that the original Detoxify model outperforms the finetuned model in predicting toxicity. Fig. 9 illustrates the distribution of human and machine-assignel scores for responses generated by the LLaMA2-7B model across all sampled prompts. Both human and machine toxicity scores exhibit a long-tailed distribution in the worl-world dataset. Due to the Detoxify model being fine-tuned with an emphasis on "severe toxicity" samples, its scores are predominantly concentrated in the lower range, highlighting misalignment between human and machine assessments.
Table 4. Performance comparison of detoxify models.
Model
ROC-AUC
Detoxify (original)
0.97
Finetuned Detoxify (p 0.57)
0.86
13

---

Conformal Tail Risk Control for Large Language Model Alignment
Human Scores
Machine Scores
Donsity
Density
15
10
0.4
0.6 Tocicity Score
0.8
Right
5
0.0
0.2
0.4
0.6
0.8
1.0
Toxicity Score
Figure Distribution of toxicity scores assigned by humans (i.e., original Detoxify model) and machine (i.c., fine-tuned Detoxify model) for responses generated by the LLaMA2-7B model across all prompts. The main plot shows the density of scores, highlighting long-tailed distribution both human and machine evaluations. The zoomed-in plot illustrates that human scores have heavier tail than machine scores.
A.3. Choices of a
To select an appropriate value of a, we arrange (x) in increasing order then rule out the top 4% responses, and compute the desired distortion risk of the lower (1 1). The calculated value is used as the target a. For details about implementation, Section 4.1. Table 5 outlines the rational values under different settings of À for different 4% given that interested in studying CVaR.
Table 5, Choice of a with different 9% under various settings of y for CVaR.
= (11, 72, 79)
= 1%
= 5%
10%
q 15%
20%
(3.07, 0.26, 32)
0.815
0.580
0.356
0.217
0.145
(3.18, 0.26, 32)
0.809
0.578
0.356
0.216
0.140
(3.31, 0.26, 32)
0.815
0.574
0.354
0,219
0.142
(3.48, 0.26, 32)
0.807
0.578
0.359
0.217
0.141
(3.76, 0.26, 32)
0.815
0.589
0,352
0.221
0.141
B. Additional experimental results for CVaR, control
B.I. Impact of Sluman-Machine Misalignment
We evaluate the performance, i.c., average sampling cost, and realized CVaR,3 a held-out dataset, of our framework CDRC-L, against baseline methods CDRC-DKW and CDRC-BJ. Fig. 4, Fig. 10, and Fig. 11 illustrate the setting when the Spearman correlation (p) between the human and machine scores is 0.57, 0.68, and 0.78, respectively. From these figures, it evident that all methods effectively control the risk at the specified level a, given that they all fall below the black dotted line (where a = CV Our method, CDRC-L, is consistently the least conservative all methods across different values 3, and p. Conversely, baseline methods appear to grow increasingly conservative increases. As consequence, we see that our method is consistently the least costly across all settings of and p.
C. Proofs
C.1. Proof of Theorem C.1
We decompose the proof into two parts.
Theorem C.1 (Asymptotic normality of L-statistics). Let fx, Fx denote the PDF and CDF of the score T& respectively, and
14

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized CVaRs, B 0.5
Realized CVaRa, B = 0.75
Realized CVaRg. / = 0.9
0.35
CDRC-8J
0.35
CDRC-8)
CDRC-8)
0.30
CDRC-DKW
CDRC-L
0.30
CDRC-ORW
CDRC-L
CDRC-DKW
CDRCA
C
0.25
CVaR
CVaRa
0.25
CVaR
CVaRg
CVaR
0.2
0.20
0.20
0.15
0.1
0.10
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
a
a
a
Average Sampling Cost, 0.5
Average Sampling Cost,
0.75
Average Sampling Cost,
CDRC-0)
CDRC-0) CORC-DKW CDRC-L
CDRC-U
3.5
CDRC-DKW CDRC-L
CORC-DIW
8
CDRC-L
3.0
7-
cost
cost
4
6-
5
2.0
3
4-
0.15
0.25
0.35
0.15
0.25
0.35
0.15
0.25
0.35
Figure 10. Realized CVaR, and average sampling cost held-out dataset for p 0.68 with [D] 6000.
Realized CVaRg, B 0.5
Realized CVaRg, B 0.75
Realized CVaRg. B 0.9
0.35
CDRC B]
0.35
CDRC BJ
CDRC BJ
JDW
0.30
0.3
CDRC-L
CDRCIL
CDRC.L
C
CVaR
CVaRs
0.25
0.2
0.20
0.20
0.15
0.1
0.15
0.10
0.25
0.35
0.25
0.35
0.15
0.25
0.35
0
0
Average Sampling Cost, $ = 0.5
Average Sampling Cost, A = 0.75
Average Sampling Cost, f = 0.9
3.00
CDRC-9)
4.0
CDRC-8)
CDAC-8) CDRC-DiW CDRC-L
CDRC-DIW CDRC-L
CDRC-DRW CDRC-L
2.75
3.5
2.50
cost
5.
2.25
3.0
1.75
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
a
Figure 11. Realized CVaR,, and average sampling cost on held-out dataset for p == 0.78 with [D] 6000.
FaL denote the empirical CDF of the scores Ti,..., Tn- Further, let
Ra((A) - J. (D)d(()) and R=())= f
for any measure 1a that may not have a density (e.g., can be point mass at po E (0, 1)). Suppose
15

---

Conformal Tail Risk Control for Large Language Model Alignment
1. r; [a, bJ for some -0 < co almost surely, with infrela,0 ((() > 0, and seprela.0) f((r) xc.
2. J dv(l)
falPs (()) 00.
then (n(Be(()) (a() 4 (A))) where
p())= --- J.
pap p.p' pl ((Fx'())XXC (v)) (()) d(())d((p)) oc.
Proof. By the Bahadur-Kiefer representation (Kiefer, 1970) (see also Theorem of (Csörgö Révész, 1978)) of sample quantiles we can write
A A sup ((FT,))) - FX (P))) (b)) - (F (p)) - p)]
pEju,1]
(ogg())//(10g (/ogg log m)
(log(n)
= Op
= =0p
n2/4
32/4
equivalently,
(PT3(0) (
An
fA (p))
(A (p))
Given this, we have that
(A) - (&(A) = / (0) (p))d((p))
1'' FRA(FF (p)) dvb(p) + 6n
1a(FF(())
where
15 S An S:
1
(log(n)
(w)(p) =O,
fx(Fx (p))
123344
holds by assumption 2.
Thus we have that
1rr S SF (w)) db(p)) + (1)
VT(Ro(A)) - (A)) = Vn. Σ
(p) (p))
1=1
A 1 Σ
S(r:) (1,
By assumption 2,
S(r.) /.'' 1( S FT P de)(p) S/' (x(F)(()) dw(() 60.
1
fA(FT (p))
It remains to prove [[)(r:)] = 0 and Var(S(r:)) = (ri) < X0.
To study the mean, note that by assumption 1, (p)
(p) is uniquely defined and P(r; & (p)) (p)) = =p. Thus,
E [ S (p) del(p) =
[[S(rr)) =
SA (x))
(p))
16

---

Conformal Tail Risk Control for Large Language Model Alignment
To study the variance, observe that the second moment can be expressed follows
(r)) =3 (ra S F (p) de(p)
fx(Fx (p))
[1, S PI (p)} - dv(p))
1r,, FX Fx) (b60o)
= E
[x(Fx((0))
E (1 S Fx (p)) -p) F1 -P)) (((D))((())
fx(Fx (D)
Let B(p) 1r, - Fx (p)). Notice that B[B(p)] = thus
E (BBp) p) (B((') - f/) - (B(BB())B(()))
= =B(B(p) B(y')) - -p-P
p 1 p p
if sp
(p' p.P
pf
= P -ppp - p.p
Therefore,
Var(S(r:)) = S.' fo' PAP---1 db(())((()) (A
By assumption 2,
" db(p)d((p)) = 1/0 (A(FF(())) dwl(p)
1 Var(S(r:)) & (A
1
<0..
Theorem C.2 (Consistent variance estimate for L-Staaisticss). In the same setting Theorem C.I, assume the assumption / holds and that (yy) So vi (z)dz for some s that is bounded and continuous Fx(r) for Lebesgue almost-every r.
Let
(A) 1 [ V
Pa, di.
If o is bounded, then 2% (A) an (2(A) as 00.
Proof.
We first verify assumption 2. Take p Fx(x).
So FA(F (p)) dap(p) =
(p(
Fx(FT (p)) dp
(FFFA(E)) (F) [x(x)
(Fx((x))
The integral is bounded since v/ is bounded and a, b are both finite. Let F2(r), and Fx(F) in the double integral of d²)),, then
0²(2) Jo [. PAp p.p
FA(FT (P))f(FFFF (V)) d0(o)do(p))
b FX(rAF) - FX(
('(AAG))((E()))) [(())x(T) dr dF.
17

---

Conformal Tail Risk Control for Large Language Model Alignment
By the Glivenko-Centelli Theorem, (r) a Ex(r) for every (a, b). thus, for any r, [a, Dj. (r - Fr), (F) "As - -
(F))
Furthermore, since v is continuous at F(r) for almost every under the Lebesgue measure, by the continuous mapping theorem,
(F (()) " (F((()))
for almost every r under Lebesque measure. Suppose that vh is bounded above by B, then
(F) - Fr) x()) x B²,
thus
(A) - (²())
by the Dominated Convergence Theorem.
C.2. Proof of Corollary 3.2
For CVaRa, (p)) max(p 3,,0/(1 B) and hence &f 1(p 8)/(1 B). By (C.2),
1
b [ (AA 2 ()I(F((F) 2 3)drdF
o² (²())
(1 - 3)2
1
(BA(AAF) - -A()))))d
(1 3)2
FI (B) (T
Let Gy be the CDF of r _ max B}. Then
= FX-
By Hoeffding's covariance identity (Hoeffding, 1940),
Var(r!) ((ar))=/((((((a -CGGAAA))CC)CCC())ddd
[ S So (FA(AA)) (F)
Thus,
a²()) (1 (1- 3)2 Var(r!).
Since is bounded, the Law of Large number implies that Var(r?) can be estimated consistently by the empirical variance of (r1 (r1....,,h rn).
C.3. Proof of Theorem 3.3
The proof is very similar to Bates et al. (2021), except that they only consider expected risk measures. If R2) S for all E A, then the result obviously holds. Assume suPxeA (A) > Since (A) is continuous and strictly increasing, it crosses exactly once. Let X* denote the crossing point, i.e., (X*) a. Then
Ro (A) > - A> X = 12,5 (A*) aa,
where the last line is due to the definition of X. By Theorem 3.1,
lim P(R (2*) R₂((²)) 8.
Since RA*A*)= = a, we have
lim sup (P.g ()) > a) 2 lim sup (A < (2))))) =
n-+00
n=000
The proof is then completed.
18

---

Conformal Tail Risk Control for Large Language Model Alignment
D. Distortion Risk Control for VaRs
D.I. Asymptotics of empirical quantiles
Take (q))) = (g(4), we obtain (A) = (B). Theorem 3.1 implies that the empirical 8-th quantile is asymptotically normal with variance
VaRg (A) 3 (3)
F (8))
Empirically, we use the standard Bootstrap procedure to estimate o RaH, (A) with 1000 samples.
D.2. Additional experimental results for VaRg
We evaluate the performance, i.e., average sampling cost, and realized VaR, on held-out dataset with [D] 6000, of our framework CDRC-L, against baseline methods CDRC-DKW and CDRC-BJ. See Fig. 13 for 0.68, and Fig. 14 for p= 0.78. Similar to what we observe with CVaRg, it is evident from the plots that methods control the risk specified level a since they fall below the black dotted line (where a VaRş) within the margin of error. Again, our method, CDRC-L, consistently the least conservative all methods across different values of 3, and p. As result, we that method is consistently the least costly across all settings of B, and p. Fig. 15 illustrates that for all settings of B, as the Spearman correlation coefficient increases, the cost of generating a VaRoo-contoolled LLM response decreases.
Realized VaRg, B 0.5
Realized VaRg, 0.75
Realized VaRg B 0.9
0.35
CDRC
0.35
CDRC CDRC-DKW
0.35
CDRC
CDRC-DKW
CDRC.DKW
0.30
CDRC.L
0.30
CORCIL
0.30
CDRC.L a VaR
VaRa
= VaR
VaRa
a VaR
0.25
0.25
0.25
0.20
0.15
0.15
0.15
0.2
0.25
0.35
0.15
0.2
0.25
0.35
0.15
0.2
0.25
0.3
0.35
Average Sampling Cost,
0.5
Average Sampling Cost, 8 0.75
Average Sampling Cost, B
0.9
CDRC-9)
CDRC-6)
CDRC-8)
1.6
CDRC-DKW CDRC-L
3.0
CDRC-DKW CDRC-L
CDRC-DKW CDRC-L
cost
1.4
cost
2.5
1.2
2.0
3
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
Q
Figure 12. Realized VaR, vs. (row 1) and average sampling cost (row 2), and for Spearman correlation between human
and machine toxicity scores at = 0.57 evaluated on held-out dataset with [D] = 6000. The confidence band is computed by taking the estimate plus/minus respective rows illustrates different setting of /3 E {0.5, 0.75, 0.9}. CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), it is able achieve risk control within the margin of error (shown in the black dotted line) while being more cost efficient and less conservative than both baseline methods.
standard error estimated from the results across the independent experiments. Each panel in the
19

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized VaRp. $ 0.5
Realized VaRg. B = 0.75
Realized VaRa. B 0.9
0.35
CDRC-8J
0.35
CDRC-8)
0.35
CDRC-8)
CDRC-DKW
CDRC-ORW
CDRC-DKW
0.30
CDRC-L
0.30
CDRC-L
0.30
CDRC-L
VaR
VaR
VaRg
0.25
VaR
VaRa
0.25
VaR
0.20
0.20
0.20
0.15
0.15
0.15
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
a
0
a
Average Sampling Cost, 0.5
Average Sampling Cost,
0.75
Average Sampling Cost,
1.6
CDRC-0) CDRC-DKW CDRC-L
2.75
CDRC-0)
4.5
CDRC-U
CORC-DKW
CORC-DIW
CDRC-L
4.0
CDRC-L
cost
1.4
2.25
cost
3.5
1.2
3.0
1.75
2.5
0.15
0.2
0.25
0.35
0.15
0.25
0.35
0.15
0.25
0.35
Figure 13. Average and realized VaR, on held-out dataset for p 0.68 with [D] 6000.
Realized VaRg, B == 0.5
Realized VaRg, B = 0.75
Realized VaRa. B 0.9
0.35
CDRC-8]
CDRC.B]
0.35
CDRC-DKW
CDRC-OKW
CDRC-OKW
0.30
CDRC-L
0.30
0.30
VaRa
VaR
VaRa
0.25
VaR
VaRg
0.25
VaR
0.20
0.20
0.20
0.15
0.15
0.15
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0
0
a
Average Sampling Cost,
0.5
Average Sampling Cost,
0.75
Average Sampling Cost,
CDRC-8) CDRC-DKW
2.4
CDRC-8)
3.25
CDRC-B)
CDRC-DKW
CORC-DKW
CDRC-L
2.2
CDRC-L
3.00
CDRC-L
cost
1.4
2.0
2.75
1.2
2.25
1.6
0.25
0.35
0.15
0.25
0.15
0.25
0.35
Figure 14. Average cost and realized VaRy held-out dataset for 0.78 with [D] 6000.
Similar to the results for CVaRg, from Fig. 16, it is evident that CDRC-L is consistently the least conservative among all methods, while still maintaining the guaranteed VaR, control within the margin of error. Furthermore, from Fig. 17, we observe that CDRC-L is consistently the most cost effective among all methods. In particular, significantly less expensive in comparison other methods in settings with small calibration set sizes.
20

---

Conformal Tail Risk Control for Large Language Model Alignment
01
0.15
02
0.20
d5
0.25
04 0.30
Rs 0.35
vs. Cost, B 0.5
Cost, B 0.75
Cost, B 0.9
3.00
1.6
2.75
1.5
2.50
1.4
4
2.25
1.3
2.00
1.2
3
1.75
1.1
1.50
2
0.57
0.68
0.78
0.57
0.68
0.78
0.57
0.68
0.78
0
p
Figure 15. Spearman correlation vs. average sampling cost for VaRa with [D]
6000.
Realized Callbration Set Size,
Average Sampling Cost Callbration Set Size,
2.2
CDRC-8) CDRC-L
Average Sampling Cost
2.0
0.25
1.8
36 0.20
1.6
0.15
CDRC-8] CDRC-DKW
1.4
0.10
CDRC-L
Target VaR
1,2
400 Calibration Set
600
800
1000
200
400 Calibration
600
800
1000
Figure Realized VaR, vs. calibration set size for 0.25, p 0.57.
Figure Average sampling cost VaR, control calibra- tion set size for 3 0.5, 0.25, p 0.57.
E. Additional model results
E.1. LLaMA3.1-8B
To evaluate our tail risk control framework using the LlaMA3.1-8B model, we choose a (25, 0.15, 0.25, 0.3) for E (0.5, 0.75), and a (0.2, 0.25, 0.3) for 0.9 for testing CVaR,, control. In addition, choose 0.5, and E {0.1, 0.15, 0.2, 0.25, 0.33 for 8 E {0.75, 0.9) for testing VaR, control. Using
{0.05, 0.075, 0.1} for 0.57, we illustrate the results for CVaR,, and VaR,, control in 18 and 19, respectively for [D] 6000.
We examine the performance, i.e. the realized CVaR, and VaR,, and average sampling cost of our framework, CDRC-L, on a held-out dataset against baseline methods CDRC-DKW and CDRC-BJ. Similar to what we observe with the LlaMA2-7B model, it is evident from the plots that all methods control the risk at the specified level a since they fall below the black dotted line (where CVaR, or VaR,)) within the margin of error. Meanwhile, our method, CDRC-L, is consistently the least conservative and hence most cost efficient of all methods across different values of B.
21

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized CVaRy, B = 0.5
Realized CVaRg, B = 0.75
Realized CVaRg. B 0.9
0.30
0.30
CDRCB]
0.3
CDRC DKW
CDRC-OKW
0.25
CDRC-L
CVaRg
CVaR
CVaRs
CVaR
CVaRa
0.2
0.20
0.1
CDRC-8I
0.15
0.10
CDRC-L
0.10
0.0
0.125
0.15
0.2
0.25
0.3
0.125
0.15
0.25
0.3
0.25
0.3
Average Sampling Cost, 8 = 0.5
Average Sampling Cost, 8 0.75
Average Sampling Cost. = 0.9
CORC-B) Bj
2.5
CDRC B)
CDRC: B]
CORC-DIO
CDRC-DKW CDRCAL
CDRC-
1.3
CORC-L
CDRC-L
2.0
cost
cost
1.2
4
1.5
2
0.125 0.15
0.2
0.25
0.3
0.125 0.15
0.2
0.25
0.3
0.2
0.25
0.3
Figure 18. LLaMAAJ1-BB:Raalize CVaR, vs. a (row 1), and average sampling cost vs. a (row 2) for Spearman correlation
between human and machine toxicity scores at p 0.57 evaluated on held-out dataset with [D] 6000. Each panel in the respective rows illustrates a different setting of E (0.5, 0.75, 0.9). CDRC-L (orange), an improvement to CDRC-DKW (green) and CDRC-BJ (blue). as it is able achieve risk control within the margin of error (shown in the black dotted line) while being cost efficient and less conservative than both baseline methods.
E.2. Llama3.2-3B
To evaluate our tail risk control framework using the LlaMA3.2-3B model, we choose E {0.15, 0.2,0.25, 0.3, 0.35) for E (0.5, 0.75), and a (0.2, 0.25, for 0.9 for testing CVaR,, control. In addition, choose {0.15, 0.2, 0.25, 0.3, 0.35) for 3 {0.5, 0.75, 0.9} for testing VaR, control. Using P 0.57, illustrate the results for C and VaR,, control in 20 and 21, respectively for [D] 6000.
Again, we examine the performance, i.e. the realized CVaR, and VaR,,, and average sampling cost of our framework, CDRC-L, on held-out dataset against baseline methods CDRC-DKW and CDRC-BJ. Similar to what we observe with the LlaMA2-7B and LlaMA3.1-88 model, is evident from the plots that all methods control the risk at the specified level since they fall below the black dotted line (where CVaR, or VaR,) within the margin of error. Meanwhile, our method, CDRC-L, is consistently the least conservative and therefore, the most least costly of all methods across different values of B.
22

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized Va 8 0.5
Realized VaRs.
0.75
Realized VaRg.
0.9
0.10
CDRC-8) CDRC-DKW
CDRC-8)
0.30
CDRC-6)
CDRC-DKW
CDRC-DRW
CDRC-L
0.25
CDRC-L
0.25
CDRC-L
VaR
0.08
VaR
VaR
0.20
0.20
0.06
0.15
0.10
0.05
0.075
0.1
0.15
0.2
0.3
0.1
0.15
0.2
0.25
a
Average Sampling Cost, /
Average Sampling Cost, 0.75
Average Sampling Cost, B 0.9
1.15
CDRC-6)
CDRC-OKW
1.20
(L3RC-8 R]
CDRC-DKN CDRC-L
1.6
CDRC-4)
CDRC-DKW
1.10
CDRC
1.15
CDRC-L
cost
cost
cost
1.4
1.10
1.05
1.05
1.00
1.00
0.075
0.1
0.15
0.2
0.25
Figure I.L.MA31-SB;: Realized VaR, vs. (row 1), and average sampling cost vs, a (row 2), and Spearman correlation between human and machine toxicity scores at p 0.57 evaluated on held-out dataset with [D] 6000. Each panel in the respective rows illustrates different setting E (0.5, 0.75, 0.9). CDRC-L (orange), an improvement CDRC-DKW (green) and CDRC-BJ (blue), as it is able achieve risk control within the margin of error (shown in the black dotted line) while being more cost efficient and less conservative than both baseline methods.
Realized CVaRg, , B 0.5
Realized CVaRg, B 0.75
Realized CVaRg. B 0.9
0.35
BJ
0.35
Bj
CDRC DKW
CDRC DKW
0.3
0.30
CDRC.L
CDRC-L
0.25
CVaR
CvaRa
0.25
CVGR
CVaRg
0.2
0.20
0.20
CDRC-8)
0.1
CDRC DKW
CDRC-L
0.10
0.0
CVaR
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.25
0.3
0.35
0
Average Sampling Cost,
0.5
Average Sampling Cost, β
0.75
Average Sampling Cost,
0.9
CDRC-8) Bj
3.5
CDRC-8)
15.0
CDRC-8 Bj
2.50
CDRC-L
CORC-DKW
CDRC-L
CDRC-DKW
12.5
CDRC-L
CDRC-DKW
3.0
cost
10.0
2.00
7.5
2.5
1.75
5.0
1.50
2.0
2.5
0.2
0.35
0.25
0.35
Figure Llama3.2-3B: Realized CVaR, vs. (row 1), and average sampling cost vs. (row 2) for Spearman correlation between
human and machine toxicity scores at p 0.57 evaluated on held-out dataset with [D] = 6000. Each panel in the respective rows illustrates a different setting of 3 (0.5, 0.75, 0.9). CDRC-L. (orange), an improvement CDRC-DKW (green) and CDRC-BJ (blue), as is able achieve risk control within the margin of errrr (shown in the black dotted line) while being more cost efficient and less conservative than both baseline methods.
23

---

Conformal Tail Risk Control for Large Language Model Alignment
Realized VaRg B 0.5
Realized VaRg. B = 0.75
Realized VaRg B 0.9
0.35
CDAC-9J
0.35
CDRC-8)
0.35
CDRC-9)
CDRC-DKW
CDRC-DKW
CDRC-DKW
0.30
CDAC-L
0.30
CDRC-L
0.30
CDRC-L
VaRg
0.25
VaR
VaRg
0.25
VaR
vaRa
0.25
VaR
0.20
0.20
0.15
0.15
0.15
0.10
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
0.15
0.2
0.25
0.3
0.35
a
a
Average Sampling Cost, = 0.5
Average Sampling Cost,
Average Sampling Cost, f
CDRC 3)
CDRC
CDRC
1.3
CDRC DKW
2.0
CDRC DKW
3.00
CDRC DKW
CDRC-L
CDRC
CDRC
2.75
cost
1.2
cost
1.8
1.6
2.25
1.0
1.4
2.00
0.15
0.2
0.25
0.35
0.15
0.25
0.35
0.15
0.25
0.35
Figure 21. Llama3.2-3B: Realized VaR,, vs. (row 1), and average sampling cost vs. a: (row 2) for Spearman correlation between
human and machine toxicity scores at p 0.57 evaluated held-out dataset with |D] 6000. Each panel in the respective rows illustrates a different setting of 3 {0.5, 0.75, 0.9}. CDRC-L (orange), is an improvement to CDRC-DKW (green) and CDRC-BJ (blue), is able achieve risk control within the margin error (shown in the black dotted line) while being more efficient and less conservative than both baseline methods.
24


Reviews:
[{"review_id": "sG5zEbPFbI", "reviewer": "Reviewer_adyZ", "summary": "The paper proposes an inference-time alignment procedure to control the risks associated with LLM outputs. It assumes access to a disutility function that can score LLM’s outputs. It works by generating LLM responses until it gets a response that has disutility score below a threshold $\\hat{\\lambda}$ determined during the offline calibration stage. The technical part of the paper is concerned with estimating this threshold in a principled way so that PAC-style guarantees can be made on the risk of the output responses from the procedure. The authors propose an L-statistic based estimator for risk and using asymptotic normality result (van der Vaart 1998) obtain an upper confidence bound on the risk estimate and use it to estimate the threshold $\\lambda$. They also consider DKW (Dvoretzky–Kiefer–Wolfowitz) inequality and BJ (Berk-Jones) statistic based upper confidence bounds as well. Empirical results on a toxicity dataset show that the proposed method with L-statistic is effective in achieving the desired risk level while the method with DKW or BJ tend to be more conservative and thus will draw more samples to get to a sample meeting the acceptance criterion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback. We’re grateful for the recognition of the **sound theoretical foundations, novel adaptation of conformal methods to control tail risks, and clear visual illustrations**. Below we address the main concerns:\n\n### **Weakness 3: Additional inference cost**\nThere are inherent added costs when ensuring guarantees for risk-aware inference. However, our results show that **DRC-L achieves tight bounds with minimal overhead**. In particular, DRC-L achieves **tight bounds** for controlling tail risks, allowing us to maintain the desired risk level with **lower cost** compared to baselines. Further, compared to a fixed best-of-N strategy, DRC-L achieves lower deployment cost to control the risk at the same level.\n\n### **Weakness 4: Limited evaluation**\nWe add results for more models (e.g., Llama3.2-3B, Llama3.2 8B, and Qwen2.5-7B) on new datasets (e.g., RealToxicPrompts (RTP)), under the same setup as Section 4.1. We observe that **DRC-L consistently satisfies the CVaR constraint with lower cost than baselines**. We highlight that our method is **dataset and model-agnostic**, and we will include results for larger models and different $\\alpha$, $\\beta$ choices in the revision. Below, we report an example realized CVaR and cost (mean $\\pm$ standard error) over 15 independent runs. \n\nTable 1: Realized CVaR and average cost on RTP dataset with Qwen2.5-1.5B model. \n\n| Method | $\\beta$| $\\alpha$ | Realized CVaR | Cost |\n|--------|------|-------|----------------------|----------------------|\n| BJ     | 0.75 | 0.25  | $0.22823 \\pm 0.00707$ | $1.11985 \\pm 0.00861$ |\n| DKW    | 0.75 | 0.25  | $0.19144 \\pm 0.00790$ | $1.21905 \\pm 0.01869$ |\n| DRC-L  | 0.75 | 0.25  | $0.24267 \\pm 0.00714$ | $1.10184 \\pm 0.00791$ |\n\nWe will include additional results in our revision.\n\n### **Weakness 5/Suggestions: Presentation**\nWe will revise Section 3 to improve clarity and motivation, connecting each step to the overall goal. Additionally:\n\n- **Simplify notation** as recommended, and unify subscript usage\n- Add a **notation table**\n- Switch from “disutility” to “reward” to align with conventions\n- Replace yellow in Figure 3 with a darker color\n\n### **Q1: Comparison to inference-time alignment methods**\nWe highlight that Best-of-N is a **fixed-sample inference-time heuristic**, while our method is an **adaptive, risk-controlling strategy**. Best-of-N always samples N responses, regardless of prompt toxicity, which can lead to overhead. In contrast, DRC uses as few samples as needed. Following the procedure in Section 4, we evaluate the realized CVaR of human toxicity scores, selecting the response with lowest machine score from $N \\in {3, 5}$ samples, with $\\beta = 0.5$ to demonstrate that our method matches or improves on these values with **lower average cost** (compare with Fig. 4):\n\nTable 1: Realized CVaR of best-of-N algorithm on a calibration set size n.\n| **n**  | **Best-of-5 CVaR** | **Best-of-3 CVaR** |\n|--------|--|--|\n| 1000   | 0.2015 | 0.2427 |\n| 2000   | 0.2013 | 0.2384 |\n| 3000   | 0.2003 | 0.2405 |\n\nFinally, best-of-N offers **no statistical guarantees**, whereas our framework delivers PAC-style guarantees for distortion risks.\n\n### **Q2: Can theoretical claims be demonstrated on synthetic data?**\nYes, we will include a **synthetic experiment with a long-tailed loss distribution** to illustrate the theoretical claims in our revision.\n\n### **Q3: What does the threshold $\\hat{\\lambda}$ and Eq. (7)'s probability look like? What is $\\psi$?**\nWe will **add theory and plots to illustrate $\\hat{\\lambda}$ and the probability in Eq. (7)**. We can **visualize the distribution** of the number of inference rounds required under different risk levels $\\alpha$ and distortion functions $\\psi$, to show how aggressively the tail is penalized. Our theoretical results show that $\\hat{\\lambda} \\stackrel{p}{\\rightarrow} \\lambda^*$, and thus expected cost converges according to Eq. (7). The distortion function $\\psi$ in our CVaR experiments is $\\psi(p) = \\max(p - \\beta, 0)/(1 - \\beta)$. For VaR, $\\psi$ is an indicator function at $p = \\beta$. We will discuss other $\\psi$ functions and their implications in the revision.\n\n### **Q4: Anecdotal evidence**\nWe agree that examples are helpful for understanding how our procedure works. Below, we provide an example from our experiments.\n\nTable 1: LLM-generated responses with human and machine toxicity scores for prompt: “If you find it necessary for Assad to bomb innocent children in order to kill”\n| **Generated Response** | **Machine Score $s(y_i(x))$** | **Human Score $r(y_i(x))$** |\n|-|-|-|\n| terrorists then you're a terrorist yourself...| 0.25 | 0.90 |\n| the bad guys I then I guess you don't mind that we do the same... | 0.11 | 0.69 |\n| then you are not an anti-war person... | 0.13 | 0.36 |\n\nIf $\\lambda = 0.2$, our method selects the last two responses, which have a maximum human score of 0.69. We will include similar examples in the appendix.", "claims_and_evidence": "Yes. The theoretical claims are backed with sufficient details and proofs. Empirical results are sound as well, clearly showing the claims about the method's ability to achieve the desired risk level and the conservativeness of the variants based on DKW and BJ.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are appropriate for the problem being tackled. The method draws samples from LLM until it finds a sample that meets the acceptance criteria. Due to this, it is able to provide guarantees on the quality of the responses. The evaluation considers the method's ability to achieve the target risk tolerance level and the inference cost of drawing samples.", "theoretical_claims": "The theoretical claims are based on standard results in statistics.", "experimental_designs_or_analyses": "I do not see any major issues except,\n\n1.  Experiments are limited to one dataset and one model. It is not clear how well do these claims generalize to other settings. \n\n\n2. Common approaches for inference-time alignment such as best-of-N, etc. are not included. It would be nice to include these baselines to see their risk levels.", "relation_to_broader_scientific_literature": "The key contribution is a procedure to control the risk of LLM responses with theoretical guarantees. While the statistical (theoretical) tools used are well-established in the literature, their adaptation to this problem setting is novel.", "essential_references_not_discussed": "This is fine.", "other_strengths_and_weaknesses": "1. The proposed method is theoretically sound and backed with guarantees on controlling risks.\n \n2. I generally liked the presentation of the ideas in the paper. Appreciate Figure 2 in clearly showing how thresholds work and the scores are computed.  \n\n3. Additional inference time is a major drawback. In particular if the estimated threshold is very small then it might end up taking a lot of inference rounds to get to an acceptable sample. \n\n4. Experiments are limited to one dataset and one model and other inference-time baselines are not included in the evaluation. \n\n5. Some aspects of the presentation can be improved. I have commented on those below.", "other_comments_or_suggestions": "1. The presentation can be simplified further. For example, section 3 gets too dry with lots of details and loses connection with the main problem. It might be worth reiterating the motivations and how the things proposed in this section help. Instead of jumping into the details, it helps to see why we are doing what we are doing and gently walking through these things will help in improving the readability. \n\n\n2. Is it necessary to use $y(x)$, at some places only $y$ is used. It might be good to just use $y$. $r_{m}$ and $r_{\\lambda}$ have subscripts with different meanings, this may cause confusion. Similarly there is another $r$ for human disutility function. Also, the general convention in LLM literature is to use $r$ for reward (higher is better), but here it is measuring the “disutility” (lower is better). These things can cause confusions and would be good to fix notations to avoid any confusions.\n\n\n3. I’d suggest using some other darker color in place of yellow in Figure 3.", "questions_for_authors": "1. Can we see the performance of other inference-time alignment procedures such as best-of-N?\n\n2. Is it possible to instantiate theoretical claims on simulated data? I assume we don’t need LLMs for this.\n\n3. How do the thresholds $\\hat{\\lambda}$ and the probability in eq. 7 look? What is the $\\psi$ function used? \n\n4. Anecdotal evidence (examples) on how the inference procedure worked in practice would be helpful.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4jYvrsQXkZ", "reviewer": "Reviewer_Wot5", "summary": "This paper focuses on the application of conformal prediction to tail events, which can lead to poor outcomes. The authors propose a lightweight calibration framework for black-box models that ensures alignment between humans and machines with provable guarantees. They utilize L-statistics, the DKW inequality, and Berk-Jones statistics for conformal risk control. Extensive experiments are conducted to validate their proposed method. The application of statistical methods to conformal risk control is an interesting direction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and questions. We are encouraged by the recognition of several key strengths:\n\n- Our **detailed theoretical analysis** and use of L-statistics in conformal risk control;\n- The **practical relevance** of our proposed framework, including its lightweight, post hoc nature and ability to provide provable guarantees without retraining;\n- The **importance and novelty** of tackling tail risk in machine-human alignment for LLMs.\n\nBelow, we address the concerns in detail:\n\n### **Weakness: Writing, Related Work, and Experiments**\nThank you for this valuable feedback. In the revised version, we will:\n\n- Add a **Related Work** section, situating our approach alongside prior work on conformal risk control, and LLM alignment;\n- Reorganize the **Introduction** to better highlight our core contributions;\n- **Expand the experimental section** with new results using additional LLMs (LLaMA3.2-3B and Qwen2.5-1.5B) and larger models, e.g., Llama3.2 8B and Qwen2.5-7B, with more datasets (e.g., RealToxicPrompts), best-of-N baselines, and cost ablations.\n\n### **Weakness: Similarity between Figures 4 and 5**\nThank you for pointing this out. Indeed the first row of Figures 4 and 5 illustrates the realized CVaR versus $\\alpha$, demonstrating  our method consistently controls tail risk. The second row shows the average sampling cost versus $\\alpha$, which decreases as the target level $\\alpha$ increases. Although they serve different purposes (tail risk control vs. efficiency), the visual similarity can be misleading.\n\nTo improve clarity, we propose **reformatting the cost results** to highlight the differences more explicitly. In particular, we present **percent cost reductions** of our method (DRC-L) relative to baselines (BJ and DKW). Sample results are shown to demonstrate the efficiency gains provided by our approach (we have complete results for other LLMs, and difference choices of $\\rho$, and $\\beta$)\n\nTable 1: Percent cost reduction for CVaR of our method (DRC-L) compared to baselines, with $\\rho = 0.78$, $\\beta = 0.5$\n\n| Method | $\\alpha$ = 0.15 | $\\alpha$ = 0.2 | $\\alpha$ = 0.25 | $\\alpha$ = 0.3 | $\\alpha$ = 0.35 |\n|------------|----------|--------|----------|--------|----------|\n| DRC-BJ     | 5.75     | 6.64   | 5.76     | 1.39   | 0.52     |\n| DRC-DKW    | 30.54    | 26.38  | 21.46    | 18.08  | 15.24    |\n\nWe will update the figures and accompanying text in the revision to improve clarity and better guide the reader.\n\n### **Justification for L-statistics vs. BJ and DKW**\nThank you for this important point. Our work is centered on **controlling tail risk** using distortion risk measures such as CVaR. Prior work like Quantile Risk Control (QRC) by Snell et al., extended conformal methods to tail risks using concentration inequalities like BJ and DKW, which provide general-purpose CDF bounds. However, these are not tailored to distortion risks and tend to be conservative (see lines 322-329). Our key contribution lies in **leveraging L-statistics**, which are inherently aligned with the structure of distortion risks by modeling them as a linear combination of order statistics weighted by a function $\\psi$. This allows us to:\n\n- **Directly bound the risk functional**;\n- Achieve **tight asymptotic approximations**;\n- Deliver **lower deployment cost** with valid guarantees.\n\nWe will revise Section 3.2 to explicitly justify this choice and contrast it with baselines.\n\n### **Is this the only viable approach for conformal tail risk control?**\nOur work builds directly on QRC (Snell et al.), which introduced BJ and DKW for tail risk control. These methods are described in Section 3.4. Our work is the **first to use L-statistics** in the given context. While other methods like BJ and DKW are viable, our empirical results consistently show that L-statistics:\n\n- Better **match the structure of distortion risks**;\n- Provide **sharper bounds** and lower sample costs;\n- Are **less conservative** for distortion risk control.\n\nWe will clarify this distinction in the revised manuscript.\n\n### **Why do BJ, DKW, and L-statistics perform differently?**\nThe performance differences stem from how each method models uncertainty:\n\n- **DKW** offers uniform control over the CDF but does not emphasize the tail—making it overly cautious for rare events.\n- **BJ** is more sensitive to small tail deviations than DKW but remains a general-purpose bound and thus conservative. Moreover, it is computationally much more costly than our method and DKW.\n- **L-statistics**, in contrast, are designed for distortion risk, directly estimating tail-weighted quantiles using the given weight $\\psi$. This structural alignment allows for tight risk bounds and consistent performance across calibration sizes. In fact, when the size of the calibration set is large, our risk control is exact and hence not conservative.", "claims_and_evidence": "Yes, they claim that their method addresses the issue of unexpectedly poor outcomes by aligning humans and machines with provable guarantees. The experimental results on the CVaR metric support their claim, demonstrating the effectiveness of their approach. Additionally, the experiment on deployment cost confirms their intuition that better-aligned machine ratings reduce the cost of calibration.", "methods_and_evaluation_criteria": "The paper proposes applying three different statistical methods for conformal risk control. However, it does not provide any justification for why these specific methods are needed. For instance, in Section 3.2, it would be beneficial to explain why the L-statistic is particularly suitable for conformal risk control rather than relying solely on theoretical proof.  \nAdditionally, in Figures 4 and 5, if my understanding is correct, the first row represents realized CVaR versus α, while the second row shows average sampling cost versus α. However, these figures appear too similar, which may cause confusion for readers. To improve clarity, it would be better to introduce more distinct differences between the figures rather than relying solely on the text captions.\nFor Evaluation Criteria, the cvar metric make sense for supporting their claims, the deployment cost also confirms our intuition that better-aligned machine ratings reduces the cost of calibration.", "theoretical_claims": "Yes, they are correct", "experimental_designs_or_analyses": "Yes, their experiments support the claims made in the results section, as they conduct extensive evaluations to demonstrate the effectiveness of their proposed method. However, they do not appear to include baseline models, which raises a question: can standard conformal risk control models not achieve the goals outlined in this paper? Additionally, their experiments are limited to LLaMA-7B and a single dataset, which restricts the generalizability of their conclusions. Expanding the evaluation to multiple models and datasets would strengthen the validity of their findings.", "relation_to_broader_scientific_literature": "This method contributes by addressing unexpectedly poor outcomes using a statistical approach, a problem that traditional methods do not specifically tackle. The focus on this issue is particularly meaningful, as mitigating unexpected poor outcomes is crucial, especially in industrial settings where such failures can result in significant costs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness: 1.Writing is a major weakness of this paper, e.g. Lack of related work, no contribution section in introduction, the experiment is also too short.\n\n2.Their experiment setting seems not comprehensive as we talked before. The caption of figure is not accurate.\n\nStrength: \n1. They provide detailed theoretical analysis for their proposed method. \n2.The proposed method is intriguing and has the potential to make a significant impact in this field.\n3.The topic is interesting and important.\n4.Figure 1 and Figure 2 are insightful.", "other_comments_or_suggestions": "Find in previous section", "questions_for_authors": "Is it possible to compare the performance of other conformal risk control methods on the problem addressed in this paper? Is the proposed method the only viable approach for conformal tail risk control? Additionally, could you explain why the three statistical methods exhibit different performance levels? What make the difference of performance to be consistent?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuJv7O5A6n", "reviewer": "Reviewer_Mvxh", "summary": "To avoid the high cost of human annotations, researchers have developed automatic scoring models to assess the tail events produced by LLMs, such as toxic answers.\nHowever, there may be a misalignment between human judgement and model scoring. To solve this issue, this study proposes a lightweight calibration framework through the lens of risk control to ensure the alignment of humans and machines with provable guarantees. In addition, the authors demonstrate the utility of their calibration framework through experiments on a semi-synthetic benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughtful feedback and are encouraged by the recognition of our paper’s strengths, including:\n\n- The **novelty** of the problem setting—tail risk control in LLM alignment with provable guarantees;\n- The **rigor and clarity** of our writing and symbolic notation;\n- The practicality of our **lightweight alignment** method that avoids LLM retraining.\n\n### **Comparison with RLHF and existing alignment methods**\n\nThank you for the suggestion. Our method differs fundamentally from RLHF-based techniques. RLHF typically requires model access and expensive retraining, while our method is **post hoc**, requiring no access to model internals or parameter updates. It is thus more scalable and deployable.\n\nFurther, RLHF lacks formal guarantees for safety or tail risk control. Our approach directly addresses this gap by providing statistically provable control over tail disutility metrics (e.g., CVaR and VaR). We will include a qualitative comparison table in the revision to emphasize these distinctions.\n\n### **Weakness 1: Practicality, real-world experiments, and prompt distribution assumptions**\nWe agree that practicality is essential. We conducted real-world evaluations using Qwen2.5-1.5B and LLaMA3.2-3B on the RealToxicPrompts (RTP) dataset, beyond LLaMA-2-7B. These experiments confirm that DRC-L consistently satisfies risk control requirements while minimizing inference-time cost.\n\nRegarding distribution assumptions: while all conformal methods require the calibration distribution to reflect deployment,**our framework supports reweighting calibration examples by density ratios (aka importance weights) to handle distribution shifts**, thus improving robustness in practice. We will add a theorem showing the validity of the reweighted DRC-L in the revision. \n\nWe also empirically find that for a given risk level, a reliable $\\lambda$ threshold can be estimated adaptively, even with modest calibration sizes. We will expand on this with clarifying details and additional ablations in the revision.\n\n### **Weakness 2: Reliance on a single model**\nThank you for the feedback. We have conducted additional experiments using the Llama3.2-3B and Qwen2.5-1.5B models on the RealToxicPrompts (RTP) dataset, following the same setting as illustrated in Section 4.1 in our manuscript. The results show that **DRC-L consistently meets the risk control requirement while achieving lower cost compared to DKW and BJ**.  For all tables, the metrics are denoted by “average ± standard error” with 15 independent experiments. Below we show the sample results of the Llama3.2-3B model with $\\beta = 0.5$, and $\\alpha = 0.25$. We have additional experiments with other models (e.g., Qwen2.5-1.5B), and different settings of $\\alpha$ and $\\beta$.\n\nTable 1: Realized CVaR and average cost on RTP dataset with Llama3.2-3B model.\n| Method | $\\beta$| $\\alpha$ | Realized CVaR                 | Cost                 |\n|---------|------|-------|----------------------|----------------------|\n| DRC-BJ  | 0.5  | 0.25  | $0.23296 \\pm 0.00520$ | $1.71155 \\pm 0.02993$ |\n| DRC-DKW | 0.5  | 0.25  | $0.22205 \\pm 0.00544$ | $1.73714 \\pm 0.03235$ |\n| DRC-L   | 0.5  | 0.25  | $0.24469 \\pm 0.00534$ | $1.65423 \\pm 0.02630$ |\n\nTable 2: Realized VaR and average cost on RTP dataset with Llama3.2-3B model. \n| Method         | $\\beta$| $\\alpha$| Realized VaR                  | Cost                  |\n|----------------|------|-------|----------------------|-----------------------|\n| DRC-BJ         | 0.5  | 0.25  | $0.21830 \\pm 0.00831$ | $1.11849 \\pm 0.01100$ |\n| DRC-DKW       | 0.5  | 0.25  | $0.22295 \\pm 0.00805$ | $1.11138 \\pm 0.01153$ |\n| DRC-L  | 0.5  | 0.25  | $0.24219 \\pm 0.00905$ | $1.08460 \\pm 0.01146$ |\n\nWe will add these results to the paper to strengthen the empirical claims.\n\n### **Clarification: Biased Scoring Model via c% Extremes**\nThank you for raising this. To simulate a biased scoring model, we retrain Detoxify on a dataset composed of texts with the **top and bottom c% of toxicity scores**, removing the middle. This produces a model skewed toward extreme judgments and serves as a proxy for biased disutility classifiers. We will revise the manuscript to clarify this process.\n\n### **Clarification: Which component is aligned—LLM or classifier?**\nThis is a great question. Our method supports **two complementary alignment views**:\n\n1. **Aligning the LLM**: Holding the classifier fixed, our procedure filters LLM outputs to meet human-aligned thresholds.\n3. **Recalibrating the classifier**: Holding the LLM fixed, we can use our method to adjust $\\hat{\\lambda}$, effectively realigning machine scores to better match human disutility.\n\nWe will include a pipeline diagram in the revised manuscript to illustrate this dual perspective and how the components interact.", "claims_and_evidence": "Lines 76-78: \"In this work, we explore how distortion risk control can be applied to *align LLMs with respect to any disutility metric* ...\";\n\nLines 59-62: \"Although *machine ratings* are inexpensive and scalable, the misalignment, or lack of rank preservation between the machine and human ratings diminishes its reliability. \".\n\nWhich component should be aligned with humans in this work: the LLM or the toxicity classifier? A more explicit clarification or a overall pipeline **diagram** could facilitate better understanding.", "methods_and_evaluation_criteria": "This work requires valid baseline comparisons, not just comparisons between variations of the proposed method. \n\nFor existing alignment methods such as RLHF, the authors should consider providing a quantitative or qualitative comparison table to summarize and highlight the advantages of the proposed method over existing studies.", "theoretical_claims": "I have no questions about the theoretical claims in the paper.", "experimental_designs_or_analyses": "1. Lines 378-379: \"... dataset that consists of c% of most and least toxic instances\". Could you provide a more detailed explanation for this?\n2. The experimental results all come from a single model, Llama-2-7B. More LLMs should be taken into consideration to avoid the randomness of results and ensure the effectiveness of conclusions.\nIn particular, possible differences in sampling costs between LLMs might make the cost analysis more complete.", "relation_to_broader_scientific_literature": "This work proposes a lightweight and provably guaranteed value alignment framework for LLM.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The writing and symbols used are rigorous and standardized. \n2. A new alignment method is proposed that does not require LLM parameter updating.\n\nWeakness: \n1. The practicality of the method is questionable. The method requires that the prompt distribution is consistent with the data, which will impose requirements on the number and quality of prompts required for correction; on the other hand, there is a lack of real-world experiments. In practical applications, for a given risk level, whether an effective λ can be found and the required conditions are unclear.\n2. The experimental results all come from a single model, Llama-2-7B, which cannot ensure the effectiveness of the method and conclusions enough.", "other_comments_or_suggestions": "See above sections.", "questions_for_authors": "See above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHxWgsarUA", "reviewer": "Reviewer_N5H5", "summary": "The paper studies the problem of making sure that the LLM outputs align with human preferences. To this end, they construct an approach where the output of the LLM is returned only if its machine (disutility) score is lower than a \"toxicity\" threshold. Since machine scores can be different than \"true\" human scores, they construct a calibration procedure based on risk control. This allows them to find a threshold that guarantees that some tail statistics (e.g. ,CVaR) of the true disutility score of the returned answers will be lower than a user-chosen control level alpha.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback. We are encouraged by the recognition of key strengths in our submission, including:\n\n- The **novelty** of the problem setting—controlling tail risk in LLM outputs with distribution-free guarantees;\n- The **practical relevance and effectiveness** of our method, supported by empirical results.\n\n### **Weakness 1: Clarifying novelty beyond QRC [2]**\nThank you for the thoughtful feedback. In [2], QRC relies on concentration  inequalities on the cumulative distribution functions (CDFs), specifically the BJ and DKW inequalities, to derive uniform upper bounds between the empirical and true quantiles. While effective, these bounds are conservative as they do not take the weight function $\\psi$ into account. In contrast, **our work proposes a novel approach by formulating distortion risk control through L-statistics**, which are tailored to the form of the distortion function $\\psi$. This allows us to:\n- **Directly bound the distortion risk functional** rather than the entire CDF;\n- **Leverage asymptotic normality** of L-statistics to obtain asymptotically tight , more efficient bounds;\n- **Reduce sampling costs significantly** while maintaining risk guarantees, as demonstrated in our empirical results.\n\nWe will revise the introduction and background to clearly differentiate our contribution from prior work.\n\n### **Weakness 2: Statistical efficiency at smaller calibration sizes**\n\nThis is a valid concern. We conducted an ablation experiment on the calibration sizes and observed how deployment cost changes with that. This is illustrated in the following tables.\n\nWe observe that as the calibration size decreases, all methods become more conservative—realized costs increase and risk metrics deviate more from their expected values. However, DRC-L is still statistically more efficient. Interestingly, DRC-L also shows the smallest increase compared with DRC-DKW and DRC-BJ. For example, when the calibration size drops from 6000 to 1000, the cost of DRC-L increases by only 7.7%, compared to 17% for DRC-BJ and 28% for DRC-DKW, highlighting the advantage of L-statistics and its robustness to sample size in risk estimation.\n\nTable 1: Realized average cost with calibration size n of our method (DRC-L) and baselines for CVaR, $\\alpha = 0.25$, $\\beta = 0.75$\n| n | DRC-BJ | DRC-DKW | **DRC-L**  |\n|-------------|--------|---------|--------|\n| 1000        | 5.3074 | 6.5431  | **4.5604** |\n| 2000        | 4.9560 | 5.8082  | **4.4184** |\n| 3000        | 4.7769 | 5.3465  | **4.3156** |\n| 6000        | 4.5313 | 5.1037  | **4.2362** |\n\nTable 2: Realized CVaR with calibration size n of our method (DRC-L) and baselines for $\\alpha = 0.25$, $\\beta = 0.75$\n| Sample Size | DRC-BJ  | DRC-DKW | **DRC-L**  |\n|-------------|---------|---------|--------|\n| 1000        | 0.1834  | 0.1247  | **0.2236** |\n| 2000        | 0.2041  | 0.1536  | **0.2305** |\n| 3000        | 0.2128  | 0.1720  | **0.2367** |\n| 6000        | 0.2162  | 0.1884  | **0.2396** |\n\nTable 3: Realized average cost with calibration size n of our method (DRC-L) and baselines for VaR, $\\alpha = 0.25$, $\\beta = 0.75$\n| n | DRC-BJ | DRC-DKW | **DRC-L**  |\n|-------------|--------|---------|--------|\n| 1000        | 2.4266 | 2.3870  | **2.1309** |\n| 2000        | 2.3063 | 2.2917  | **2.1167** |\n| 3000        | 2.2181 | 2.2524  | **2.1174** |\n| 6000        | 2.2335 | 2.2140  | **2.0955** |\n\nTable 4: Realized VaR with calibration size n of our method (DRC-L) and baselines for $\\alpha = 0.25$, $\\beta = 0.75$\n| Sample Size | DRC-BJ  | DRC-DKW | **DRC-L**  |\n|-------------|---------|---------|--------|\n| 1000        | 0.1954  | 0.2006  | **0.2423** |\n| 2000        | 0.2117  | 0.2150  | **0.2444** |\n| 3000        | 0.2233  | 0.2197  | **0.2442** |\n| 6000        | 0.2237  | 0.2274  | **0.2495** |\n\nWe hope this addresses your concern. Please let us know if there are any other aspects you'd like us to discuss further.\n\n### **Weakness 3: Terminology clarification on “Conformal Risk Control”**\n\nWe appreciate this point. As the reviewer notes, [4] defines conformal risk control in expectation, while our method provides high-probability guarantees, making it more closely aligned with the RCPS framework [1].\n\nIn our manuscript, we use the term “conformal risk control” more broadly to refer to the overarching goal of risk-aware inference under distribution-free guarantees. However, we agree that this may cause confusion and will revise the terminology to better reflect our actual contribution. Specifically, we will reframe our method as **“conformal tail risk control with high probability”**, to distinguish it from the expectation-based guarantees in [4] and emphasize its closer connection to [1].\nWe appreciate this clarification and will ensure the distinction is made explicit in the revised manuscript. Let us know if you would like us to elaborate on this point further.", "claims_and_evidence": "/", "methods_and_evaluation_criteria": "/", "theoretical_claims": "/", "experimental_designs_or_analyses": "/", "relation_to_broader_scientific_literature": "/", "essential_references_not_discussed": "/", "other_strengths_and_weaknesses": "Strengths:\n- The setting considered is interesting and, to the best of my knowledge, novel.\n- The experiments mainly confirm the claims made in the paper. \n\nWeaknesses:\n- The manuscript is missing a related work section. Moreover, the background is written quite confusingly in my opinion. This makes it hard to understand what theoretical statements are novel and which are part of the prior work. Concretely, from my understanding the original risk control paper [1] has been extended from expected loss to VaR/CVaR loss already in [2]. So the contribution of this manuscript is not in moving from the expected loss to VaR/CVaR, but rather only in considering L-statistics to construct an upper confidence bound needed to find the risk-controlling threshold (Section 3.2). Could the authors elaborate on this? Especially, could the authors clarify in detail how their work extends and differs from [2]\n- The main \"cost\" of calibration are the human annotations needed. For $n$ prompts and $N$ LLM samples per prompt, this amounts to $n \\times N$ human annotations. The size of the calibration dataset in the experiments is $n=6000$ and $N=40$ which would require $240K$ human annotations. This contradicts some of the claims made by the authors of their approach being 'light-weight'. The authors circumvent this in their experiments by replacing human annotations with the Detoxify model. One possible way to address this issue, would be to show that their calibration is still statistically efficient (i.e., the empirical test risk being close to the diagonal) for smaller calibration set sizes, e.g. when using only $n=100$ prompts. Note that risk control for smaller calibration sizes has been studied before, see for example [3]\n- One nitpick: the authors talk about conformal risk control which I find a bit confusing, since the original conformal risk control paper [4] is about controlling the risk in expectation (w.r.t to the draw of a calibration dataset), whereas this manuscript is about providing the guarantee with high probability making it more similar to [1] than to [4]\n\n[1] Bates, S., Angelopoulos, A., Lei, L., Malik, J. and Jordan, M. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM) 2021\n\n[2] Snell, J.C., Zollo, T.P., Deng, Z., Pitassi, T. and Zemel, R. Quantile risk control: A flexible framework for bounding the probability of high-loss predictions. ICLR 2023\n\n[3] Jazbec, M., Timans, A., Veljković, T.H., Sakmann, K., Zhang, D., Naesseth, C.A. and Nalisnick, E. Fast yet safe: Early-exiting with risk control. NeurIPS 2024\n\n[4] Angelopoulos, A.N., Bates, S., Fisch, A., Lei, L. and Schuster, T. Conformal risk control. ICLR 2024", "other_comments_or_suggestions": "/", "questions_for_authors": "/", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel approach to inference-time alignment of large language models (LLMs), using established statistical tools to determine a threshold for accepting model outputs. A primary concern in the original submission was the limited empirical validation, which relied on only one dataset and one model. Additional major concerns included the discussion of related work and the overall clarity and quality of the presentation.

In their revision, the authors have provided a substantially expanded set of experimental results and have indicated their intention to address the remaining issues. However, the final quality of the paper, particularly in terms of presentation and contextualization, is still uncertain.

That said, the paper offers a meaningful contribution by introducing a novel and practically relevant method for improving LLM alignment, and demonstrates its potential through empirical evidence.
Decision: Accept (poster)