Large language model validity via enhanced conformal prediction methods
Authors: John Cherian
Isaac Gibbs
Emmanuel Candes
Venue: NeurIPS 2024 poster (2024)
Keywords: conformal inference
large language models
conditional guarantees
calibration
boosting
Primary area: safety_in_machine_learning

Abstract: We develop new conformal inference methods for obtaining validity guarantees on the output of large language models (LLMs). Prior work in conformal language modeling identifies a subset of the text that satisfies a high-probability guarantee of correctness. These methods work by filtering claims from the LLM's original response if a scoring function evaluated on the claim fails to exceed a threshold calibrated via split conformal prediction. Existing methods in this area suffer from two deficiencies. First, the guarantee stated is not conditionally valid. The trustworthiness of the filtering step may vary based on the topic of the response. Second, because the scoring function is imperfect, the filtering step can remove many valuable and accurate claims. We address both of these challenges via two new conformal methods. First, we generalize the conditional conformal procedure of Gibbs et al. (2023) in order to adaptively issue weaker guarantees when they are required to preserve the utility of the output. Second, we show how to systematically improve the quality of the scoring function via a novel algorithm for differentiating through the conditional conformal procedure. We demonstrate the efficacy of our approach on biography and medical question-answering datasets.

Large language model validity via enhanced conformal prediction methods
John J. Cherian
Isaac Gibbs Department of Statistics Stanford University ig bggbbfsttdoooddd edu
Emmanuel Candès Department of Statistics
Department of Statistics
Stanford University
Department of Mathematics Stanford University candes@stanf ord. edu
ccheriansstanfor edu
Abstract
We develop new conformal inference methods for obtaining validity guarantees on the output of large language models (LLMs). Prior work in conformal language modeling identifies subset the text that satisfies a high-probabillty guarantee of correctness. These methods work by filtering claims from the LLM's original response a scoring function evaluated on the claim fails to exceed a threshold suffer calibrated via split conformal prediction. Existing methods in this from two deficiencies. First, the guarantee stated is not conditionally valid. The trustworthiness of the filtering step may vary based on the topic of the response. Second, because the scoring function is imperfect, the filtering step can remove many valuable and accurate claims. We address both of these challenges via two new conformal methods. First, we generalize the conditional conformal procedure of Gibbs al. (2023) in order adaptively issue weaker guarantees when they are required to preserve the utility of the output. Second, we show how to sys- tematically improve the quality of the scoring function via a novel algorithm for differentiating through the conditional conformal procedure. We demonstrate the efficacy of our approach on biography and medical question-answeriig datasets.
Introduction
Large language models (LLMs) are breakthrough in machine learning. In addition their ex- traordinary performance natural language processing benchmarks, LLMs such ChatGPT and Gemini are now used by hundreds of millions of users around the world [24]]. But even though these models match or even surpass human performance on an increasingly complex and diverse set of tasks, their reliability remains in doubt. For example, LLMs often confidently hallucinate facts that do not exist, and can generate toxic outputs that may offend or discriminate [[2]] This "mis- alignment" between user goals and model behavior hinders LLM deployment in settings where the potential for AI assistance appears highest, e.g., legal work customer service interaction [EI].
Since an LLM output not always trustworthy, growing body of work aims to quantify uncertainty regarding given output's 's validity. While there are many approaches to this problem [28, g. 9, [18]. this paper considers particularly popular method for black-box uncertainty quantification: con- formal inference [30, 2. 3]. Conformal inference provides generic methodology for transforming predictions of any modeling procedure into valid prediction sets that are guaranteed to contain the true outcome with high probability. Several recent papers have applied conformal inference to define set L.LM responses that contains at least one factual response with high probability E. I 25. [2 But while generating candidate set of outputs may be reasonable strategy in some question-answerring -nswering problems, is not a generalizable approach for the diverse and unstructured tasks faced in real-world deployment.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Output
Conformal Factuality
Our Method
Fixed level: 90%
Adaptive level: 83%
The shingles vaccine typically
shingles vaceine typically recommended for adults aged $2 and older- The
The shingles vace peccine typically recommended for adults aged 50
The for aged 50 given
given with the socand-deed
The with
the nonths after the first dese. It is currently
dose
advinistered
rontths-ssaeee the siest 40se, I2 44 succensly
adninistered 2
adninistered
sunths after the
first dose. st 56 cucsentiyy
recomended that individuals rece lve the shingles vaccine once in their inc. howaver, it is always best to consult for persona recommendations.
recorendedd ttht individdntts rereevivee she chinshin vaccane once-an------ in-tnoac
recoreended that Chat--ine individuol receive the sin-ssinnper--ae vaccine anca their Lifatize. Howeeer, 18 always best consult with for persona lized recomendot.onn
Howneer, St alaays best so--omeult
hea ithcaee provider
healshcara proveeee for personalized recommendations.
Lthcare provider
Figure The left panel displays the output of GPT-3.5-Turbo for the prompt "How often is shin- gles vaccine required?" The first filtered output (center) calibrated using the frequency score (see Appendix [EI) and the marginally valid conformal factuality method of Mohri and Hashimoto [211]] at a fixed level of 90%. The second filtered output (right) is calibrated using a score obtained via our conditional boosting procedure (Section ) at a level of 63%, which is chosen and calibrated using our adaptive method (Section (22 to approximately ensure that at least 70% of the claims are retained. Both filtered outputs are guaranteed to include no false claims with the stated probability.
More recently, Mohri and Hashimoto | propose to forgo sets of LLM outputs and instead utilize conformal inference to filter out invalid components of the LLM response. At a high level, given an LLM generation parsed into set of distinct sub-claims, their method censors all sub-claims for which a pre-defined scoring function lies below some threshold. Mohri and Hashimoto [2I] then show how calibrate this threshold such that the retained claims are factual with high probability.
While these methods represent a promising step towards usable guarantees for LLM outputs, they are not yet practical. One limitation is that the guarantee attained by previous methods only holds marginally over a random test prompt. The true prohability of output correctness may then vary substantially based on the prompt's characteristics. For example, we show in Section !| that the probability of output correctness (even after applying the conformal factuality method) is substan- tially lower for responses whose subjects are likely to be underrepresented in the model's training corpus. Second, existing methods remove too many claims to be practically useful. Recall that we remove sub-claims for which some pre-defined score falls below calibrated threshold. If this score is perfect, only false claims will be censored. In practice, however, these scores are only weakly correlated with the ground truth. As Figure II demonstrates, high probability factuality guarantee can require the removal of a significant proportion of the generated text." !I The conformal guarantee is not useful if the filtered response has limited value for the end-user.
1.1 Summary of contributions
In this subsection, we will preview and summarize our results. A more complete description of our theory and experimental setup is deferred to Sections C and [
As in prior literature on conformal language modeling, we will assume the existence of an annotated calibration set of i.i.d. coompt-repponse-clain annotation tor C, is obtained by using an LLM to parse the response into a list of scorable sub-claims, while W; might correspond to human verification of the underlying factuality of each claim. To simplify notation, we will refer to these tuples using the shorthand, D:
-aamtmatoooooooooon tuples. (P,,, R, C,, W.) The vec-
At first glance, the twin goals we have outlined for this paper, improved conditional validity and enhanced quality of filtered outputs, appear to be irreconcilable. Indeed, prior work establishes that precise conditional guarantees in black-box uncertainty quantification require larger prediction set sizes, i.e., smaller filtered outputs [S. 29].. We contribute two methods to mitigate this trade-off, thus enabling the practical application of conformal prediction to LLMs.
Our first method, which we call conditional boosting, allows for the automated discovery of supe- rior claim scoring functions via differentiation through the conditional conformal algorithm of Gibbs et al. [[ Automated conformal score improvement was introduced by Stutz. et al. [27]]; their paper shows how minimize conformal prediction set size in classification setting by differentiating through the marginally valid split conformal algorithm. As we show, however, in Section M. optimiz-
'We note that when conformal prediction is applied to more typical supervised learning tasks, this problem is equivalent to the challenge of prediction set "inefficiency." i.e. large prediction set size.
2

---

Contormal Factuslty
Condit Boosing
Conditional Boosting Levellcappiv
100
% Claimss Retained
Realized level
D0
0.70
0.65
40
0.60
20
0.55
0.50
0.5
0.5
0.8
HeaMSSearchOA K-OA Golden
K.QA Siver Dataset
LiveQA
MedicatenQA
Nominal level
Figure Empirical demonstration of our methods. The panels display results for our conditional boosting and level-adaptive methods. We aim to issue outputs with factual errors, and for the latter method, we choose the level with the objective of retaining at least 70% of the original claims in the prompt. The left panel compares the binned nominal probabilities of factuality reported by our method against the realized probability of factuality for data points belonging to each bin. These probabilities estimated using 500 test points over 100 calibration-test splits. The plotted bins, which are also given inputs to method, are [0.5. 0.55], [0.55, 0.6),...,, (0.8, right-hand panel displays the claim retention obtained with unboosted scores (blue). boosted scores (orange), and boosted scores level-adaptive CP (green). The first methods implemented a fixed error rate of a calibration test splits with each containing 2354 calibration points 500 test points.
0.85]. Finally, the
Boxplots this panel show the distribution retained claims for 100
ing the score function subject only to marginal coverage constraint can lead to poor conditional properties.
Optimizing through the conditional conformal algorithm is not straightforward. Our key technical contributions are a proof that (under mild assumptions) the cutoff output by the conditional confor- mal method is differentiable and a computationally efficient method for computing this derivative. By running gradient descent using this algorithm we discover new scores that enable greater claim retention.
The right panel of Figure 2 demonstrates the efficacy of our method. Here, learn an optimal linear combination four candidate scoring functions. We compare the learned, boosted scores (orange) against baseline method (blue) that uses the "frequency" scoring method developed by Mohri and Hashimoto [2T]. As the figure shows, the boosted score allows for higher claim retention across all datasets (mean retention of 39% 24% for the boosted unboosted scores).
boosting to
Our second method, which we call level-adaptive conformal prediction, allows the validity of the conformal output to depend on characteristics of the queried prompt. In our LLM experiments, we adapt the level, i.e., the claimed probability of correctness, individually to each prompt in order to ensure that issued outputs retain at least 70% of the original set of sub-claims. For example, in Figure M prompt GPT-3.5-Turbe to output a response to question from the MedicationQA dataset [Ki]. Outputting a filtered response that achieves the stated factuality criterion with probability 90% requires near-complete censorship, but by relaxing the level 63% using our method, can preserve almost the entire response.
Given that are now issuing output-adaptive probability of correctness, it is crucial that issued probability is calibrated. Calibration requires that the true probability correctness matches the issued one. For example, if a weather forecaster claims that there is 70% chance of rain, their forecast calibrated if actually rains for 70% of the days on which a 70% forecast is issued.
Figure D displays the advantages of our approach to this problem. First, the left panel of Figure D verifies that the level-adaptive probabilities we report are empirically well-calibrated. Second, right panel of Figure E quantitatively demonstrates the improved claim retention of our method and verifies that for each dataset included in the MedLFQA benchmark [13] our level-adaptive confor- mal prediction retains least 70% of the original output's claims in most examples. Finally, by

---

and conditional boosting methods, we retain most claims and output non-trivial guarantees of response factuality; the left panel shows that the issued probabilities vary between 50 and 85%. By contrast, while the fixed level method guarantees a 90% probability of correctness, the method retains very little of the original LLM output.
combining our
To emphasize that these results are accompanied by formal guarantees, we preview one instantiation of our theory here. Since it well-known that exact conditional guarantees in conformal inference are impossible to achieve without strong distributional assumptions [S], 29]], we present an inter- pretable alternative: group-conditional calibration. For example, in this dataset, we might group questions by medical area or data provenance; we would then hope to show that across health condi- tions or data sources, the claimed probability of factuality matches the true probability of factuality.
Equation (L), which follows from Theorem G.2 presents one guarantee that our method can satisfy, Here, denote the (random) output of our data-adaptive level function by 0x+1 and our filtered set of claims by F(Cn-1). Our method then satisfies the following guarantee simultaneously over groups G E G (e.g., prompt topic, data provenance) and some discretization of [0, 1] given by the sub-intervals (e.g., all sub-intervals with endpoints belonging to {0, ..,,0....,11.... .1).
P (((C++) is factual 0x++1 E I, E EG) E[an+1 | an+1 E E E G].
(1)
More concretely, (II) establishes that the issued probabilities are well-calibrated in the following sense: among similar prompts, the outputs that we claim be factually correct with probability, say, between 70 and 80% will be actually factual between 70 and 80% of the time. In Section B.ll we show how our framework can be adapted guarantee that the LLM's response satisfies other alignment targets beyond factual accuracy.
The remainder of the paper is outlined as follows. In Section 21 introduce the formal notation of our paper and contextualize our approach by reviewing related work in conformal inference. Sec- tion then presents our methodology for conformal language modeling. We first generalize the conditional conformal procedure of Gibbs et al. [TT]] to obtain high-probability control of arbi- trary monotone risks. We then state and give intuition for the key technical results underpinning our level-adaptive and boosting methods. Section A outlines synthetic experiments displaying the improvements of our approach over existing methods, gives more detailed description of the ex- periment presented in Figure @ above, and presents another experiment that filters short biographies output by an LLM.
We release a filtered version of the McdL.QA benchmark that removes some non-health-rrelted prompts, the generated and parsed text used to run our experiments, as well as the notebooks used to produce the figures in this paper at hathbb.cmmiiieerrn/ooooooaaaaaaa
We also update our Python package for conditional conformal inference support level-adaptive conformal prediction. This package is available to download at oethlcoommm/ciiicaaaooooooiooooo and can be installed from PyPI.
2
Background and related work
2.1 Conformal prediction with conditional guarantees
Split conformal prediction provides generic procedure for transforming the outputs of any black- box model into valid prediction sets [x EnT. Let {(X,, (X),,))) ) be a set of covariate-ground truth pairs where Xn+1 denotes a test value for which we would like to output a response. Then, split conformal outputs a prediction set C( such that
IY E (((x+1)) =1
(2)
for any user-specified value a E (0, 1).
While powerful, the guarantee given in (z) only holds on-average over the test value. Critically, in the LLM context we consider, this means that methods calibrated using split conformal prediction run the risk of displaying systematically worse performance on the most safety-critical examples.
Eqqution ( closely resembles measure of calibration error known in the theoretical computer science lit- erature as multicalibration [[2]] Formally, however, multicalibration is defined conditionally on the calibration dataset, while the expectation in (M) is over both the calibration and test points.

---

This problem is addressed in Gibbs et al. [[TO], which introduces novel target for obtaining coverage conditional on covariate information. In their work, they observe that exact covariate contittecondiiionall coverage can also be expressed as marginal guarantee over any measurable function E F, i.e.,
P(Yn+1 C(x++1) Xn+1)
(lf(()+1) (1( C(Xx+1)) a))l
for all measurable f.
Since exact distribution-Fee free covariate contitiona coverage requires the analyst issue vacuous pre- diction sets [29], Gibbs et al. [I design prediction set that satisfies the same marginal guarantee over user-specified function class F, i.e.
Blf(X=+11) (I{Yn+1 E C(Xn+1)) (1 []]]=
for all f F.
(3)
To make this guarantee concrete, consider the case where F == {(1{X E G)) 3 E R for some set of subgroups G. Then, (L) is exactly equivalent to group-conditional coverage, i.e., P(Yn+1 C(Xn+1)) Xx+1 G) 1 for all E G.
To understand their construction, let ((X,)) E denote a conformity score that measures how well Y "conforms" to an estimate of its value. A typical choice in the regression setting is S(X,Y) YY XXX for some fixed regression function Ai();; in the classification setting, we might choose
Y)X,Y) = 1/#(Y X) for some estimated conditional probabilities #.
Gibbs al. [IIJ] estimate high-probabilityy upper bound for these scores by fitting augmented quantile regression in which the unknown test score, S(Xn+1, Yn+1) is replaced by an imputed value S. Formally, let la (r) == (1 a)[r]+ + air)- denote the pinball loss. Then, they define
1 La g(Xn11)), ni n+
gs argmin
(a(s,,, Y.) g(Xi))
and output the prediction given by C(Xn+1) = ty: S(Xn+1, S (s(X (Xn+1))
Since Õ(X++1) can be mildly conservative, Gibbs et al. [MTI] also define smaller randomized prediction set, Grand. (Xm+1) that achieves exact coverage. We will typically prefer to work with this
set, but defer detailed definition of its construction to the Appendix. As the following theorem
shows, this randomized set achieves the guarantee stated in (L))
Theorem 2.1 (Proposition 4 of Gibbs et al. [IC]). Let F ( : B E Rd denote any fivite x+11 dimensional linear class. Assume that ((X,, Si) are exchangeable and that solutions to (8) and its dual are computed symmetrically the input data. Then, for all F,
Bif(X (1 E (Xn+1)) - [a)]] = 0.
2.2 Conformal factuality
As discussed the introduction, prediction sets are not suitable for many LLM use-cases. As an alternative, Quach et al. [25] and Mohri and Hashimoto [2I]] use conformal inference to develop filtering methods that remove false claims from an LLM's output. We will focus specifically here on the work of Mohri and Hashimoto [21]] since this is most similar to the new methods that we will denotes the claims made in an LLM's response and W.- (W )= denotes binary variables for which Wij indicates that the claim is true.
propose in this article. Recall that C, {C
ls
Mohri and Hashimoto [2T]] aim to output a set of filtered claims, F(CA.) C C, that contains no errors with high probability, i.e.,
P (3 f(Cn+1) such that W W( = =0) S a.
(5)
To achieve this target, Mohri and Hashimoto [2I]] define a scoring function p(P., Cit) IR that takes prompt and claim input and summarizes the LLM's internal confidence in the claim. Here,
5

---

larger values of pP,,,C Ctj) Mohri and Hashimoto [
indicate that the LLM believes that C₁S is more likely to be true. Then, set
= F(C; (Cty ((P,, 2 2f];
where f is the [11 [1-o)(++1]
-qantile of the conformity scores,
a+1 S(C,, = - inf(7 VC,, F(C,T), Wy = 1}.
(6)
Mirroring the proof of split conformal prediction [23], Theorem 1 of [ shows that if {(P, R2, C₁ W are exchangeable, this method satisfies (H5)
Methods
3.1
Generalization to alternative targets
Our first contribution in this paper will be to generalize the conditional framework Gibbs al. [TO] to accommodate generic LLM alignment tasks. To do so, we extend the conformal risk control framework [G) [.5]] to provide high-prou probabiitty control of monotone loss with conditional guaran-
tees.
given loss function L(F(),), W:) that measures the qual- ity of the filtered output F(C,)) relative to the ground truth W. Our goal will be ensure that (F W W++1 S X) 2 1 - a, for some user-specified tolerance À in the prior section - was the binary indicator that F(C++1) contains a false claim. More generally, -): could measure the presence of toxic discriminatory content in the response.
More concretely, suppose that
For example,
To incorporate general losses into the conditional conformal framework of Gibbs et al. [[ we require two assumptions. First, assume that the method is always permitted to abstain from issuing a response and thus L(6, -) = 0. Second, we assume that the loss is monotone. Namely, for any sets of claims F (C,) C CC. must be the case that (C.), Wa) S ((,) W2)-
With these assumptions, we extend the calibration procedure of Gibbs et al. [M0]] follows. Let X; = X (P,. R&) denote a set features computed using the prompt and response. Matching (6). define p(P,, Ctj) to be score measuring the quality of claim Cij. We define the filtered set of claims by F(C,) F(C, f) == {Ciy p(P. Cit) 2 f]. and the conformity score via S(C,, W,) == inf(r L(F(C,:T) W.) A
Our two assumptions (monotonicity of the loss and L(0.-) 0) imply that S(C,, We) is well- defined and equal to the minimum loss-controlling threshold. Finally, we set
1
1
gs argmin
la (S(C,, W.) 9(X,)) + la g(Xn+1)),
(7)
and filter claims at the cutoff F(X.+1) = su S S (x(Xx+1)} Similar to the prediction sets of Gibbs et al. [I0I), + ((m++)) can be conservative, and, thus, we prefer to use a randomized analog Frand. (Xa+1); 1;; its formal definition can be found in Appendix Al As the following theorem shows, this randomized cutoff satisfies the desired guarantee.
B E Rd denote any finite dimensional linear class. Assume that (D. 11+ are exchangeable, that solutions to (α)) and its dual are computed symmetrically in the input data, L(--)) is monotone in its first argument, and L(D,-)
Theorem 3.1. Let F {
Then, for all e F,
(L.
Frand. (Xa+1)), W S A) (1 -))] 0.
As above, we can make this guarantee concrete by choosing F to be a linear combination of group indicators specifying the topic the prompt, iee,, F= {(1(X E T)) 1 E R191.. For any finite set of groups G, this choice of F yields the high-probability guarantee,
([T((F(C.+1;1
Wn+1)= S X E G) = 1-a -a
for all G e
Like other conformal guarantees, the result of Theorem 3. only holds marginally over the calibra- tion data, [ Nevertheless, Gibbs et [IC] observe that the cnciiratioo-condiitiin validity of their method concentrates around the nominal level as the calibration set size grows.

---

3.2 Level-adaptive conformal prediction
In addition to adapting the filtering cutoff X,+1. it may also be beneficial to adapt the desired error probability at test time. @ In this manuscript, we adjust a to ensure sufficient claim retention for each test output. In other scenarios, we might set a stricter a for high-stakes prompts and more lenient one otherwise. For now, suppose that we are given some desired level function a(-), and our goal is to ensure that L(F((n+)), Wn+1) S À with probability 1 1 a(Xn+1).
Recall that in the previous section, we use the pinball loss, (-(+) learn the a quantile of S(C,, W.). To control the eπror rate adaptively, here we will use data-dependent loss for the i-th point, (x.)).) Then, similar to the previous section, we define
gs argmin SEF TI Σ (atx Wi) - g((i))
+ la (S g(Xn+1)),
1 n +
(8)
and filter the cutoff 91a (Xx+1) sup(s S 9s(Xx+1)) As in the previous section, it is more convenient to work with a slightly smaller randomized cutoff, mad. The following theorem shows that our previous guarantee for fixed a extends this new setting.
Theorem 3.2. Under the assumptions of Theorem 2
Ma. rand. (Xn+1)), W Wn+1) - (1 ((x+)(((X))))))=-----00(--)))))) - V
Note that we recover the guarantee presented in (l) by defining
(()) = (1((() (( IEI,Ge and F= ( R*).
While this choice of function class elicits an interpretable guarantee, it can be quite large in practice. This can slow down the computation of the conformal cutoff and reduce claim retention. Alterna- tively. this class might be chosen to exactly satisfy other popular (and more efficient) approximations multicalibration such low-degree multicalibration [АЛ]]. In Appendix BM we briefly explore the consequences of choosing an insufficiently complex function class in synthetic regression example. more substantial exposition of these trade-offs can be found Gibbs et al. [[X]].
While the above theory treats a(-) as fixed, in order to ensure that the outputs of our method meet our desired quality criterion (e.g., small interval length or sufficient claim retention) we will need to learn a(-) from the data. To do this, we split our training data into two folds: one used to estimate ax-)) and the other is used to run the calibration method described above. After making these splits, ox(-) can be learned using any regression method. In our experiments, we will aim learn the smallest possible values of a(-) that meet our target quality criterion. A detailed description of our procedure for doing so is given in Appendix B
Estimating a(-)
3.3 Conditional boosting
In this section, we describe new method for automated conformity score design subject condi- tional coverage constraints. As discussed in the introduction, the goal of this procedure is to find conformity scores that when combined with our filtering method, allow the filter retain as much of the LLM output as possible while still ensuring validity. For the sake of simplicity, we will assume that we are boosting the conformity score defined in (E). Our approach, however, will generalize to any parameterized score function. We note here that the concurrent work of Kiyani et al. [[6]] presents another approach to this problem by reframing it min-max optimization task.
Let (.(.) denote a parameterized claim-scoring function that assigns a measure of confidence to each sub-claim. We run the conditional conformal method on a set of size n and optimize 0 such that the number of retained claims is maximized on a hold-out set of size m, i..,
f argmax
I {pooPate C(r+n))) 2 t}
9
AAlhhough our methods' implementations differ, we remark here that the level-adaptive guarantee is inspired by similan lar result in Zhang and Candès [3]]
7

---

Here, F denotes the filtering threshold output by the conditional conformal method on the held-out test point x Crucially, f; (0) carries a hidden dependence on 0: f; is obtained by solving a quantile regression problem on scores that depend on po(.).
There are therefore two obstacles to optimizing (H) via gradient descent. First, the indicator function is non-differentiable. To resolve this, we proceed as in Stutz et al. [27]]] and approximate the indicator using a sigmoid function. Second, it is unclear how backpropagate through (0).
Observe that for a linear function class F = { 3: 3 E RR, the regression ( by which we obtain (0) is a linear program. Using this observation, we find that (0) can be expressed as the solution to linear system of equations given by subset of the dataset that we denote by B, i.e.,
(0) ((Xn+t)) ( SB())).
(10)
Here, (xX) and (0) used to denote the matrix of features and vector of scores for the subset of the data given by B. In the linear programming literature, this subset is typically referred as the "optimal basis." " It can be explicitly identified by selecting the points which the quantile regression interpolates the scores. Then, so long as B is invariant to small perturbations of 0 and is differentiable in 6, we can obtain the derivative of (0) with respect to 9 by backpropagating through (II). Proposition [ identifies a simple condition under which this is possible.
Proposition 3.1. Let (0) denote the non randomnddeeeed filtering threshold defined via (S). Assume that the threshold is finite and that the augmented quantile regression for all S > (0) admits unique non non- deneeeeeeeette basic solution. Let B denote this basis. Then, (0) has derivative
(f) = =b (xX++:)) T. (+( 0sSs(())
(11)
The technical condition on the basis in Proposition B.I nearly always holds; if it does not, we remark that uniqueness can always be obtained by "dithering" the scores and/or design matrix, i.c., adding small i.dd. noise, at each step of the boosting procedure [[]].
The rest of our procedure emulates the ConfTr algorithm of Stutz et al. [27]] Having set aside portion of the dataset for the final calibration procedure, we replicate the conformal prediction algo- rithm each iteration. Namely, we randomly split the remaining data into a "calibration" and "test" set. We then run the level-adaptive conformal prediction method on the calibration set, compute the quality criterion on the test set, and backpropagate on the objective given by (H). Algorithm which presents complete description of the procedure, can be found in Appendix C
4 Experiments
In this section, we present a subset of our experimental results on two previously studied benchmarks in medical mmestion-answent and biography generation. Additional experiments these data sets, as well as further validation on a synthetic dataset, can be found in Appendix DI
4.1 Medical long-form question-answerte
In this section, we summarize the experimental setup for the claims and figures presented in Sec- tion II]I Our experiment considers the long-form medical question- -asserring dataset (MedLFQA) published in Jeong et al. [[3]] It combines several previously established benchmarks in the med- ical question-answering literature: HealthSearch (n 3047), K-QA (n 1077), LiveQA (n = 100) and MedicationQA (n = 627). Each prompt in these datasets is also accompanied by either an LLM or human- genneated response to each question [K M. I 26]. Following prior work [TT. [ treat these responses as ground-trutte ttuuh for model evaluation. Also matching Jeong et al. [ in our plots, we distinguish between the subset of questions (n = 201) in the K-QA dataset that have gold-standard physician responses (denoted by K-QA Golden) and those questions with LLM-generated responses (denoted by K-QA Silver). To reproduce our results, our GitHub repository includes a filtered and combined dataset that removes some non-health-relatee prompts contaminating the HealthSearchQA dataset.
To obtain the dataset used in our experiment, we query GPT-3.5 with each of the prompts in the combined dataset, and then ask GPT-4o to parse these prompts into self-contained sub-claims. Then, to obtain our "ground-truth"" labels, we ask GPT-3.5-Turre to evaluate whether each claim is
8

---

substantiated by the pseudo- peeudo-ground-- truth response for that prompt. The prompts we use for each of these tasks are included in Appendix +
In all of the medical question- -nnteerin experiments displayed, we run the conditional conformal method with the following scoring function,
S(C,, W;) inf(T: F(C;; contains no unsubstantiated claims}.
To run our procedure, we must first split the dataset into two parts. We use the first split to both run the conditional boosting method and estimate a(-), while we the second split to run level- adaptive conformal prediction and evaluate our method's guarantees.
On the first split (n 1421), boost our claim score and subsequently estimate the level required for 70% claim retention. We achieve the former by optimizing a linear combination of four scores previously described in the LLM factuality literature: the frequency, self-evaluation, and ordinal scores described in Mohri and Hashimoto [24]] and the token-level probability assigned to a single- token self-evaluation [T_]] Appendix ETI describes how these claim scores computed. The function class F for which we run the conditional boosting method is defined by the linear combina- tion of several prompt-response features. For the sake of brevity, we defer a detailed description of this function class, our method for estimating ax-)) -)) and ablations of our method to Appendix D.21
On the second half of this dataset, run reve-addattive conformal prediction over the F used in the conditional boosting step augmented by indicator functions that correspond to ax-) falling in some sub-interval with endpoints lying in {0, 0.05, ................................ 1}. Finally, to produce the plots in Figure E run this step over 100 random calibration-test splits of size 2354 and 500, respectively.
4.2
Wikipedia biographies
0.95
100
100
0.90
Redized level
0.85
Clams Rotaned
-30
% Claimn Reteined
0.80
86
80
80
0.75
75
0.70
20
0.65
&
Floed Lewel Adopthve
Unboostted Scomes
0.60
Scores
Nominalleval
View Count
10
10
Freq.
Medium Group
Rare Very Rare
Figure 3. Empirical demonstration of our method on the Wikipedia biographies dataset. The left two panels display results for our level-adaptiv method, which aims issue biographies with or fewer errors, while retaining at least 80% the original claims in the prompt. The left panel compares the binned nominal probabilities reported hy our method with bin width 2.5% against the true realized em- pirical values evaluated over 381 test points and 100 calibration-test splits. The center panel compares the number of claims retained by our method (orange) against the fixed level method of Mohri and Hashimoto [M]] (bluc). In this plot, the y-axis displays a moving average of the number of claims re- tained with window size 1000, while the r-axis moving average of the number of views (in Jan. 2023) of the Wikipedia article associated with each prompt on the log-scale. Finally, the right-hand panel dis- plays the claim retention obtained with boosted (orange) and unboosted (blue) scores at fixed level of 0.1. Boxplots in this panel show the distribution of retentions for 100 calibration-tes splits each containing 7246 calibration points and 381 test points.
Our second experiment reconsiders the biography dataset analyzed in Mohri and Hashimoto [2T]. We sample 8516 names from Wikipedia and query GPT-3.5-Turbo with the prompt "Write me a short biography of [NAME]." We then ask GPT-4o to parse the prompts into self-contained sub- claims. While the prior work only annotates 25 biographies, we need to scale up this experiment to validate our proposed methods. To get around expensive human annotation, we use variant of the FActscore procedure developed by Min et al. [20]. Their method includes relevant Wikipedia passages identified by the BM25 ranking function in the prompt to the LLM and asks the claims are supported. Scoring of the LLM outputs (i.e., computation of p (P., Cij)) is done using the fre- quency scoring method in displayed figures except for the boosting comparison (the right panel of Figure G). In that plot, compare the claim retention achieved by optimal ensemble of

---

100
Saln confermal
Con donal Conformaa
0.30
% Clalms flttiiedd
Miscoverage
0.25
0.20
70
6.15
6.10
0.05
0.09
Marigna
Very Tree.
Freq.
Nedium
Rane
Veer Bare
Very freq.
Free.
Medium
Rare
Very Parre
Figure 4. Comparison of the split conformal calibration method of Mohri and Hashimoto [2]]] (blue) against our conditional conformal method (orange). The left and right panels displays the miscoverage and percentage of claims retained by the two methods against the number of views received by the Wikipedia pages in January 2023. The displayed boxplots are computed over 200 trials in which we run both methods on a calibration set of 5890 points and evaluate their coverage on a test set of size 2500. The displayed groups correspond to view counts that are binned into the intervals (-oc, oo), [1000000, oo). [100000, 1000000), [1000, 100000), [100. 1000). 00. 100). The fraction of the data set helonging to each group (in plotted order) is 8.7%, 30.9%, 39.5%. 19.3%, and 1.5%, respectively.
three cheaper-to-compute scores (seff-evaluation, ordinal, log-probability) to the retention of a uni- form mixture of those scores [2I1, [e]. A detailed description of the claim scoring and data collection methods mentioned this paragraph can he found Appendices E.J and Ei2 Method ablations and experiments comparing the efficacy of each claim scoring approach are included in Appendix D
In all of the experiments displayed here, we run the conditional conformal method with the scoring function given by
S(C,, W.) = =nf(== ((C::T) contains at most 3 false claims).
Empirically, we find that the claim scoring methods we apply are not as well-correlatdd with factu- ality as they were for MedLFQA. As result, in order issue non-trivial guarantees, i.e., to ensure both that 1 - a(Xn+1) remains reasonably large and that the method does not filter too much of the response, we allow the filter keep up three false claims. Experimental results for the more typical O-error guarantee are also included in Appendix D.3. For the sake of avoiding redundancy with the previous subsection, we defer the remaining experimental details to Appendix D
Experimental results displaying the effectiveness of our level-adaptive and conditional boosting pro- cedures this dataset can be found in Figure RL In Figure HI we also provide additional comparisons contrasting the conditional conformal method of Section I.T with the marginal method proposed in Mohri and Hashimoto [2I]]. As anticipated by our theory, find that method provides accurate coverage regardless of the popularity of the Wikipedia page, while the split conformal method of Mohri and Hashimoto [pII] gives variable results (left panel). As the right panel of the figure shows, this conditional accuracy is obtained by retaining more claims for frequently viewed topics and less claims for rare topies on which the LLM more uncertain.
5 Limitations
While we believe the described methodology can improve LLM factuality, we highlight some limita- tions here. First, our theoretical results assume that the prompt-response tuples i.i.d. In settings where user interactions change over time, the test prompts may be incomparable to previous prompts. Even though our framework can be applied guarantee validity under pre-specified class of co- variate shifts (see Appendix A.2 for details), robustness guarantees under other types of distribution shift will require additional research. Second, since our filter is wrapper around existing claim scores and thus the utility of our method depends on the quality of the underlying scoring algorithm. If the claim scores are weakly correlated with factuality, the filtered output will be accompanied by vacuous nominal guarantee. Nevertheless, discovering new features that accurately predict L.M hallucinations an active field of research, and our wrapper can easily leverage new approaches.
'While a non-zero error guarantee is not always desirable, it may be suitable for tasks where the cost of an error is low and/or high error rates appear unavoidable, e.g., "Please suggest ten recommendations for my vacation." or "Give me list of already- approvvdd drugs that might be repurposed to treat COVID."
10

---

Acknowledgments
E.J.C. was supported by the Office of Naval Research grant N00014-24-1-2355, the National Science Foundation grant DMS-2032014 and the Simons Foundation under award 814641. I.G. was also supported by the Office of Naval Research grant N00014-24-1-2300 and the Simons Foundation award 81 $14641, as well as additionally by the Overdeck Fellowship Fund. J.J.C. was supported by the John and Fannie Hertz Foundation. The authors are grateful to Anav Sood and Tim Morrison for helpful discussion on this work.
References
[1] A. B. Abacha, E. Agichtein, Y. Pinter, and D. Demner-Fushman, Overview of the medical question answering task at trec 2017 liveqa. In TREC, pages 1-12, 2017.
[2] A. N. Angelopoulos, S. Bates, J. Candès, M. Jordan, and L. Lei. Learn then test: Calibrat- ing predictive algorithms to achieve risk control. arXiv preprint, 2022. arXiv:2110.0105:
[3] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal risk control.
In The Twelfth International Conference on Learning Representations, 2024. URL https:
V ropenreview net/ (it.rrrmmmm--3XEEHLL-22
[4] A. Azaria and T. Mitchell. The internal state of an Ilm knows when its lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967-976, 2023.
[5] R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2): 455 482, 08 2020. ISSN 2049-8772. doi: 10. 00.109//maaiaaiaaa011.
[6] A. Ben Abacha, Y. Mrabet, M. Sharp, T. Goodwin, S. R. Shooshan, and D. Bridging the gap between consumers medication questions and trusted answers. In MEDINFO 2019, 2019.
[7] A. Charnes. Optimality and degeneracy in linear programming. Econometrica: Journal of the Econometric Sociery, pages 160-170, 1952.
[8] G. B. Dantzig and M. N. Thapa. Linear programming: Theory and extensions, volume 2. Springer, 2003.
[9] G. Detommaso, M. Bertran, R. Fogliato, and A. Roth. Multicalibration for confidence scoring in Ilms. arXiv preprint arXiv:2404.0m 04689, 2024.
[10] I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305. /26/6, 2023.
[11] Gopalan, M. P. Kim, M. A. Singhal, and S. Zhao. Low-degree multicalibration. In Confer- ence Learning Theory, pages 3193-3234. PMLR, 2022.
[12] U. Hébert-Johnson, M. Kim, O. Reingold, and G. Rothblum. Multicalibration: Calibration for the wwapputttiooaall--ddnnfff.c pages 1939-19 1948. PMLR, 2018
masses. In International Conference on Machine Learning.
[13] M. Jeong, H. Hwang, C. Yoon, T. Lee, and J. Kang. Olaph: Improving factuality in biomedical long-form form question answering. arXiv preprint arXiv:2405.12770 2024.
[14] S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207. 00227, 2022.
[15] M. Kang, N. M. Gürel, N. Yu, D. Song, and B. Li. C-RAG: Certified generation risks for augmente language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 22963-23000. PMLR, 21-27 Jul 2024. URL https: html,
//proceedigs, mlr prwss/
[16] S. Kiyani, G. Pappas, and H. Hassani. Length optimization in conformal prediction. arXiv preprint arXiv:2406.1 18814, 2024.
[17] B. Kumar, C. Lu, G. Gupta, A. Palepu, D. Bellamy, R. Raskar, and A. Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint, 2023. arXiv:2305.184000

---

[18] P. Manakul. A. Liusie, and M. J. Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303. 08896, 2023.
[19] I. Manes, N. Ronn, D. Cohen, R. I. Ber, Z. Kugler, and G. Stanovsky. K-qa: A real-world medical q&a benchmark. arXiv preprini arXiv:2401. 14493, 2024.
[20] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:230S.142511 2023.
[21] C. Mohri and Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402. 10978, 2024.
[22] D. Nadeau, M. Kroutikov, K. McNeil, and S. Baribeau. Benchmarking llama2, mistral, gemma and gpt for factuality, toxicity, bias and propensity for hallucinations. arXiv preprint 20l.:22504.0778, 2024.
[23] H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman. Inductive confidence machines for regression. In T. Elomaa, H. Mannila, and H. Toivonen, editors, Machine Learning: ECML 2002, pages 345-356, Berlin, Heidelherg, 2002. Springer Berlin Heidelberg. ISBN 978-3-540-
36755-0.
[24] J. Porter. ber 2023.
Chatgpt
active
user count reaches new milestone, Novem-
https://wwwttheverre com 2ttm/2023/11///2343/3333333 aestgpt-active-useee oww.aa/roo..oaaaaaaaaaoooeerr--- 05-20.
URL
Accessed: 2024-
[25] V. Quach, A. Fisch, T. Schuster, A. Yala, H. Sohn, T.S. Jaakkola, and R. Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024. URL
[26] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, Wei, H. W. Chung, N. Scales, A. Tanwani,
H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620 (7972):172-180. 2023.
[27] D. Stutz, K. D. Dvijotham, A. T. Cemgil, and A. Doucet. Learning optimal conformal classi- fiers. In International Conference on Learning Representations, 2021.
[28] N. Varshney, W. Yao, H. Zhang. Chen, and D. Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation. arXiv preprint arXiv:2307.0m 03997,, 2023.
[29] V. Vovk. Conditional validity of inductive conformal predictors. In S. C. H. Hoi and W. Bun- tine, editors, Proceedings of the Asian Conference on Machine Learning, volume 25 of Pro- ceedings of Machine Learning Research, pages 475-490, Singapore Management University. Singapore. 04-06 Nov 2012. PMLR.
[30] V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World. Springer- Verlag, Berlin, Heidelberg, 2005. ISBN 0387001522.
[31] B. Weiser. Avianca airline lawsuit uses chatgpt, May 2023. URL https: / www .ytimms.. 2omm//2023/66/2////rggoooo////// wwwpiioo//aviniaaaaaaaaaaaaaaaaa -ww.iii---eetttt-aaaaeeeee-- 2024-05-20.
html. Accessed:
[32] F. Ye, M. Yang, J. Pang, L. Wang, D. F. Wong, E. Yilmaz, S. Shi, and Z. Tu. Benchmarking Ilms via uncertainty quantification, 2024. arXiv:2401112794.
[33] Y. Zhang and E. J. Candès. Posterior conformal prediction. arXiv preprint 2xXi:2249.119712 2024.
12

---

A Additional details about conditional conformal
In addition to the methods discussed in Section 2.1. Gibbs et al. [M0]] also consider extensions of Theorem 2. to other function classes and regularized quantile regression procedures. In this sec- tion, show how all of these results can be applied to our present setting. We then describe the randomization procedure that is used to construct our filtering threshold, â. rand. (Xn+1) and prove Theorems [ and 3
Following Gibbs et al. [TO]] let J. be vector space and P extend our procedure to such function classes and regularizers, we consider regressions of the form
-
R a convex penalty. Then, to
1 Σ (S(X) W.) wx)))(X,))++ n+1 [
gs argmin BEF VE +
(12)
Following our work in the main text, this leads to the filtering threshold ñ.. (Xn-.) = max S S (x(x)1)) As discussed earlier, this definition of ñaa (Xm+1) can be slightly conservative and thus it can be preferrable to work with randomized analog. We derive this analog now. First, note that (Il2) is equivalent to the program
n+1
minimize P.qER gEF
Σ (1 - +X + 1)P(g),
subject to
S(C,, W.) ((x) p + 0, v1 < sisn.
S g xn+)) - Pn+1 1--1-
pi 2 0, V1 sin+1.
Our randomized variant of +(X,+1) will be based on the dual of this program. Following the calculations of Gibbs et al. [I0], the Lagrangian for this problem
*+1
Σ + (n 1)P(g)
(S(SC, W.) 9(X,) - Pt q:)
n+1
+ ( Pn+1 + qn-1)
Σ
(Aips + Eegi),
where x and & are constrained to be non-negative. Letting P*(n) = min ((n
E (X))) and minimizing the Lagrangian over gives
n+1
(1)a
(C, W1)+ Σ (((i-Pp)). > (G)),
and additionally minimizing over p; and produces the constraints
A; = (1 - aXX)) Ti & = aXX)) + Finally, since d & 2 these constraints are equivalent the inequalities -a(xi) m mnso a(X)). Putting this all together, we arrive at the dual formulation
maximize mER=+1
Σ MSCCC W.) + Th41 - (P),
subject to a(x)) < n < (1-a((x)) VI +isis++1.
Let 1/ denote a vector of solutions to this program. Then, the critical observation Gibbs al. [IOO] is that the above calculations give close relationship between the event S S gs (Xx+1) and the value n In particular, assuming that strong duality holds, complementary slackness tells us that > 9s (X = 11
S < gs (Xn+)))) = T = -a(X++1) and // E (-ax( X, 1,) a(x++1)) = S = gs (X,+1). The randomization scheme derived in Gibbs et al. [TO] for fixed alpha then corresponds to randomizing over the event that 5 9s (s(X,++1) x1))) In particular, following that work, here we define the randomized threshold
=
n.. rand, (Xn+1) = max{$ n < {U},
where Uni,([-a(X,) (XX))) is a random variable drawn independent of the data. The following theorem states the coverage guarantee of this method.
13

---

Theorem A.1. Assume that F is vector space and that all f.g F, the derivative of P(g- ef) exists. Assume additionally that strong duality holds between the primal and dual programs derived above and that t vp W.11) computed using an algorithm that is symmetric in the imput data. Finally, assume that {(P, R2, Xi, C₁ Wi)ez are exchangeable and that L(;,)) is monotone and such that L(W,-)= O. Then, for all f E F
BIJ(Xx+1)( (FCCt)CC
(Ca+1; N.. rand. (Xn+1)), W Wa+1) S {} (1 a(Xa+1))))
d Plos(c.... W. ef)
=
de
This theorem has two key assumptions that may appear unusual at first glance. First, have assumed that the dual solutions yp 1. are computed using algorithm that is symmetric the input data. This an extremely minor assumption and is satisfied by any standard method for solving convex programs (e.g. interior point solvers). For much more detailed exposition about efficient algorithms for computing {S nh S U}, we refer the reader to the original work of Gibbs al. [[[]]. Second, we have assumed that strong duality holds. As will see shortly, this assumption is always satisfied for the linear function classes considered in the main text. For more general vector spaces, Gibbs al. [TC]] prove that this condition holds for other common choices of F such as reproducing kernel Hilbert spaces and Lipschitz functions.
We now prove main results.
Proof [Proof of Theorem AI By Theorem of Gibbs et [IT], - mh4t non-decreasing in S. So, by the monotonicity assumption on I(.e)) and our definition of the conformity score, we have that
1LL(F(CC n2a rand (Xu+1)), Wn+1) A} + 1/mx41
S......W...)) SU}.
Thus,
E[f(Xx+1)(1 1ntii th aan (X W wn+1) S {} 111 a(xxxx)))))]
= =I (1
S S[Cn+:.Wn41) SU} - (1 - a a(Xx+1))))
EBBu (f (m Mm++1
(.C.I.WW.11). SU) - (1 - a (())))))) X X,41, SlC....W..t
-E[f(XX,+ ) m1+1
S(C..1,
For case of notation let S, = S(C,, W,), for S <is n+1. Investigating the case where S gives the Lagrangian
n+1
n+1 Σ
(1 - a(Xi))p + a(Xi) (n (gs())...) +
11
Snt1 (S. - (8...(X)) - pt + qi)
1=1
a+1
Σ 1=1
(Peps + Eig)
Now, following the calculations in the proof of Proposition 4 of Gibbs et al. [00), we find that taking a derivative along the direction + I8..1 + eʃ and applying KKT stationarity to the above gives the equation,
n+1 Σ 1=1
0 = de
(n + (yp..s.... + +ef)
Sat1 f(X.).
So, by the exchangeability of the data we have that
S(C.41 Wnts)) ((C....W.t)) - =[f((x+1)11
1
12
f(X.)
Snti
n +
1=1
d
=
(n + 1gP( + +
de
==
as desired.
14

---

A.1 Proof of Theorem (3. and Theorem 3
Theorems and from the main text can now be proven directly as corollaries of Theorem A.1
Proof [Proof of Theorem [ This result is the special case of Theorem A. in which P(-) = 0, o(X;)
constant function, and F is finiteddimensional linear class. Under this setup, the primal and dual programs outlined above are linear programs. Since the dual program admits the interior feasible solution 1y 0, Slater's condition immediately implies that these programs satisfy
strong duality.
Proof [Proof of Theorem [> This result is the special case of Theorem A. in which P(-) 0 and is finite-dimensional linear class. Similar to the previous result, the resulting primal and dual programs are linear and satisfy strong duality by Slater's condition.
A.2 Interpretation of the guarantees terms of covariate shifts
As discussed in the original work of Gibbs et al. [TC], the guarantees the loss we developed in Theorems T and 3.2 can be readily interpreted as form of robustness with respect to a class of distribution shifts affecting Px (but not Prlx) known as covariate shifts.
First, to precisely define such shift, suppose that E F is non-negative. Recall that our features Xn+1 X (Pn+1, 11+1) are functions of the prompt and response. Consider the setting in which the training data, {(,, R2, C, W.) are sampled i.i.d. from a distribution Q, while the test point sampled from the
distribution
[(Xn+i) dQ (Q(p..). (Cu+1, W Wn+1) Q(C.W)((P.R) [x]
Pu-1;Rat11)
Here, the notation re-weighted by f. e.g. using the subscript to denote the re-weighting, we have that for any set A,
Eg [x(]( dQ refer the distribution in which the covariate space
s(X++1)
Py(Kn+1 E A) [x(f(x)1 E AH [x(((X)
Then, using these definitions, our Theorems G. and have the following immediate corollaries.
Corollary A.1 (Extension of Theorem D. Let Py denote the covariate shift setting described
above. Then, under the setting and assumptions of Theorem 3
Py (L Frand (Xan 1)), Wn+1) & A) = 11 - Vf ( Eq [X(X) < 20}.
Corollary A.2 (Extension of Theorem (.2).. Let E denote expectations with respect to the covari- ale shift setting described above. Then, under the setting and assumptions of Theorem B.2
( (L Aam ran (xn+))), W +1) s A} (1- axxx+1())) ( [q(((x) < xo).
B Details of the level-adaptive procedure
B.1 Method for learning ax.-)
To make our approach learning a(-) precise, we formally define our quality criterion using binary function Q(:-)) that takes in data point and a cutoff and returns 1 or 0 if the criterion is satisfied or not, respectively. For example, for the claim retention objective described Section [.Il
F(
Q(C,
= :=
2 0.7
C
15

---

0.7
0.6
0.5
0.2
GO0DO
0.1
00
000
0.0
10a
10a
10a
10
10a
Views
Figure For hold-out set of size 124, we plot the optimal level threshold, ot claim retention (I against the number of Wikipedia page views (on the log scale) for the associated person. Letting 2, denote the view count for person i, the black line denotes the estimate of the 0.25 2.25--uantile of az | vy
obtained by regressing over the function class given by {Bo- + Σ. 3,v" B E R.
As discussed the main text, when running our method we split our training data into portion for fitting ax-)) and portion for calibrating the conformity score. Then, we further divide the por- tion of the data that used to a(-) into two folds. We use the first fold to run the conditional conformal procedure at fixed grid of quantiles, to evaluate Q( fo (X.)) for each choice of a. Here, (X,) denotes the conditional conformal cutoff evaluated for test claim C; at fixed level -
{0.01..... 0.99), and the second dataset
After collecting these data, we compute the maximum a at which the quality criterion is achieved. Formally. for each point (C,, in the dataset held out from fitting, we compute
a inf (a: vB 2 a, Q(C,, fo())) 1}.
(13)
With this data in hand, our goal now is to fit a function a()) that predicts at from af), In principle, this could be done using any number of regression algorithms. In our experiments, we choose estimate quantile of a X, using the same function class used to conformal procedure. We then truncate these estimates to lie above 0.1. See Figure 5 for a plot of this estimated quantile and the underlying data points used to fit this regression. Our choices of quantile and truncation are motivated by our desire to ensure that the adaptive level will guarantee the quality criterion for most test points without issuing completely vacuous guarantees, i.e., fitting some ox() whose range is concentrated around 1 or O.
conditional
B.2 Dependence of the level-adaptive guarantee on the function class
The quality of our level-adaptive guarantee strongly depends on the choice of function class, F. To understand this, recall that split conformal prediction corresponds to taking J to the class of constant functions, F fz B : R... In this case, Theorem 5W states that
E [[ Wn+1) S A}] E[ a (Xnn+)))]
ic. with probability E[1 - [[]++1)] the error rate of our method lies at the target level. This guarantee is extremely weak; objective is to ensure that the nominal probability is pointwise close to the true level, i.c. [P W wn+1) S X a(Xx+1)) (1 a ((Xn+1)) is small. In the empirical results in which our methods were able approximately achieve this
main text, target. These result were obtained by designing F to include functions that depend on a(X,+1)).
To demonstrate the importance of this choice of F more explicitly, Figure B compares the realized difference between the true coverage, IP(Y++1 (((n+1)) a(Xx+1)) and the nominal level
16

---

1.0
1.0
0.8-
Realized coverage
Realized coverage
0.2
0.0
0.0
0.2
0.4 0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Nominal coverage
Nominal coverage
Figure Comparison of the realized and nominal levels our level-adaptive method for various choices of F. Here, (X,, Yi) (0, I2) and we set the conformity score to be simply S(X, Y)) Y, In this experiment, we use our level-adaptive method (Section [22 in the main text) to construct prediction sets for Y; given covariates X;. We set (xX) function with temperature 1. We then run the level aeveladaptive method on a calibration set of size n 1000 and evaluate the method on test point; the plotted points (center, right) are obtained from 500 trials. The left panel shows results for F {xr + 3 E R), while the right panel displays results for F ((1,
((X) where a(-) denotes the sigmoid
axx)) TB BE ].*
B:
1 1- a(Xx+1) when F for 3 B RJ and F = ((1,((())) a(X)) E R23 on a simple synthetic dataset. As anticipated, when consists only of constant functions, the nominal guarantee is spurious, and the true probability of coverage is uncorrelated with the claimed level (left panel). This is corrected by adding a(X) to the feature space of F; despite being quite simple this choice empirically achieves the desired calibration (right panel). We leave more substantial investigation of the trade-off between complexity, empirical calibration, and efficiency to future work.
C Details of the Boosting Procedure
For the boosting procedure, we will assume that we are working with the derandomized variant of conditional conformal method of Gibbs et al. [IO]. In addition, these results will hold only for the finteeddimensional function class variant of their method with regression penalty, i.c. in the notation of Appendix A we assume that
(:()
P(-) =0
We first restate and then prove Proposition 3
Proposition C.1. Let (9) denote the non-randomized filtering threshold defined via (8). Assume that the threshold is finite and that the augmented quantile regression for all S > (0) admits unique non non-degenee den-eeeeeerate basic solution. Let B denote this basis. Then, (f) has derivative
( (x)++)) T (x(
0gSs(H)).
(14)
Proof Assume without loss of generality that i 1. We drop the subscript on è for notational convenience. To differentiate between the imputed value of the (n + 1)-st score and the vector of all observed scores, will use S to refer to the former and S to the latter. When the optimal basis is unique, we will assume that the primal solution is defined via the interpolator the basis. Otherwise, we assume it is obtained via some symmetric algorithm.
First, we establish that 0 + ((X++1) T (8) is locally linear in f for +()). First, observe that the non-interpolated points of the quantile regression correspond to the non- con-basic variables in the solution, and that their residuals are equivalent the non-basic "reduced costs" of the LP. Standard results from LP sensitivity analysis establish that when these reduced costs are bounded away from (implied by our assumptions non-degeneracy and uniqueness), the basis of the I.P unchanged under sufficiently small perturbations of the objective coefficients (i.c., for perturbations of 0) [8].
17

---

Since 3s is defined by the solution to linear system of equations involving the basis, the local constaney of the basis implies the claim of local linearity. Second, we establish that S + (0) for S > +()) is constant. This is immediate from the KKT conditions for (α)). The optimal basis is unchanged for all S > +(0).
Fix any Og We claim that there exists ball around 0g such that f(0) (XX++1) B+(00)+1
To prove this, first note that we know that for 0 sufficiently close to 0o. =(00) + 1 > ((xn+1) (1). Since S + Bs(0) is constant above +(0), this immediately implies that for > ((Xn+1))
( (1())))1 (0) (e(e))+(00) and thus S A Hence, +(0) ((Xx+1)) (-(0))+1
Now, assume by contradiction that +(0) < ((Xn+1) B (1 For ease of notation let Φ (Xx+11) (r(0))++ (0). Now, since =(0) < u must have v > ((Xn+1) (o). Recall that we additionally know that +(0) + ((xm-1) Br (00)++100) So, by our assumption that S +> (0) is constant above +()) we find that +-(00)+1(0) = (O). Thus, +(X++1) Br(())+2(0)
& (X,+1) (0). which gives the desired contradiction.
The above proves that (0) ((Xa+1) (rr (11 (0) locally in ball around 0g- The desired result follows by our previous proof of the differentiability of the right hand-side.
With this result in hand, our boosting procedure is now given in detail in Algorithm II below. In this algorithm we use the notation ((1) = [ (Xx))ee and (1) = (S,,C, W.))er to refer the feature matrix and scores for the data points in I.
Algorithm 1: Conditional boosting
Data: Boosting dataset Dooost = (D, (=1 conformity score function Se(.,)), function class basis 4, initialization 0g. sigmoid temperature X, level boosting iteration count T".
Lm
for t E [T' do
{= 0:
D1, D: = (andomSppit((Duuuure
for i D2 do
B = getOptimalBasiisa.@000 (DD)) ((1)),, ((i))):
3 ( (i))s
L = l + ox - p Ciy)):
0, Adam(e, (1,00,000000) ste)p:
return Or.
In Algorithm M above, getOptimalBasis() refers to the subroutine that computes the optimal basis B appearing in the statement of Proposition CAl for text point i.
In order to decrease computational complexity, the experiments in this paper plified version of Algorithm M. In particular, the subroutine getOptimalBasis obtains the optimal basis in Algorithm M by identifying the dim(u) points that are interpolated by the regression when > f (0). Identifying this set is computationally burdensome since each gradient step in requires computing 6 (0) (and therefore 3.) for each point in the second split. Thus, the gradient computa- the gradient can be obtained by defining B as the optimal basis from the quantile regression fit to D2 alone. This approximation, which does not account for the test-point augmentation in the conditional conformal procedure, is much faster since yields single 3 to use for all the test points. The experiments in this paper are run using this simplified procedure. To concretely modify Algorithm M. B is defined the optimal
using sim-
tion step scales quadratically in (D2). A linear-time approximation
18

---

basis of the quantile regression fit to D1 alone, the computation in the nested for loop is removed, and the estimated threshold in the sigmoid function becomes ({+))
D
Additional experiments
D.1 Synthetic data
200
15tutz at)
Conditional miscoverage
0.25
(Sturz
0.20
10.15
0.10
-80
0.05
0.00
Figure 7. Comparison of the marginal boosting procedure of Stutz et al. [27]] (blue) against ditional bonsting method (orange). The left panel shows the prediction produced by each method, while the right panel displays the conditional coverage, P(Yn-1 ((++1)) xx1) against the values the first feature, (1) Using Adam optimizer with learning rate set 0.001, the plotted scores are boosted for 500 steps on synthetic dataset of size n 1000 and evaluated on another test dataset of size n. 2000.
To demonstrate how our methods improve upon previous approaches, consider synthetic regres- sion dataset with substantial heteroskedasticity. The data in this experiment are generated by sam- pling two features x lid Unif(1, 10) and x Unif(5, 10), while the labels sampled from distribution whose variance is given by the sixth power of the first feature, i.e., Y; ~ AO (x(") We then construct prediction intervals using the score function S₃(X, = |Y1/|XT 0j.
Figure 0 compares the ConfTr algorithm of Stutz et al. [2]] against Conditional Boosting method. While the left panel shows that the former method may sometimes issue shorter intervals, these come the cost of degraded conditional validity (right panel). In particular, the prediction sets resulting from ConfTr have extremely poor coverage on the subset of data with high conditional variance. By contrast, learning using our procedure still attains near-perfect conditional coverage.
530
1.
<20
0.5
8030
Interal Mecralenngh lenghh
6032
230-
0.4
4032
2030
100
0.2
0.29
0.5
0.15
2.55
0.2
0.5
0.2 04
0.6
0.8
Nominal coverage
Nominal coverage
Nominal coverage
Figure& 8. Performance of level-adaptive method synthetic dataset. We use 2000 points estimate the adaptive ax-) function. We run the level-adaptive method on calibration set size 1000 and evaluate the method on test point; the plotted points (center. right) are obtained from 200 trials. The left panel shows the distribution test set for fixed values of a E (0.05. 0.25, 0.5, 0.75, 0.95}. The center panel displays the inter- val lengths obtained when the level a(X++1) is now chosen adaptively to ensure a maximum pre- diction set size of at most 500 (red line). Finally, the right panel compares the realized coverage P(Y++1 E C(X++1) (xX++1)) against the nominal level, 1 a(X,+1) reported by our method with F {So Σ B,x B5a(x) R*).
interval lengths obtained on the
19

---

Figure E applies the adee--ddaptive conformal prediction procedure to the same data Sataagenerating pro- cess. In this example, we use the naive absolute value score Y) |Y). Since the score is poorly chosen and the data is noisy, the resulting prediction interval can be very large. The left panel of the figure shows boxplots that display the empirical distribution of interval lengths over the test set at various fixed levels of a E {0.05, 0.25, 0.5, 0.75, 0.95]. Unsurprisingly, at most fixed levels for a, many intervals are substantially larger than the maximum target length of 500. This is corrected when we fit using adaptive level, ((X,) (center panel), which is estimated to ensure that the interval lengths fall below 500. Finally, the right-hand panel verifies that the guarantee of Theorem 322 is accurate. The issued values of a(Xn+1) closely track the true coverage probabilities; note that the latter can be computed in closed-fore fossed-fommmo for our data gentaaanng process.
D.2 Additional results for Section A
Additional details for the experiment set-up The function class F is defined by the linear com- bination of intercept, the number of characters in the prompt, the number of characters in the response, the mean frequency score assigned to the claims, the standard deviation of the frequency scores assigned the claims, and group-indicators corresponding the source dataset. The results shown in Figure Q are obtained by running the conditional boosting algorithm for 1000 steps using the Adam optimizer with learning rate set to 0.001.
We estimate a(.) by dividing the first split of the data into two further sub-parts, On the first half of this further subdivisionx (n 720), run the fixed-level conditional conformal procedure (with the same F used to boost the scores) for each a in a grid of 50 evenly spaced values between 0 and 1. Using the second half of this dataset, We record the smallest a for which at least 70% of claims are preserved at all larger values of a. We then obtain our estimate of ax-)) by fitting 0.85-quantile regression over F this dataset. For interpretability and numerical stability in the next step, the final output of the function is truncated to lie between 0.1 and 0.5.
evaluate the percentage of retained claims at each
Additional experiments Figure decomposes the effect of each method in this paper claim retention and calibration for the MedLFQA benchmark. For a fixed-level guarantee, we observe that the boosting method substantially improves claim retention. When we adapt the level to guar- antee high claim retention, the improvement of the boosting method is visible in the second panel of Figure H. Namely, we find that when the level-adaptive method is augmented with boosted scores we obtain the same claim retention while issuing stronger guarantees as compared to the un-boosted scores. Finally, the third panel of Figure verifies that our reported values of a(X,+1) (approx- imately) match the realized error rates.
Using the same experimental set-up as the ablations, Figure III compares the effect of the choice of claim scoring method on the nominal guarantee offered by the level-adaptive procedure. Note that the proobbbbbaylity and frequency claim scores appear to offer the strongest set of guarantees. How- ever, the markedly lower level of claim retention for the log-probabilin score make the guarantees difficult directly compare.
D.3 Additional results for Section
Additional details for the experiment set-up To define features for the conditional conformal function class, we obtain page metadata from the Wikidata API. The metadata we use is the number of views of the biography's 's Wikipedia article in January 2023. Since this data is quite skeht- the view counts for the most popular articles can cause numerical issues in our optimizer. Conse- quently, we trim the largest entries to the 0.95-quantile of the view data.
Here, we also provide some additional details regarding the function classes used in our biography experiments. all examples, we aim to provide uniform coverage guarantees regardless of the underlying popularity of the topic. In the fixed-level filtered biographies, we run the conditional conformal method with F = {Bu + Σi the corresponding Wikipedia page in January 2023. In the level-adaptive examples, we use a nearly identical workflow to the previous section. The only difference is that here we estimate ax-) using 0. 75-quantile regression over (see Figure 5 for a plot of this estimate). Additionally, in the final level-adaptive fit, we add linear term to this function class given by Ba(a(-) 0.1)2. Note that in
= 3,0² 8 E IR,, where denotes the number of views of
20

---

Boosting
Condl Bocoting
Boozting Adladivc
Conditional Boosting
Lovil Adopthre
697
Roallpo Lape
KC4 Cuen
02
058 0.0 de Oas
Figure 9. The left panel of figure shows percentage claims retained by various methods on the McdLFQA benchmark Jeong et al. (2024). This benchmark contains many datasets each which is displayed on the x-axis, Before boosting, we define our claim score by an equally- weighted ensemble of previously published claim scoring functions. We run 100 trials in which we resample a split (n = 1441), calibration split (n = 2354), and test split (n 500). We plot 4 filtering methods: the fixed-level (1 the function class given by dataset indicators and prompt metadata (prompt length, response length, as well as the mean and standard deviation of the "log probability" claim score over the output) (blue), the same conditionally valid method with adaptive level (Xx.+> (orange), the conditionally valid method with boosted scores (green), and combination method that incorporates both andiitionllboostiinn and level- adel-vdae CP (red). All methods are set-up ensure that the final output contains 0 false claims. The middle panel shows that boosting allows the level-adaptive procedure to issue guarantees with higher confidence. The right panel verifies that reported nominal levels af (x++1) match the empirical crror frequencies over equi- squi-spaced bins width 0.05; these bin indicators are included the function class used in the level-adaptive procedure.
0.9) method that guarantees conditional validity over
Reequetcy
Sei Snseestiation
Inpprosshilt
Ordinal
St CCuuma Ritaneed
Datasst
Nommnallval
Figure 10. The sct-up is identical to that of Figure S Here we do not consider boosted scores, but display the performance of the level-adaptive method for each claim scoring approach in isolation.
experiments, truncate (x)- to lie above 0.1. As result, substantial proportion of outputs receive that estimated quantile and thus, (a(-) - 0.1)² correlates with the more variable portion of the quantile function.
Though the first two panels of Figure B and Figure E are run using the more performant frequency score, the right panel of Figure B is obtained by finding an optimal linear combination of the three cheaper-to-obtain claim scores (self evaluation, token probability, ordinal). We obtain the displayed result by running 1000 steps of Adam (with PyTorch defaults, i.c.. learning rate 0.001) through the conditional conformal procedure given by the same FF described in the previous paragraph.
Additional experiments Figures DD and I decompose the effect of each method on claim reten- tion and calibration for the FActscore benchmark. Notably, we observe that optimally ensembling claim scores using conditional boosting improves claim retention and/or shifts the distribution of issued guarantees to the right. Furthermore, regardless of the guarantee issued, the third panel of both these figures confirms that our method well-calibrated. Finally, as the second panel of Figure I confirms, if we aim to guarantee 0 false claims in the final output, is unlikely that we issue nominal probability above 50%. This motivates our choice to instead guarantee that the filtered output contains at most three false claims.
the 3 theaper-to-compute
21

---

Boosting
Boosinn
Booszing Adapeiv
Boooding
Roal 0oallsooi Love)
046
Trequency
Figure This figure replicates previous ablation study (namely Figure i for the FActscore datasct of Min et al. [20]] For the sake of brevity, we only describe the differ- ences compared to Figure & The frequency
each biography topic varies; the displayed groups correspond to Wikipedia view counts in Jan. 2023 that are binned into the intervals (1000000, oo). [100000, 1000000), [1000, 100000). [100, 1000). [0, 100). The function class used for calibration is defined by the linear combination of these bin indicators and the view count of each Wikipedia article. We run 100 trials in which we resamplc calibration split 5338), and test split (n 500). All methods are set-up ensure that (with high probability) the final ouiput contains more than 3 false claims.
poosting/(---esiimattin J-estmmation split (n 847),
No Bocsting
Ginnl Bocoling
LevenAdactive
Conditional Boosting Lavel-Adapitee
3.42
9.40
Frequency
Nommal lovel
Nontha level
Figure 12. The set-up identical to that of Figure M but all methods set-up ensure that (with high probability) the final output contains than false claims.
Using the experimental set-up the ablations, Figures IE and 14 compare the effect of the choice of claim scoring method on the nominal guarantee offered by the level-adaptive procedure. Note that once again the log- probability and frequency claim scores appear offer the strongest set of guarantees.
Lepprobabilty
Seyp
24
Figure 13. set-up identical of Figure I Here not consider boosted scores, but display the performance the level-adaptive method each claim scoring approach isolation.
22

---

Requoncy
Soff craliation
Log probabiity
60+
6.16
026
3.50
Figure 14. set-up identical that of Figure T Here not consider boosted scores, but display the performance the level-adaptive method each claim scoring approach isolation.
E
Additional scoring details
E.1
Claim scores
We collect several features to use for estimating a confidence level in each claim. Unless otherwise noted, the claim scoring function used in this paper is defined by the "frequency" score examined by Mohri and Hashimoto [2I]]. We compute this score by querying GPT-3.5-Turbo times at a temperature of T 5. For each claim, re-query the LLM ask if the claim is supported in the resampled generation. These scores which can be 1 for contradicted, 0 for not present, and 1 for supported are then averaged over the 5 responses. For example, claim that is present in three out of the five responses, but absent in the other two would receive a score of 0.6.
Since the frequency score requires many API queries, we also investigate several cheaper and faster- to-compute methods for claim scoring. Following Mohri and Hashimoto [2]]], we also collect the self-reported probability obtained by directly asking the LLM to report its estimated probability of claim correctness to three significant figures. We also compute the ordinal score, i.c., the order in which the claim appeared in the original response. Going beyond Mohri and Hashimoto [RI]], compute the LLM's internal probability of each claim by querying the model with the list of claims and asking it to output single-de chagle-character (T or F) assessment of its beliefs. The internal probability of the model can then be computed by exponentiating the log probability of the returned token.
E.2 Prompts
To parse the initial response into scorable sub-claims, prompt the LLM (GPT-4o, in exper- iments). After first parsing the initial response into its component sentences, we copy the method of Min et al. [2M]] (made available GitHub under MIT license) and use in-context learning to improve the parsing accuracy the LLM. We first use the BM25 ranking function to match our sentences to similar examples previously parsed by Min et al. [[0]]. We prepend these gold-standard examples as well as the new sentence to be parsed to the prompt,
Please breakdown the following sentence into independent facts: [...]
To annotate the subclaims, we either retrieve the response given in the MedLFQA dataset or emulate Min et al. [20] and retrieve Wikipedia passages that are most relevant (using the BM25 ranking function) to the subject of the biography. The prompt includes this passage followed by the previous claims in the output that have already been annotated ensure self-consistency,
Answer the question about [...] based on the given context and your previous
answers. Title: [...] Text: [...] Previous input: [...] True or False? Output: [...]
Input: [...] True or False? Output:
To evaluate the support the claims in new response, we prompt the LLM using
You will get list of claims and piece of text. For each claim, score whether the text supports, contradicts, or unrelated to the claim. Directly return jsonl, where each line is "id":[CLAIM ID], "score": (core":[CCORE]] Directly return the jsonl
23

---

with NO explanation or ANY other formatting. For the [SCORE], return 1 for supports, -1 for contradicts, and 0 for unrelated. The claims are: [...] The text
We obtain self-assessed probabilities by prompting the LLM using
You will get list of claims and the original prompt that motivated these claims. For each claim, assess the probability correctness. Directly return a jsonl. where Directly return the jsonl
each line is "Cd'I[C ID), "gptscooee: with NO explanation or ANY other formatting. For the [SCORE), return the esi- mated probability of correctness to three significant figures. The original promptis: [...] The claims are: [...]
We obtain the internal probability of the model using a very similar prompt,
You will get list of claims and the original prompt that motivated these claims. For each claim, assess the correctness. Directly return jsonl, where each line "id": [d":[CLAIM ID), nation or ANY other formatting. For the [BOOL], return "T" or "F" quotes so that it is valid json. The original prompt is: [...] The claims are: [...]
Directly return the jsonl with NO expla-
24

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction outline the three main contributions of the pa- per; 1) a general framework for using conformal calibration to meet LLM safety objectives, 2) method for adapting the calibration be valid conditional properties of the input prompt and also to adapt to a data-dependent error tolerance level, and 3) a boosting method for improving the conformity scores used in conditional calibration. The general frame- work for conformal safety in LLMs is presented in Section [.
Conditional calibration and adaptive tolerance levels are discussed in Sections [X and S supported theoretically by
Theorems [ and (.2, and empirically by synthetic data comparisons in Figures E and & and the real data comparisons in Figures α& and BL Finally, boosting the conformity scores is discussed in Section 3+ and supported empirically on synthetic data in Figure M and real data in Figure
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the current theory and methods are discussed in the Limitations section of the paper (Section 5)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
25

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The main theoretical results presented in this paper are Theorems + and 6.2 and Proposition B.I. The assumptions of these results are clearly stated in the main text and elaborated on in Appendix A and Appendix QQ Complete proofs of Theorems 3 and S can be found in Appendix and proof of Proposition B can be found in Appendix O
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed descriptions of the experiments can be found in Section El and the associated sections in the Appendix. The data and code to reproduce the experiments is available the linked Github repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, deseribing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
26

---

(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Data and code to reproduce the experiments is included in the linked GitHub repository. Additionally, detailed descriptions of the experiments can be found in Section and the associated sections of the Appendix.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: wtwiddde//odeeummsssseeees ooiiooooiisiioooooooioco for more details.
/ //nips.co wttpcc//ppppppceecceeec wwtppulli/guuideeeeeeeeooee/eeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Detailed descriptions of the experiments including information about all rele- vant settings can be found in Section ! and the associated sections in the Appendix. Code to reproduce the experiments that also contains this information is publicly available.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
27

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appropriate error bars displaying uncertainty can be seen in Figure R. In other experiments in the paper (e.g. those in Figure HD prefer to not show bars, but rather boxplots displaying the entire distribution of the values observed in the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The majority of the experiments performed in this paper Book Pro in total runtime of less than a few hours. Since this is not significant use of do
Mac-
include this information in the paper. The remaining computational burden of this paper consists of calls to the OpenAl GPT API, which again are relatively low cost and take only few hours to run. Since we cannot speak to the backend cost of running this API and the costs on our end are relatively minor, we do not include information about these computational costs in the paper.
computational resources,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code Of Ethics
28

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp://paarrg..ccppppp///llllaa
Answer: [Yes]
Justification: There is no human subject otherwise sensitive data in this article that re- quire careful consideration. Additionally, we do not anticipate any serious harmful use cases for methods.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This article is concerned with methods improving the safe and human- aligned use of LLMs. Thus, as we discuss throughout the article, we hope this paper will have positive societal impact. We do not anticipate any serious harmful use cases for our methods.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release a model or any item that has a significant potential for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The only data used in this experiment are Wikipedia pages scraped from publicly available eaches of English language Wikipedia under the CC4.0 license. We use MIT-licensed code from Min et al. [ and credit them.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license. copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com datassts has curated licenses for some datasets. Their licensing guide can help determine the li-
cense of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: This paper does not release any new assets aside from the proposal of new methodology. Code for implementing our methods and the experiments in this paper is included with the submission.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
30

---

Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "2uNMQNxFAU", "reviewer": "Reviewer_CDbE", "summary": "The paper provides a conformal prediction framework for the generation of hallucination risk-bound. The method can involve any loss function instead of the 0-1 loss as traditional conformal classification. They also provide a conditional (group) conformal prediction guarantee. They optimize the loss function with improved conformal prediction efficiency.", "questions": "Please refer to the weakness part for concrete questions and concerns.", "limitations": "discussed in Sec 5", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and well-written. The background and related work in Sec 2 can be helpful for readers not familiar with the background.\n2. Conformal prediction for LMs can provide statistical guarantee on the generation risk, which is important for universal deployment of diverse LLMs.", "weaknesses": "1. [Generalization to alternative targets]: the rigorous proof is lacking about why the prediction set provided by lines 191 and 192 can lead to the conformal prediction guarantee. In line 181-187, the paper claims some assumptions on the loss function to be used. However, how these assumptions are used to derive the final valid conformal prediction guarantee is missing.\n\n2. [Generalization to alternative targets]: conformal risk control methods [1] with a similar guarantee should be included in the discussion.\n\n[1] Angelopoulos A N, Bates S, Fisch A, et al. Conformal risk control[J]. arXiv preprint arXiv:2208.02814, 2022.\n\n3. [Generalization to alternative targets]: missing related work [2,3] on risk control of LLMs with conformal prediction\n[2] Kang, Mintong, et al. \"C-rag: Certified generation risks for retrieval-augmented language models.\" ICML 2024\n[3] Quach, Victor, et al. \"Conformal language modeling.\" ICLR 2024\n\n4. [Level-adaptive conformal prediction]: Equation (8) use different $\\alpha$ for the pinball loss (different $\\alpha_i$), which can breaks the pinball loss formulation in Equation 4 and might also deteriorate the original conclusion of prediction coverage of $1-\\alpha$. Do the authors have any justifications on it?\n\n5. [Level-adaptive conformal prediction]: I think the designed level-adaptive conformal prediction is not that effective. If our goal is to use an adaptive level $\\alpha$ for a particular example, why not directly do the conformal calibration with the particular $\\alpha$ so that we can get the desired coverage?  I am confused on the final goal of the method in Sec 3.2. What the adaptivity here different from what I am thinking?\n\n6. [Conditional boosting]: The conformal training method (reference [18] in the paper) optimizes $\\theta$ in an end-to-end way, but the paper claims that it is unclear how to backpropagate through the estimated cutoff in Line 236. What makes the difference in the scenario?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and constructive feedback. In response to the reviewer's specific comments:\n\n1) The proof of this result is given in Appendix A. Our assumption that $L(\\emptyset, \\cdot) = 0$ is necessary to ensure the score function, $S(\\mathbf{C}_i, \\mathbf{W}_i)$ given on line 191 is well-defined. On the other hand, our assumption that the loss is monotone in the filter is necessary to derive an equivalence between loss control and the dual variable whose distribution is analyzed in the proof of Theorem A.1. As stated in Appendix A.1, Theorem 3.1 then follows as a corollary of Theorem A.1. In particular, note that on Line 423 in the proof of Theorem A.1 we explicitly utilize our monotonicity assumption. In our revision we will add additional clarification to the main text of the manuscript discussing why these assumptions are necessary.\n\n2) The \"Conformal Risk Control\" article of Angelopoulos et al. (2022) is referenced in our discussion of related work (see line 32). The approach taken to quantifying the uncertainty of LLM outputs taken in Angelopoulos et al. (2022) is quite different from our work and so we have not given an in-depth comparison. Nevertheless, we agree that the guarantees and high-level approach taken there are similar to our work and we will look to clarify this connection further in our revision.\n\n3) We have referenced Quach et al. (2024) in our discussion of related work (see line 32). We thank the reviewer for bringing Kang et al. to our attention and will add this article to our discussion of related work in our revision. Like Quach et al., Kang et al. analyzes how many LLM generations are required to control a monotone risk. While the conformal methods used are related, we would emphasize that our approaches are both quantitatively and qualitatively different, i.e., we filter a single generation to obtain high-probability control of the loss.\n\n4) Our goal is to adapt the coverage level of our method to the underlying difficulty of filtering falsehoods out of each example. To do this, we fit an adaptive function, $\\alpha(X)$ in order to ensure that running our conformal method at level $1-\\alpha(X_{n+1})$ will preserve at least a user-specified fraction of the LLM response. Then, we run an augmented quantile regression-type procedure in which data point $i$ is assigned loss $\\ell_{\\alpha(X_i)}(S(\\mathbf{C}_i,\\mathbf{W}_i) - g(X_i))$. \n\n5) We are not entirely sure what exact alternative method the reviewer has in mind. Perhaps the reviewer is asking why we cannot simply use the value $\\alpha^*(X_{n+1})$ such that when we run split conformal at this exact level, the desired percentage of claims are retained. Alternatively, perhaps the reviewer is interested in the method where we use $\\alpha(X_{n+1})$ as a plug-in everywhere so that data point $i$ receives loss $\\ell_{\\alpha(X_{n+1})}(S(\\mathbf{C}_i,\\mathbf{W}_i) - g(X_i))$. Unfortunately, both these methods can incur selection bias and neither is valid in general. On a technical level, both of these approaches treat the test data point asymmetrically to the training data. This breaks the exchangeability of the data and thus does not prove a validity guarantee. On the other hand, our method, which indeed modifies the pinball loss, maintains exchangeability of the data and thus guarantees control of the loss. This result is rigorously proven in Theorem A.1. \n\n6) The conformal training method in Stutz et al. (2021) backpropagates through the split conformal cutoff. Since the split conformal cutoff is defined to be a quantile of the calibration set, the value of cutoff is actually a particular calibration set score, i.e., $\\text{cutoff} = S_{\\lceil (n + 1) (1 - \\alpha) \\rceil}$. Differentiating with respect to some $\\lambda$ that parameterizes the scoring function is then relatively straightforward as one may simply examine what happens to the score $S_{\\lceil (n + 1) (1 - \\alpha) \\rceil}$ in isolation. It is worth noting that Stutz et al. (2021) use a \"differentiable sorting\" algorithm to obtain smoother gradients. By contrast, our cutoff is defined by the function of a solution to an optimization problem. The derivative of such an optimum is typically obtained via a second-order Taylor approximation (cf. influence functions). However, in our case the objective of the optimization problem is not differentiable and thus we cannot apply such an approximation. As we show in Proposition 3.1, this technical challenge can be overcome; despite the non-differentiability, we prove that the optimum remains locally linear in the parameter of interest.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u9E7rXTnXb", "reviewer": "Reviewer_9DaN", "summary": "The paper presents two new conformal inference methods for obtaining validity guarantees on large language model (LLM) outputs. The authors consider the task of filtering invalid claims from LLM responses to ensure high probability factuality guarantees on the filtered output. They discuss two limitations of existing approaches and how their methods improve upon them – (i) previous methods provide a marginal guarantee over a random test prompt instead of conditional validity, and (ii) existing methods may remove too many claims as the score function is imperfect. To improve upon these, the authors generalize the conditional conformal framework of [1] to adapt the filtering threshold that controls monotone risks and adjust the error rate adaptively (level-adaptive conformal prediction). The second method introduced in the paper termed conditional boosting enables automated discovery of superior claim scoring functions via differentiation through the conditional conformal algorithm i.e., finding conformity scores that allow retaining more claims in the LLM output while also ensuring validity. The authors perform experiments on synthetic and real-world datasets to demonstrate the performance of their methods and improvement upon existing approaches.\n&nbsp;\n&nbsp;\n\n[1] I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.", "questions": "1. It seems that both methods require an additional step of data splitting. Would this introduce high variance if there are not enough samples in some subgroups? (it would also be helpful to mention the number of samples in each group e.g. in Fig 2, where there are just 381 total test points)\n\n2. Is there any specific reason why experiments were not performed on other real datasets studied in Mohri and Hashimoto?\n\n3. Typo: shouldn't it be $Y_{n+1}$ in Theorem 2.1?", "limitations": "The authors discuss limitations in the Limitations section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The main novel contribution is the conditional boosting procedure that allows automatic discovery of new scoring functions that enable higher claim retention via differentiation through the conditional conformal algorithm. While the paper leverages the conditional conformal procedure in Gibbs et al [1] and conformal factuality framework from Mohri and Hashimoto [2], the generalization of [1] to incorporate general losses and adaptive error rate is a novel and important contribution. To add, the previous work is adequately cited in the paper.\n\n\n**Quality**: The submission is technically sound and the claims are well supported by theory. The experimental results empirically demonstrate the performance improvements of the methods introduced over previous approaches.\n\n\n**Clarity**: The submission is well-written and clear for the most part. I appreciate the introduction to conditional conformal procedure of Gibbs et al. [1] in section 2.1. It would be helpful to organize the empirical analysis better to understand the setup and make performance improvements offered by different methods more clear.\n\n\n**Significance**: The conformal inference methods proposed in the paper provide practically useful guarantees for LLM outputs. I believe the conditional validity formulation and evaluation are important contributions and the results will be of interest to the community.\n\n\n**References**\n\n[1]  I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.\n&nbsp;\n\n[2] C. Mohri and T. Hashimoto. Language models with conformal factuality guarantees.  arXiv preprint arXiv:2402.10978, 2024.", "weaknesses": "The main weaknesses and scope for improvement are wrt the empirical evaluation:\n1. Lack of ablations\n- While the experiments include comparison with the split conformal calibration method of Mohri and Hashimoto [2] for the Wikipedia biographies dataset, I did not see comparison of the level-adaptive method with the fixed level conditional conformal procedure in [1] (where $\\mathcal{F}$ can probably just be a linear combination of group indicators?). I believe this would also help understand the contribution of the level-adaptive method.\n- How do different sub-claim scoring functions compare to each other in practice?\n\n2.  Lack of justification behind the choice of score functions and function class $\\mathcal{F}$\n- Why is the conformity score chosen as naive absolute score value in Fig 4, 7? How would the results change if a better score function is selected? While the validity will always hold true, the empirical performance can give better practical insights (as is the intended goal of the paper).\n\n3. Additional comment on organization of results: In its current form, it is hard to understand the experimental setup and variations of the method used for different figures. The experiments section can be organized better for greater clarity.\n\n\n**References**\n\n[1]  I. Gibbs, J. J. Cherian, and E. J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.\n&nbsp;\n\n[2] C. Mohri and T. Hashimoto. Language models with conformal factuality guarantees.  arXiv preprint arXiv:2402.10978, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We will look to prepare a revision that clarifies our experimental choices and provides additional experiments addressing the reviewer's concerns. We hope the following responses address the reviewer's concerns.\n\nWeaknesses:\n1) We agree that this would be helpful, and we performed additional experiments with the requested ablations for both the Wikipedia and new medical question-answering experiment. Discussion of these results as well as associated figures can be found in our global rebuttal to all reviewers.\n\n    In our experiments, the frequency scoring method is most effective, and the improvements obtained by our method are robust to changes in the score. In our revision, we will prepare additional figures reproducing our results for alternative sub-claim scoring approaches. \n\n2) The difference between the two methods in Figures 3, 4, and 7 may be less pronounced if a superior scoring function was selected. The goal of this experiment is not to show a comprehensive comparison, but instead to demonstrate the theoretical pitfalls of marginal boosting (i.e., conformal training). In general, if the scoring function is perfectly chosen, no boosting method (targeting marginal or conditional validity) substantially alters the performance of the method. This is also apparent in the Wikipedia example where the frequency scoring function far outperforms the other approaches and boosting simply recovers the performance of the best scoring approach. We believe that in many realistic scenarios, users do not have *a priori* access to a strong scoring function and thus, as this example demonstrates, conditional boosting can offer significant benefits over marginal boosting. This can be partially seen in our results on the Wikipedia and medical question-answering datasets where boosting produces a stronger ensemble of many weak scoring functions.\n\n3) We apologize for the lack of clarity in the experimental set-up. We will aim to prepare a revision that more clearly organizes this information. \n\nQuestions:\n\n1) The results we prove hold in expectation over the calibration set. We agree that the calibration-conditional validity of the filtering method will be more variable if the calibration set is smaller (or similarly if there are fewer samples in some subgroups). We will add additional information about the sample sizes to the figures in our revision. \n\n2) The primary reason we used the Wikipedia/FActscore dataset was because of the availability of high-quality synthetic annotations. Mohri and Hashimoto hand-label the correctness of each line of the response for $50$ examples. Unfortunately, running our method with any complex conditional guarantee on a such a small calibration set would lead to highly variable outcomes. Thus, we have chosen to generate synthetic ground-truth using GPT-4 with the original Wikipedia page passed in as additional context. Prior literature has validated that this method leads to an accurate factuality annotation for the FActscore benchmark. By contrast, we are unaware of any existing methods for obtaining synthetic ground truth on the other datasets considered by Mohri and Hashimoto. Nevertheless, we do agree that additional experimental validation would be helpful. As a result, we have also analyzed a medical long-form question answering dataset (released just prior to the submission deadline) where both expert and high-quality synthetic annotations have been made available.\n\n3) We thank the reviewer for identifying this typo and will correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RfKDGMnQB", "reviewer": "Reviewer_R2Vh", "summary": "The paper introduces two new conformal prediction methods aimed at enhancing the validity of large language models. The first method generalizes the conditional conformal procedure to issue weaker guarantees when necessary, thereby preserving utility. The second method improves the quality of the scoring function through a novel algorithm designed to differentiate the conditional conformal procedure. The proposed methods are validated using both synthetic and real-world datasets, addressing deficiencies in existing approaches by ensuring conditional validity and retaining valuable and accurate claims.", "questions": "How sensitive is the methodology's performance to variations in the size and composition of the calibration set? How representative is this dataset of real-world large language model outputs?", "limitations": "The authors adequately address the limitations of their work in Section 5.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The introduction of generalized conditional conformal procedures and improved scoring functions addresses critical gaps in current approaches.\n- The paper is technically sound, with well-supported claims and robust experimental validation. The theoretical results, particularly Theorems 3.1 and 3.2, and Proposition 3.1, are well stated and proved. The methodology is well-developed, and the experimental design effectively demonstrates the efficacy of the proposed methods.\n- The methods are evaluated on both synthetic and real-world datasets, effectively showcasing their practical utility and robustness.", "weaknesses": "- The methodology assumes the existence of an annotated calibration set of prompt-response-claim-annotation tuples, which may raise questions about the generalizability of the approach to different datasets or domains.\n- Some readers may not be familiar with the metrics used to evaluate the method. Providing additional explanations for these metrics would enhance clarity and understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive comments and feedback. We address specific concerns below: \n\n1) We agree that it is costly to obtain these tuples and it may not be feasible to do so in all cases. Nevertheless, given the large resources invested in LLM development by corporate labs, we do not believe that it is unrealistic that human-labels could be obtained for many real-world datasets of interest. Additionally, we hope that our work will help to provide a useful framework for further development in this field, and we have identified at least a few generation tasks where this methodology may prove helpful.\n\n2) We are not certain which metrics the reviewer is referring to. In our revision, we will aim to ensure that the notions of calibration and retention utilized in our work are clearly defined.\n\n3) The validity guarantee output by our method holds in expectation over the calibration set regardless of its size. However, it will be more variable conditional on the calibration set if the dataset is smaller or relevant subsets are underrepresented. In general, this method will perform best if the complexity of the factuality guarantee (i.e. the complexity of the function class $\\mathcal{F}$) is not too large relative to the size of the calibration set. This leads to a model selection question that is analogues to what one faces in many standard regression problems. A detailed discussion of these issues can be found in Gibbs, Cherian, and Candes (2023). Since this article is primarily concerned with novel improvements to the methodology of Gibbs, Cherian, and Candes (2023) that enable its application to LLMs, we do not discuss these selection details in much depth here.\n\n4) The Wikipedia dataset might appear synthetic since most people are not interested in writing biographies. Nevertheless, we believe it is still representative of the types of tasks LLMs may be used for. To help further demonstrate the range of possible applications of our method, we have performed additional experiments on a second medical question-answering dataset. We believe this example constitutes a highly relevant and realistic application for chat bots. Figures showing results from this experiment can be found in our global rebuttal to all reviewers. Overall, we find that our methods perform well on this example, obtaining both good calibration and improved claim retention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I76UU2OqCg", "reviewer": "Reviewer_tBPp", "summary": "The paper proposes a new way to filter out LLM generations such that the resulting text only contains a set number of invalid claims with a certain probability. To do so, the authors adapt a new conformal mechanism to be level adaptive i.e. conditioned on the prompts rarity for example. They also propose a boosted version which additionally reduces the number of wrongly removed items in the generations set of claims. They then conduct experiments on synthetic and wikipedia data to confirm they claims.", "questions": "Please see above.\nI am more than happy to raise my score if the above have been addressed.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a level adaptive CP algorithm which to the best of my knowledge has not been done before in the LLM setting. \n- The paper is clearly written (for someone who has published in CP before it was easy to follow)\n- The paper tackles an important problem of hallucinations in LLMs which is timely and crucial for trustworthy LLMs\n- The paper also proposes a boosted CP version and elegantly solves the new challenges albeit the main ideas were already introduced in Stutz. There is nonetheless novelty in the additional challenges to make it conditional. Prop 3.1 in particular is an interesting observation given the linearity\n- The experiments overall show the superiority of the proposed methods compared to the current baselines. (there seems to be only one)", "weaknesses": "-  The paper starts off unfairly in figure 1, where the authors fixed a level of 90% for the baseline and 76% for the proposed method. This is misleading as one could easily also pick a lower coverage rate for the baseline and it would potentially look much less bad. This is concerning as it misleads the reader quite drastically until one reads the caption. Could you please explain why you chose 90% for the baseline but then only 80% for yours? Maybe i misunderstood your intentions with this figure.\n- Secondly, I would like to note that the original conformal guarantees are finite sample however, if i understood correctly, this is no longer the case here eq (3), thm2.1. Hence my question is that are there any other baselines to compare against that are not conformal? I believe that this paper misses a substantial related works section.\n- Thirdly, the experiments in the wikipedia dataset were choses with at most 3 wrong claims. Why did you choose 3  and not 0, 1 or 2 ? The fact that these experiments werent added in the appendix as ablation makes me question that 3 was chosen specifically to show the benefits of this paper. Please add these additional experiments for completeness as this number 3 was chosen out of nowhere. This also leads me to my next concern which is that of, even if we have at most 3 wrong claims, we can still not point to which one is wrong ... Could the authors please elaborate on the usefulness of this in the first place? Shouldnt you set the threshold to 0 wrong claims? This is a major concern from my side as this guarantee doesnt allow me to do anything useful in practice.\n-  Fourthly, I also agree with the authors that the exchangeable assumption is a big drawback of this method in practice. The method that needs to be trained with the conditional boosting, is fixed for a given type data. This can be quite limiting.\n- Lastly, there are not sufficient experimental results to support your claims. You have a single toy experiment with some Wiki dataset. Additional experiments or at least extensive ablation studies would be needed to convince me about the usefulness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the reviewer for their constructive criticism, and we hope that this rebuttal can both address some of the limitations that they identified and certain misunderstandings.\n\nFirst, we would like to briefly note that the level-adaptive CP work is novel both in and outside of the LLM setting. In the more canonical regression setting, no previous work shows how to adapt the conformal guarantee at test-time to ensure moderate set sizes. \n\n1) We respectfully disagree with the assertion that the paper starts off unfairly in Figure 1. The intent of that figure is to show that our method identifies the appropriate confidence level at test-time for preserving most of the output's claims. To further clarify, our method issues a guarantee that is different for each test output.  While it is true that running the conformal factuality method at 80\\% would result in fewer claims being filtered for this example, this value would not work well universally over all test examples, e.g. other test examples may require 70\\% or 90\\% levels. Thus, the shortcoming of previous methods identified in this figure cannot be corrected by simply running those methods at a different level. One of the main contributions of our work is to develop a method that uses a data-driven approach to learn a desirable level for each individual test point and then issues a filtered response that is well-calibrated to the learned level. We apologize for any confusion caused by this figure. In our revision we will modify the text following Figure 1 in order to provide more context for why we believe this is an appropriate comparison. \n\n2) We believe there may be some confusion here about the statements of our results. The theorems stated in our work are finite-sample exact guarantees that match the style and assumptions of those given in the conformal literature. For instance, note that while our result is more general, if $f$ is a constant function, the stated guarantee in Theorem 3.1 is identical to the typical coverage guarantee of split conformal prediction. We will aim to update the text surrounding our results to clarify this fact. We are unfamiliar with any non-conformal baselines for filtering claims from an LLM output with statistical guarantees (finite-sample valid or otherwise), but if any of the reviewers are aware of alternative approaches that we have not considered we would be happy to compare against them. \n\n3) We agree that we ought to clarify our unconventional error guarantee. While we understand the reviewer's concerns, we chose a non-zero error guarantee because it is impossible to preserve most claims with this guarantee without reducing the level (value of $1-\\alpha$) well below $50$\\% (see Figure 3 in the rebuttal attachment). On the other hand, for $3$ errors, the level function we end up with outputs values that are similar to what we might typically associate with ``high probability guarantees'', i.e. $1-\\alpha(X)$ is in the range of 80\\%-95\\%. We agree that the utility of a $3$ errors or fewer guarantee is less obvious than having $0$ errors, but if the probability of full correctness is undesirably low (maybe the user does not wish to issue a 40\\% guarantee in order to preserve most claims), this trade-off may be worthwhile. Moreover, the intent of this experiment is to demonstrate the flexibility of our method to accommodate weaker error guarantees compared to exact factuality, not to suggest that $3$ errors is the desirable outcome for all applications. Overall, we agree with the reviewer that this choice requires further explanation in the main text, and we will revise the manuscript accordingly.\n\n4) We agree that the exchangeability assumption is a limitation. We are not certain what the reviewer means by a given type of data, but in general we believe that the limitations of our method are no more significant than any existing approach to this problem. Additionally, as our methods target a form of conditional validity, they will provide robustness to some settings beyond exchangeability, e.g., when the test data undergoes a covariate shift.\n\n5) Many of the reviewers inquired about additional experiments and we agree that the single real-data example given in the manuscript is in need of additional support. We have performed additional experiments that include the requested ablations as well as a large second experiment on a more real-world dataset involving medical question-answering. These results are shown in our global rebuttal to all reviewers and we hope that will assuage the reviewer's concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their thoughtful comments and time spent considering our manuscript. 

Two of the reviewers (CDbE and tBPp) inquired about additional experiments and, in particular, ablations demonstrating the contributions of each of our methods individually. Additionally, reviewer R2Vh asked if the Wikipedia biographies dataset considered in the paper is representative of real LLM use cases. To address these concerns, we have performed ablation studies on both the Wikipedia dataset presented in the manuscript and on a new dataset consisting of medical question-answering examples. In all cases, we find that our adaptive level and conditional boosting methods are effective in improving the percentage of claims retained both individually and in combination. Figures 1 and 2 in the attached document display the results of these new experiments.

The reviewers raise a number of additional questions about both the theoretical guarantees of our method and the parameter choices made in our experiments. We thank the reviewers for these comments. We will revise the manuscript to clarify each of these issues and to explain the reasoning by our experimental set-up in detail, e.g., Figure~3 in the attached document explains why we chose a relaxed error criterion for the FActscore experiment. In general, we find that the results of our experiments are robust to our choices. More detailed responses outlining our planned changes and addressing the specific concerns raised by the reviewers can be found in the individual rebuttals below.

Meta-review: This paper presents conformal inference methods for obtaining validity guarantees on large language model (LLM) outputs, a problem known as conformal factuality (Mohri and Hashimoto 2024). It leverages previous work (Gibbs et al. 2023) to obtain **conditional** validity through level-adaptive conformal prediction. It also introduces a training method called "conditional boosting" which differentiates through the conditional conformal algorithm to obtain better conformity scores. Experiments on synthetic and real-world datasets demonstrate the performance of their methods and improvement upon existing approaches.    

Most reviewers point out as strengths the originality and soundness of the proposed approach (particularly the conditional boosting procedure) and the non-trivial generalization of (Gibbs et al. 2023) to incorporate general losses and adaptive error rate. The main weaknesses are the limited discussion about alternative frameworks (such as conformal risk control), the limited empirical analysis (with lack of ablations and not very well organized experimental section), and somewhat insufficient discussion about the practical limitations of providing non-zero error guarantee in a conformal factuality framework with LLMs. 

While I agree with the reviewer tBPp that a guarantee of three factual errors is in general not very useful in practice -- and I strongly recommend acknowledging this limitation more explicitly in the paper or presenting examples of situations where this can be useful to avoid overselling the practical usefulness of the method -- the other contributions of this paper are solid and potentially useful. The author rebuttal adds more experiments which will enhance the paper and alleviates some of the other concerns. I also urge the authors to include more details (as mentioned in the answer to CDbE) as what the advantages / disadvantages of their proposed framework are with respect to other proposed frameworks in the conformal prediction literature, such as conformal risk control.
Decision: Accept (poster)