Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees
Authors: Yu Gui
Ying Jin
Zhimei Ren
Venue: NeurIPS 2024 poster (2024)
Keywords: conformal prediction
selective inference
foundation model alignment
Primary area: safety_in_machine_learning

Abstract: Before deploying outputs from foundation models in high-stakes tasks, it is imperative to ensure that they align with human values.
For instance, in radiology report generation, reports generated by a vision-language model must align with human evaluations before their use in medical decision-making. This paper presents Conformal Alignment, a general framework for identifying units whose outputs meet a user-specified alignment criterion. It is guaranteed that on average, a prescribed fraction of selected units indeed meet the alignment criterion, regardless of the foundation model or the data distribution. Given any pre-trained model and new units with model-generated outputs, Conformal Alignment leverages a set of reference data with ground-truth alignment status to train an alignment predictor. It then selects new units whose predicted alignment scores surpass a data-dependent threshold, certifying their corresponding outputs as trustworthy. Through applications to question answering and radiology report generation, we demonstrate that our method is able to accurately identify units with trustworthy outputs via lightweight training over a moderate amount of reference data. En route, we investigate the informativeness of various features in alignment prediction and combine them with standard models to construct the alignment predictor.

Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees
Yu Guil Ying Jin
Zhimei Ren
Department of Statistics, University of Chicago
Data Science Initiative, Harvard University
Department of Statistics and Data Science, University of Pennsylvania yuguiQuchicn edu, yjin@hep... med. harvard. edu, zrenCwharton upenn. edu
Abstract
Before deploying outputs from foundation models high-stakes tasks, it is im- perative ensure that they align with human values. For instance, in radiology report generation, reports generated by a vision-language model must align with hu- man evaluations before their Conformal Alignment, 2 general framework for identifying units whose outputs meet a user-specified aligmment criterion. It is guaranteed that on average, pre- scribed fraction of selected units indeed meet the alignment criterion, regardless of the foundation model or the data distribution. Given any pre-trained model and new units with model-generated outputs, Conformal Alignment leverages set reference data with ground-trum truth alignment status train an alignment predictor. It then selects new units whose predicted alignment scores surpass a data-dependent threshold, certifying their corresponding outputs as trustworthy. Through applications to question answering and radiology report generation, we demonstrate that our method able to accurately identify units with trustworthy outputs via lightweight training over a moderate amount of reference data. En route, we investigate the informativeness of various features in alignment prediction and combine them with standard models to construct the alignment predictor.
medical decision-making. This paper presents
Introduction
Large-scale, pre- ire-trained foundation models remarkably powerful in generating relevant and informative outputs of various forms for diverse downstream tasks, marking a new era of artificial intelligence [34, 57, 47]. However, it has been recognized that they be prone to factual errors [43], hallucinations [15, 51], and bias [4], among others. These issues raise prevalent societal concerns regarding the reliable use of foundation models in high-stakes scenarios [7, 6). It remains significant challenge to ensure that outputs from these highly complex models align with human values [36].
Towards uncertainty quantification of black-box models, conformal prediction [50, CP] offers versatile, distribution-free solution. While CP has been applied to classification and regression tasks addressed by foundation models [25, 27, 46, 48, 54, 2], its use to ensure the alignment of general-forn outputs remains largely unexplored. Recently, [36] propose to generate multiple outputs for one input until it is guaranteed that at least one output meets specific alignment criterion. Such a guarantee, however, may not be immediately practical, as users must still determine (manually) which output(s) is aligned. [32] modify the machine-generater outputs ensure factuality with
Alphabetical ordering.
"The code is available at
aiggmmen..........m..mmmmmmmmm.. gnment.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

high probability, which in turn may sacrifice model power by making responses vague." As such, determining guarantees that are practically meaningful for downstream tasks and developing methods to achieve them remain an active research area.
This paper introduces distinct type of guarantee: we aim certify whether each model output is aligned or not, such that those certified are ensured to be mostly correct. This guarantee is directly relevant downstream tasks, as allows immediate and reliable use of the certified outputs without further modifications. On the other hand, we abstain from deploying model outputs for units (data points) that are not selected intuitiveeli. those are where the model lacks confidence (i.e., "knowing they do not know"), , and they can be evaluated by human experts. Such practice therefore provides an automated pipeline for the safe deployment of model-gene -geeeraee outputs.
Present work. We present Conformal Alignment (CA), a general framework that determines when to trust foundation model outputs with snite-sample, distribution-free guarantees. The framework is based on CP ideas, but unlike standard CP that searches the sampling space to construct a prediction set that contains desired output, we leverage the quantified confidence as an instrument to select units whose already-generated outputs are trusted. Given any model and any alignment criterion, Conformal Alignment ensures that prescribed proportion of the selected units' model outputs indeed meet the alignment criterion.* Concretely, suppose m E + new units awaiting outputs. Given an error level a E (0, 1). Conformal Alignment controls the false discovery rate (FDR) in selecting units with trustworthy outputs:
m i=1 1 selected and not aligned) a,
FDR E
(1.1)
>x=1 (i selected}
where the expectation is taken over the randomness of the data. FDR control offers an interpretable measure of the quality of selected deployable units. Our method builds upon the Conformalized Selection framework [18, 17]. leveraging a holdout set of "high-quality" reference data to guide the selection with calibrated statistical confidence for trusted outputs. Like in CP, (1.1) holds in finite-sample as long as the holdout data are exchangeable with the new units.
Our method inherits CP's (1) rigor, providing distribution-fies free, sinite-sample error control, and (2) versatility, applicable to any model and any alignment criterion. In addition, by selecting rather than modifying outputs, our approach preserves the (3) informativeness of the original outputs, and remains (4) lightweight, e.g., avoiding the need to retrain large models.
Demonstration of workflow. We illustrate the pipeline of Conformal Aligmment via a use case in radiology report generation (detailed in Section 5) in Figure I: each prompt corresponds to an model generates report. Our framework identifies a subset
X-ray scan, for which vision-language of generated reports that are reliable in the sense that at least, say 99%, of the selected reports are guaranteed to be aligned if they were to be compared with human expert report.
The workflow begins with (test) prompts Dies, corresponding X-ray scans X E X. A foundation model f: X' - ) generates a report f(X) E y for each scan. However, only high-quality reports should be handed over to doctors for clinical decisions. To achieve this, we leverage set of X-ray scans that are independent of the model's training process, each with human expert reports available. These scans are randomly split into training set Drr and calibration set Deat. Conformal Alignment trains a predictor on Du for predicting the alignment score that assess how likely generated output aligns. Finally, new reports are selected if their predicted alignment scores exceed -riven threshold, which is delicately set with Deal such that the FDR is strictly controlled at the desired level.
Practical relevance. We apply Conformal Alignment to two use lisess-uestion answering and radiology report generation to demonstrate its efficiency. With quite lightweight training of the alignment predictor (for example, training tree-based model would suffice), are able to accurately identify trustworthy outputs and select as many of them as possible. En route, we exhaustively investigate practical aspects such as the efficacy of various predictors in these applications and the amount of "high-quality" data needed for accurate selection to guide practical deployment.
SSee Appendix D.7 for more detailed discussion on the types of guarantces.
In this work, use "alignment" to refer to desired property of output that may vary with the context.

---

Tasks & LLM output
Conformal Alignment
Selective deployment
Diest
A()
f(n; Doamo Diest)
aighment
Use LLM
Alignment predictor
+
Calibrated cutoft
fraction are truly aligned
repor?
Prrats
Predicted
Use LLM report?
sigrament
Human mportt
LLM reportt
Aligne??
D
Duilb
Poodene aigeesn
Not confidint
Human-quality
Duata
D Dollo
Predictor training
Confidence calibration
Reliability Guaranteed!
Figure 1: Pipeline of Conformal Alignment instantiated in the radiology report generation example.
2 Problem setup
Suppose we have pre-trained foundation model f X' + Y that maps prompt to an output. Throughout, we view the model as given and fixed. Assume access a set of holdout units D D= (X, E, Y=)= where X; E & is the input prompt and E; E is any reference that may be used in judging alignment. An aligument function A y E ++ takes as input the generated output f(X) and the reference E, and outputs an (true) alignment score A = A(f E) [36]. In the radiology report generation example, A might be the similarity score between the report (X) and human expert report E. We shall detail the choice of A in each use case. Note that for unit without reference information, the alignment score of its generated output unknown, and we do not seek to evaluate it, which often requires expert annotation or human judgment.
m] such that most of their (unobserved) alignment scores exceed a pre-fixed threshold e E IR. The error is measured by
For test set Dest {Xn+J 1m 1==1. we aim select subset S S [m] := {L....,
Eje(m 1{A+t) S c. S} max((S))1)
FDR E
where |.| denotes the cardinality of set. In particular, we aim to enforce that FDR S a for pre-specified level a E 1). The FDR measures the averaged proportion selected units that not aligned, thereby directly quantifying the "cost" of deploying outputs in S. Such a measure (under different terminologies) appears to be considered empirically in selective prediction, although rigorous control has been lacking therein (see e.g., [52, 49, 37J). Besides FDR control, it is also desirable that as many units can be safely deployed, which translates into maximizing the power:
Σ Ee((m) [ml L(Ants > c,j E S} max Σ je(m) 1{Ants > ct,
Power
(2.1)
should be emphasized that our framework prioritizes FDR control over power, in that we strictly enforce the former while optimizing the latter under the constraints. This is related to but slightly
differs from the selection with controlled risk setting in selective prediction [11, 12]: see Section B. for more details. Such a setup is motivated by scenarios such as medical decision- decision-making and knowledge access, where the cost of committing type-I error (i.e., selecting a non-aligned unit) can be grave, and should be of primary concern. In addition, we define power as the proportion among truly aligned units, instead of the number of selections-indess powerful in generating aligned outputs, to begin with, it is natural that only a few of the outputs shall be deployed.
if foundation model not
2.1 Related works
Uncertainty quantification and conformal prediction for foundation models. Conformal predic- tion [50, 26, 41] is a distribution-firr framework for uncertainty quantification of generic prediction algorithms. This work adds to the growing literature that applies/cxtends CP to foundation models,
3

---

which varies in definitions of uncertainty and alignment, types of guarantees, rules for decision- making, etc. Below, we summarize representative ideas and contrast them with the present work.
line of work applies CP to construct prediction sets that cover the outcome for a single test unit in classification or regression problems addressed by language models [25, 27, 46, 48, 54]. While desired, their reliability in downstream tasks is only empirically evaluated: we note that coverage of prediction sets does not carry over the selected units [18, 19]. In contrast, this work applies to outputs of general forms, considers multiple units simultaneously, and offers guarantees relevant to downstream tasks. In particular, our methodology is built upon the series of papers in conformalized selection [18, 17]. While they aim to find large values of outcomes, we motivate and tailor the framework for the alignment of foundation models (a more detailed discussion in Appendix B.2).
Two works [36, 32] addressing the alignment of foundation models with CP ideas have been briefly discussed in Section 1. Our setup draws ideas from [36], but the guarantees we provide differ from both. In particular, our method may be better suited to situations where is desirable to avoid modifying the outputs or enlarging candidate sets which requires retraining the models.
More generally, [24, 29, 16] investigate the metric of uncertainty for natural language outputs. This is orthogonal to the present work, as our method is compatible with any alignment criterion. Nevertheless, our work adds to the discussion on desirable uncertainty quantification guarantees.
Selective prediction. Our framework is closely related to the idea of selective prediction, where one is allowed either to predict or to abstain from making a prediction [9, 11, 12, 33, 1]. In the context of large language models, [23, 53, 40, 55J have similarly considered the goal of teaching the model not to predict when the model is uncertain about its output, but theoretical guarantees on the alignment of the selected outputs are lacking. In contrast, this work rigorously formalizes the guarantee and provides an end-to-end framework that achieves it under mild exchangeability condition. As mentioned carlier, the strict control of FDR is closely connected to the "selection with controlled risk" setting in selective prediction [11, 12], which aims to control a slightly different error measure; we provide a detailed comparison of the error measures and methods in Appendix B.1.
Alignment of foundation models. There is a rapidly growing literature on the alignment of founda- tion models, wherein topics include alignment evaluation [30, 42), prompt design [22], training with human feedback [3, 35, 38), among others [8]. We rely on commonly-used alignment criteria in our applications [44, 29, 24]. However, achieve strict error control via post-hoc adjustment/selectionn instead of deriving training strategies to improve certain alignment metrics of the model.
3 Conformal Alignment
Given set of data DD with reference information, our procedure starts by splitting 'D into two disjoint sets: the training set Dir and the calibration set Deat- With a slight abuse of notation, we shall also use Dr, and Deal denote the indices of the units in the sets when the context is clear. We then fit a model g X' + IR on Due to predict the alignment score based on X (which may also involve information from f). Next, we generate model outputs f(X:) and compute the predicted alignment scores A, = XX) for every E [n + m]. We shall use (A,, At)see. to determine the selection set.
Following the framework of conformalized selection [18, 17], we gather statistical evidence for trustworthy outputs via hypothesis testing, where the null hypothesis for j E [m] is
H2 : Ants S.
(3.1)
Rejecting H, then reflects evidence that the (true) alignment score of unit j is above the threshold c, and therefore the generated output f(X++j) aligned. Under this framework, the task of selecting aligned units boils down to simultancously testing the m hypotheses specified in (3.1). For this purpose, we construct the conformal p- p-value: for any j E [m].
1 1 LIED. D.ED_ 11A, < c, A. 2 An+s}
(3.2)
Deall +1
It can be shown that when the test unit is exchangeable with the calibration units, the p-value defined above is valid in the sense that P(p & tt, Ants S c) < t for any E (0, 1) [18]. Intuitively, when generated output is likely to be aligned, we expect An+s have large magnitude, and py to be

---

small. As a result, we reject the null hypothesis that is, declaring sufficiently large alignment score- for a small Py. where the threshold for p-values is determined by the Benjamini-Hoehh (BH) procedure [5]. Concretely, let P(1) S ... P(m) denote the ordered statistics of the p-values; the rejection set of BH applied to the conformal p-values is S = Uj- E [m] Pj S ak" /m), where
ad
k = max [m]] P())
with the convention that max Ø = 0. We describe the complete procedure in Algorithm 1, and establish its validity in Theorem 3.1. For notational simplicity, we denote Z; (X,, E,) for all i E [n + m] (note that E; is not observable for n).
Algorithm 1 Conformal Alignment
Input: Pre-trained foundation model f; alignment score function A; reference dataset (X,, D test dataset Dist = (Xx++) algorithm for fitting alignment predictor G; align- ment level C; target FDR level a.
1: Compute the alignment score A; A()(X)), E,), Vi E D.
2: Randomly split D into two disjoint sets: the training set Dir and the calibration set Deal-
3: Fit the alignment score predictor with Dur: < (Duu).
4: Compute the predicted alignment score: A, + (X)), Vi E Deal U
5: for E [m] do
6: Compute the conformal p-values P, according to Equation (3.2).
7: end for
8: Apply BH to the conformal p-values: S BH(p: ..m..
Output: The selected units S.
Theorem 3.1. Suppose that for any j [m], {Zn+z} U (ii)ii DD are exchangeable conditional (Zu+e i.e., for any permutation of {l,..., + o} and any {a1...., Za, Antil J. it holds that
PP | _...... Zn = 2n, Zn+j = 2ntj
PP Z1 (1)) .... Zn [m( Zn+j 2m(n+j) 1 (Zn+))ez))
Suppose the predicted alignment score {A: Jic Du have no ties almost surely and sup,, 1g(2)) s M for some M 0. Then for any a (0, 1). the output S from Algorithm satisfies FDR a.
The proof of Theorem 3.1 is adapted from [18], and we include it in Appendix A for completeness. Remark 3.2. The exchangeability condition required in Theorem 3.1 is satisfied when the samples in Deat Diest are i.i.d., or when Den is drawn from set of samples without replacement, or when they are nodes on graphs in the transductive setting [14]. The assumption of no ties is without loss of generality, since one can always add a small random noise break the ties.
The following proposition characterizes the power (2.1), well as the fraction of selected units that can be deployed with confidence, when the samples are i.i.d. The proof is in Appendix A.2. A finite-sample power analysis based on concentration inequalities is in Appendix A.3.
Proposition 3.3. Under the same condition of Theorem 3, assume further that (X;, P.)EE(+mm)) are i.i.d. Define (t) IPA < c, g(X) 2 t) and t(a) sup(t: F(l a). Suppose that there exists a sufficiently small & > such that E(HG(()) < a for t E (f(a) - E, {a)). then
t
lim m
Power = P. (X)()(X)) S ((a)) 14>))
lim Paal.m
Σ
1lje > c) P(H(g(()) < <t(a),
Figure visualizes the asymptotic cutoff on (X) beyond which will selected by our method: it is the smallest value such that the red area on the right is less than a-proportion of red and blue areas combined. The area the blue part on the right of the cutoff is thus the asymptotic fraction of selected units, whose proportion among the entire blue area is the asymptotic power. Intuitively,
5

---

given a model (i.e., fixing areas for the red and blue), the power of Conformal Aligmment depends on how well g discerns those A against those A S c. Therefore, is important to identify features that informs the alignment of the outputs; we will elaborate on this point and deliver practical recommendations in our experiments in Sections and In addition, the fraction of deployable units (blue) additionally depends on the model power, i.e., the fraction of units whose outputs are indeed aligned. These two factors both affect the fraction of units selected by our method (the blue area on the right of the cutoff), which will also be demonstrated in the experiments.
4 Experiments for question answering (QA)
density
In this section, implement and evaluate Conformal Align- ment in guestion-answering tasks, where we consider a con- versational question answering dataset TriviaQA [21] and closed-boot reading comprehension dataset CoQA [39]. For each QA dataset, we use language models OPT-13B [57] and LLaMA-2-13B-chai [47] without finetuning to generate an answer f(x;) via top-p sampling for each input X, following the default configuration. The alignment score A, measures how well the generated answer matches the true reference an- swers provided in both datasets. Following [24, 29]. we let A, E {0.1] indicate whether the rouge-L score [28] between the LLM- answer and reference answer is no less than 0.3, and set c 0. See Appendix C.1 for more details.
A Asc
A>C
outoff À
g(X)
for selection
Figure 2:
Visualization of asymp- totic selection rule (red dashed line), with density curves of gXX) for A I c (red) and A > c (blue).
Dataset partition. Recall D is the reference dataset and Drest is the test set we select from. We fix |Dueal 500 and split as follows. Fixing 71, 7/2 E (0, 1), 71 + 72 < 1, we randomly sample [(71 + 72) [DI] instances without replacement from D as Dir for training the alignment predictor g. Within Du. a random subset of size 171 [P[] is reserved for hyperparameter tuning in computing certain features to be introduced later, while the others are used in fitting g given the features. We then set the calibration set Deat= = DVDr, For the results presented in this section, a = 0.2, "/2 = 0.5. Additional ablation studies for the choice of (71:72) are summarized in Section 4.1.
A Alignment score predictor. With the training set Dir. according to Algorithm 1, our goal is to train an alignment score predictor y that maps X; to A A;ethe estimated probability for A; > based on available information such as the input, model parameters, and outputs. To this end, we train a classifier with binary label A, and the following features (as preparation, we additionally randomly generate 19 answers for each input, leading to M = 20 answers in total for each input):
Self-evaluation likelihood (Self_Eval). Following [22, 29]. we ask the language model itself to evaluate the correctness of its answer and use the likelihood, (true), as a measure of confidence.
Input uncertainty scores (Lexical_Sim, Num_Sets, SE). Following [24], we compute a set of features that measure the uncertainty of each LLM input through similarity among the M = 20 answers. The features include lexical similarity (Lexical_Siiee ____mmmmm i.e.. the rouge-L similarity among the answers. In addition, we use natural language inference (NLI) classifier to categorize the M answers into semantic groups, and compute the number of semantic sets semantic entropy (SE). Following [24, 29], we use an off-the-shelf DeBERTaa-arge model [13] as the NLI predictor.
and
Outpul confidence scores (EigV(J/E/C) Deg(J/E/C), Ecc (J/E/C)) We also follow [29] to compute features that measure the so-called output confidence: with M generations, we compute the eigenvalues of the graph Laplacian (EigV), the pairwise distance of generations based on the degree matrix (Deg), and the Eccentricity (Ecc) which incorporates the embedding information of each generation. Note that each quantity is associated with a similarity measure; we follow the notations in [29] and use the suffix J/E/C to differentiate similarities based on the Jaccard metric, NLI prediction for the entailment class, and NLI prediction for the contradiction class, respectively.
We use all above features to train the alignment predictor with standard ML models including logistic regression, random forests, and XGBoost. Existing works often use individual features in calibrating model confidence [22, 29, 24]; in this regard, we evaluate these features by using each individual feature to train the predictor g and reporting the resulting power, which delivers additional insights upon their informativeness in predicting model alignment.

---

OPT-13B: |D|=100
OPT-13B: |0|=500
OPT-13B: 2000
1.0
1.0
1.0
FDR Power
FDR and Power
FDR and Power
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
11MMA-2-13BBchat
|D|=100
LLaMA-2-13B-chat
|D|=500
LLaMA-2-13B-char
|
1.0
1.0
1.0
FDR Power
FDR and Power
FDR
FDR and Power
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 3: Realized FDR (blue) and power (red) for conformal alignment applied to the TriviaQA
dataset (with 71 0.2, 72 0.5) various FDR target levels. The top row corresponds the results from OPT-13B and the bottom row to those from LLaMa-2-13B--a each column corresponds to a value of [D). Shading represents the area between one standard deviation above and below the mean.
1.00
OPT-13B
OPT-136
1.00
LLaMA-2-13B-that
LLaMA-2-13B-hhst
0.75
0.75
Power
Power
0.50
0.50
0.50
0.50
025
0.25
0.25
0.25
Cont Align
Salf Evai
0.00
0.00
0.00
0.5
targat
target level
largat
target
Figure 4: Comparison of FDR and power between Conformal Alignment and the heuristic baseline where we select units by thresholding self-evaluation scores Self, Eval with cutoff 1-the target FDR level. The comparison is conducted on the TriviaQA dataset.
4.1 Experiment results
This section reports the results on the TriviaQA dataset, where a subset of 8000 questions are considered, with the alignment predictor trained via logistics regression; additional results when the predictor is trained via random forest and XGBoost are in Appendix D.I. Parallel results on the CoQA dataset with 7700 questions can be found in Appendix D.2. We also provide QA examples in Table 1 in Appendix D.5 to illustrate the performance of our method.
Conformal Alignment strictly controls the FDR while heuristic baseline does not. Fig- ure 3 shows the realized FDR and power for Conformal Alignment at target FDR levels E (0.05, .., 0.95). averaged over 500 independent experiments. The FDR curves demonstrate the finite-sample error control. It is worth noting that the FDR is tightly controlled; it becomes constant for sufficiently large a under which all units can be selected without violating the FDR.
To demonstrate the importance of statistically rigorous treatment, we compare Conformal Alignment with a heuristic baseline, where one asks the language model to evaluate confidence (i.e., the Self_Eval feature above), and threshold the obtained likelihood {gfelf) with a cutoff of 1
if were the "probability" of alignment, i.e., Staseline {j [m] : ggell > a). The resulting FDR and power for the TriviaQA dataset are in Figure 4. The baseline fails to control FDR, showing that the self-evaluation score over-confident. In addition. it is also less powerful than our procedure despite the higher error rate, implying that the self evaluation is less informative for predicting alignment. Comparing LLaMA-2- 1---A-2--BB-hha 13B-chat and OPT-13B, also observe that more powerful models may tend to be more (over-)confident in their output.
7

---

OPT-138
LLaMA-2-138--00
1.0
Bel Evil
0.8
0.8
Lestcal, 5km
Fgght))
Power
0.6
Power
0.6
Ees(C)
Degit))
FighWi]]
0.4
0.4
Eus(E)
DigiE)
Fighli]
0.2
0.2
Eooil)
Dugil)
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Tevel FDR
Target of FDR
Figure 5: Power versus target FDR levels for TriviaQA dataset when the alignment predictor is trained with logistic regression over individual features with [D] = 2000, ?1 = 0.2, 7/2 = 0.5, averaged over 500 independent experiments. Note that the FDR is always controlled though not depicted.
More powerful model enables more powerful selection. The performance of Conformal Align- ment also varies with the language models. We observe that more powerful LLMs can generate answers with higher accuracy which further leads to stronger signals in selection. For OPT-13B, the largest value of FDR is 0.5; we have nearly no discoveries and the power is close zero when = 0.05. In comparison, with LLaMA-2-13B-- high 0.50 even an FDR level of 0.05. As such, one may also use the FDR/power curve from Conformal Alignment as a measure to compare LLMs' performance.
FDR never exceeds 0.25, and the power is as
What feature is informative for alignment?? In addition to the capability of LLMs, the features used train g also play vital role the power of selection. To compare the informativeness of the aforementioned features for the alignment of outputs, we use one score at a time as the feature in training g and show Power/FDR curves Figure When using the OPT-13B model, Self _vv_lllllll and Num_Sets are the least powerful, while Deg (J) and Ecc (J) are the most powerful in predicting alignment. Such comparison is similar for aLLMA-2-13HH 1-A---3BB-chat. We defer the FDR curves to Figure 14 in Appendix D.3 which remain strictly controlled. We include more detailed analysis on feature importance via Shapley value in Appendix D.6.
How many high-quality samples are needed? important for applying Conformal Aligmment. In practice, it is desired if handful of data suffices to accurately identify aligned outputs. study the effect of reference sample size [D] on performance, in Figure 3, we vary [D] in {100, 500, 2000} with fixed 71 0.2 and 72 = 0.5. In general, a few hundred bigh-quality samples suffice to achieve relatively high power, showing that Conformal Alignment does not require too expensive extra labeling. More specifically, that when [D] increases, there is a slight improvement in power, especially when a is small; for large values of a, the power is robust to (D). In addition, a larger labeled sample size reduces the variance in both FDR and Power and improves the stability in selection.
The size of high-qualiie samples [D] needed is
observe
Ablation study. In Appendix D.4, we present ablation studies for the choice of 71 and 72 in data splitting. In practice, larger values of Y1 and 72 allow more samples for training the alignment predictor but decrease the resolution of the conformal p-values. With fixed [D] and 72, we observe that the performance of our method is not sensitive to the choice of ? ; in particular, a small value of ' (e.g., 0.2) is sufficient for powerful selection. When fixing D and m = 0.2, we suggest setting 22 [0.3, 0.5] to balance the sample sizes for both the predictor training and the BH procedure.
5 Experiments for chest X-ray report generation
In this section, we apply Conformal Alignment chest X-ray report (CXR) generation. Following the pipeline in Figure 1, we apply our method to (a subset of) the MIMIC-CXR dataset [20]. In practice, imagine a healthcare institute employs a foundation model to automatically generate radiology reports based on chest radiographs. Conformal Alignment can be used to determine which reports to trust and refer doctors for medical decisions, and the rigorous guarantees it offers imply that a fraction of the approved reports indeed match the results if they were to be read by human expert.

---

|D|=100
[0|=500
[O]=2000
1.0
1.0
1.0
FDR and Power
0.8
FDR and Power
FDR and Power
0.6
0.8
0.6
0.4
0.4
o
FDR
FDR
FDR
0.2
0.2
0.2
Power
Power
Power
0.0
0.0
0.0
02
Target level of FDR
Target level of FDR
Target level of FDR
Figure 6: Realized FDR (blue) and power (blue) for Conformal Alignment applied to CXR report generation with alignment predictor from logistic regression. The results are averaged over 500 runs with 71 = 0.2, 72 0.5. Each subplot corresponds to one value of [D].
We obtain the base foundation model f by fine-tuning an encoder-decoder model (pre-trained Vision- Transformer and GPT2) using separate training set. For generated report f(X;) with a reference report written by a radiologist, the alignment score A; defined as follows: we use CheXbert [44] to convert both reports to two 14-dimensional vectors of binary labels with each coordinate as the indicator of positive observations; we then define A, E {0, 1} as the indicator of whether there are at least 12 matched labels, and set O. We employ the same set of features and the same procedure in Section 4, except for Self Eval which not available from the current model. More details about the experimental setup are in Appendix C.2.
5.1 Experiment results
summarize the main findings from our experiments below; additional results with random forest and XGBoost predictors, well ablation studies for the choice of hyperparameters, are deferred to Appendix E. To illustrate the role of the alignment score predictor, we also present examples of CXR images with reference and generated reports in Table of Appendix E.4.
Tight FDR control. Figure presents the FDR and power curves for Conformal Aligmment where the alignment predictor is trained with all features using logistic regression, averaged over 500 independent runs the procedure. Again, we observe that the FDR is tightly controlled at the target level; it remains constant for sufficiently large since by then all units be selected without violating the FDR. We also observe satisfactory power, although is lower than that is the QA tasks since CXR generation can be more challenging task, and the model in use is fine-tuned with limited number training samples. We shall expect even higher power for better tuned models.
Hundreds of reference data suffice. Comparing subplots in Figure 6, as the size the reference data D increases, we observe slight improvement
1.0
Num. Sets
in power. However, we again find that [D] = 500
Lexical Sim
already suffices for powerful deployment of Con- formal Alignment. In practice, we expect this sam- ple size also suffices for more powerful foundation models that are more carefully designed for this specific task.
Eigh(C)
Eco(C)
Power
0.6
Deg[C)
EigV(E)
0.4
Eco(E)
Deg(E)
0.2
EgV(J)
Ecc(J]
Informativeness of individual features. To study the informativeness of different features for alignment, we use one feature at a time to train the logistic regression classifier and compare the re- sulting power versus target FDR levels in Figure 7 from Conformal Alignment (curves showing FDR control are in Figure 23 of Appendix E.2). In the CXR task, features based on Jaceard similarity are still the most powerful while those based on II.I prediction exhibit slightly lower power. More complicated texts and more tokens CXR reports could be potential reasons for the relatively poor performance of NLI prediction.
0.0
Deg(J)
Target leval of FDR
Figure 7: Power at each FDR level for CXR dataset when the alignment predictor trained with logistic regression over each individual feature. fixing [D] 2000, 71 0.2, 72 0.5.

---

Discussion and limitations
We have presented Conformal Alignment, a principled framework for ensuring the alignment of foundation model outputs with provable, distribution-fie FDR control. Compared with standard conformal prediction, we provide solution that may more relevant to downstream tasks than existing ideas. We demonstrate that our framework is flexible, lightweight, and effective through concrete applications to question answering and radiology report generation tasks, where we also provide practical recommendations on sample size, data splitting, and feature engineering.
The choice of alignment score in different applications remains open problem. For example, in the radiology report generation example, as pointed out by [56], there can be discrepancy between the score given by CheXbert and that given by human evaluation; such, will be interesting to use our framework with actual human evaluation labels. More generally, interesting future directions include designing implementation tailored for other appropriate applications, tackling parallel or sequential outputs, improving power in data-scarce scenarios (e.g., with better base models and more informative features), extending to multi-task and in-context learning settings, and generalizing our framework to control other error notions beyond FDR that may be meaningful for downstream decisions.
Acknowledgments and Disclosure of Funding
Z.R. is supported by NSF grant DMS -41-124
10

---

References
[1] Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint 10.0022/ 2021.
[2] Anastasios Nikolas Angelopoulos, Stuart R Pomerantz, Synho Do, Stephen Bates, Christopher P Bridge, Daniel C Elton, Michael H Lev, R Gilberto Gonzalez, Michael I Jordan, and Jitendra Malik. Conformal triage for medical imaging ai deployment. medRxiv, pages 2024-02, 2024,
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
[4] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limita- fions and opportunities. MIT Press, 2023.
[5] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
[6] Nick Bostrom and Eliezer Yudkowsky. The ethics of artificial intelligence. intelligence safety and security, pages 57-69. Chapman and Hall/CRC, 2018
In Artificial
[7] Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira Tsansva-Atanasova. Artificial intelligence, bias and clinical safety, BMJ quality & safety, 28(3):231-27, 2019.
[8] Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Serean O Arik, Tomas Pfister, and Somesh Jha. Adaptation with self-evaluation to improve selective prediction in llms. arXiv preprint arXiv:2310. 1689, 2023.
[9] Chi-Keung Chow, An optimum character recognition system using decision functions. IRE Transactions Electronic Computers, (4):247-254, 1957.
[10] Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sam- ple distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, pages 642-669, 1956.
[11] Ran Yaniv et On the foundations noise-free selective classification. Journal of Machine Learning Research, 1(5), 2010.
[12] Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
[13] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 2020.
[14] Kexin Huang, Ying Jin, Emmanuel Candes, and Jure Leskovec. Uncertainty quantification over graph with conformalized graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.
[15] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qian- glong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05222. 2023.
[16] Xinmeng Huang. Shuo Li. Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, and Edgar Dobriban. Uncertainty in language models: Assessment through rank- calibration. arXiv preprint arXiv:2404.03163. 2024.
[17) Ying Jin and Emmanuel J Candès. Model-free selective inference under covariate shift via weighted conformal p-vuuues. arXiv preprint arXiv:2307.09291, 2023.

---

[18] Ying Jin and Emmanuel Candès, Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1-41, 2023.
[19] Ying Jin and Zhimei Ren. Confidence on the focal: Conformal prediction with selection- conditional coverage. arXiv preprint urXiv:2403.03868, 2024.
[20] Alistair EW Johnson, Tom J Pollard, Seth Berkowitz, Nathaniel R Greenbaum, Matthew Lungren, Chih-ying Deng, Roger G Mark. and Steven Homg. Mimic-cxI, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):117, 2019.
[21] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint urXiv:/705.03555 2017.
[22] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan. Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodde Nova DasSarma, Eli Tran-Johnson. et Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522) 2022.
[23] Amita Kamath, Robin Jia, and Percy Liang. Selective question answering under domain shift. arXiv preprini arXiv:2006.0966,, 2020.
[24] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.0960 2023.
[25] Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. arXiv preprint arXiv:2305. 18404, 2023.
[26] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Associ- ation, 1136523):004--11 1011- 2018.
[27] Shuo Li, Sangdon Park, Insup Lee, and Osbert Bastani. Traq: Trustworthy retrieval augmented question answering via conformal prediction. arXiv preprint arXiv:2307.04642 2023.
[28] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[29] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305. 19187, 2023.
[30] Yang Liu, Yuanshun Yao, Jean- Francoiss Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: survey and guideline for evaluating large language models' alignment. arXiv preprint 230i::2308.03374 2023.
[31] Pascal Massart. The tight constant in the dvoretaky--kiffer-wofovw inequality. The annals of Probability, pages 1269-1222 1283, 1990.
[32] Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402. 10978, 2024.
[33] Hussein Mozannar and David Sontag. Consistent estimators for learning defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.
[34] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
12

---

[36] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn. Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. arXiv preprint arXiv:2306.10193, 2023.
[37] Stephan Rabanser, Anvith Thudi, Kimia Hamidich, Adam Dziedzic, and Nicolas Papernot. Selective classification via neural network training dynamics. arXiv preprint arXiv:2205. 13332, 2022.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arxiv 2023, arXiv preprint arXiv:2305.1829) 2023.
[39] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[40] Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928. 2023.
[41] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
[42] Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of Ilm alignment on global representation. arXiv preprint arXiv:2402. 15018, 2024.
[43] Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. Language models that seek for knowledge: Modular search generation for dialogue and prompt completion. arXiv preprint arXiv: 13224, 2022.
[44] A Smit, S Jain, P Rajpurkar, A Pareek, AY Ng, and MP Lungren. Chexbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arxiv [esel]. published online april 20, 2020, 2004.
[45] John D Storey. The positive false discovery rate: a bayesian interpretation and the q-value. The annals of statistics, (1(6):2013222 2035,, 2003.
[46] Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. Api enough: Conformal prediction for large language models without logit-access. arXiv preprint arXiv:2403.012/6 2024.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09288. 2023.
[48] Dennis Ulmer, Chrysoula Zerva, and André FT Martins. Non-exchangeable conformal language generation with nearest neighbors. arXiv preprint arXiv:2402.00707. 2024.
[49] Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings. arXiv preprint arXiv:2203.002/1, 2022.
[50] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world, volume 29. Springer, 2005.
[51] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.00439,, 2021.
[52] Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference on Computer Vision, pages 148-166. Springer, 2022.
[53] Qi Yang, Shreya Ravikumar, Fynn Schmitt-Ulms, Satvik Lolla, Ege Demir, laroslav Elistratov, Alex Lavaee, Sadhana Lolla, Flaheh Ahmadi, Daniela Rus, et al. Uncertainty-aware language modeling for selective question answering, arXiv preprint arXiv:2311. 2023.
13

---

[54] Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking 1lms via uncertainty quantification. arXiv preprint arXiv:2401. 12794, 2024.
[55] Hiyori Yoshikawa and Naoaki Okazaki. Selective-lama: Selective prediction for confidence- aware evaluation of language models. Findings of the Association for Computational Linguistics: EACL 2023, pages 2017-2028, 2023.
[56] Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Ng. al. Evaluating progress in automatic chest x-ray radiology report generation. Patlerns, 4(9), 2023.
[57] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. URL https://axxi, org/abs/220..01066 3:19-0, 2023.
14

---

Broader impact
crucial ensure that LLM outputs are aligned with human values, especially in high-stakes scenarios (e.g. medical decision veeision-making). This paper provides generic framework that identifies reliable LLM outputs and controls the misalignment error rate in a rigorous way. The proposed approach (conformal alignment) can be applied to a wide range of real-life scenarios, e.g. medical decision-making (radiology report generating) and conversational chatbot (question answering).
A Deferred theory and proofs
A.1 Proof of Theorem 3.1
Let (ax, a) 2M 1fa > c} g(z);; define also V; V(X,, A,) and V. V(X;, c) for every i E [n + m]. By construction, V.2 = - Ā; Vi equals to - A, when A, S c and equals to 2M - A; otherwise. The conformal p-value defined in (3.2) can be written as
1 EiEDa
1(A S c, A, 2 Ants) 1 EieDa 1(V S Pats)
Pj
1 [Deat]
1 Peal
which is the p-value considered in [18]. It can also be checked that (ei U U hiem have not ties almost surely. Applying Theorem 2.6 of [18] concludes the proof.
A.2 Proof of Proposition 3.3
The proof an adaptation of Proposition 2.10 of [18]. We first note that the proof of Proposition 2.10 goes through with U = 1, which is the version shall use in what follows.
To start, in the proof of Theorem 3.1, we take V(x 2M 1fa > cJ - -g(x). Let F(o)
P(V(X, A) I v) for any u E R. Then
F(2) = PXV A) s v) = PP(A S c, + < ,, g(X) S s)
= -H(- +PA < <c22 - x S u))
As a result, F(V( = F(-g(()) = H (gx))), and for any t E R.
t
P(F(V(X, c)) tt) F(H(((()) St)
Invoking Proposition 2.10 of [18], we have
P(F(V(X, c)) S t(a){ A> c)
lim IDe m 000
Power
P(A> c)
S (t))))))))(()))))))
P(A>0) c)
-P(H(((X)) <t(a) [A>
Similarly,
Power Σ 1fAj > 0}
2m (JESSAn
lim IPal.m-00
lim
mo
m
= P(H(g(X)) S t(a) A > c) P(A x c)
P(H(g(X)) S ((a), A>)
We have therefore completed the proof.
A.3 Finite-sample power analysis
In this part, we provide finite-sample power analysis based on concentration inequalities. We define
(t) = (x( 2 ttA> c), (t) P(g(X) 2 t, c).. F(t) = P(g(X) t).
15

---

The empirical version of these quantities are defined as
(1) = m )=1
[a( (o((x+5) 24, 4, Ants >c), c), (1) = Σ
2
1=1
(t) = 19 x+3) 2 At, sc), (t)
1
( s))
(t) (l Fa(t) 1 Σ ((x(,))
We also define mn = -ド 1(A++) > cf as the number of truly aligned test outputs.
Proposition A.1. Under the same condition of Theorem 3.1, assume further that (X,, Ei) are i.i.d. For any 8 E (0, 1), with probability least 1 8,
(T fm F(A c) + Cm
E/ 1fg(X (x,,, n+ 2 f. An+3 > c)
F- (T(T+) + cm
ΣØ 1 [Ants > c}
(PA c) cm
where
T4
inl 1: n
1 1+ +ee
nen fm
Sa sag.
T- inf 1:
1
1+ 1+ ncn
S sag.
n+
Fm
n n+1 Fm + cm
and 6m (oog(8/5((2n)) en vlog(8/0)((2m)
Proof of Proposition A.I. Fix any constant 8 E (0, 1). Consider the function o(X, A) M 1(A > c) + g(X), where M > 2sup=ex 19(x)) < oc is sufficiently large constant such that infzex G(x, c + 1) > supaex 0( c). Therefore, we know that for any |d] < M/2, it holds that
(t) = P( 2 M +t), F_ = Pt s ī(X, A) < M/2).
And similar relations hold for the empirical versions of these quantities. Applying the Dvoret- -e--ieefrrWolfwwiii (DKW) inequality [10, 31] with union bound for the empirical c.d.f. of g(X, A) for both calibration and test data, we know that with probability at least -- 6,
sup
F+ (t) [t) S Em sup (t) F-(t) S Cm sup (m))) F(t) S Cm
sup F (t) F+) cny
sup F. (t) F-(1) s 'n and m+ + - P(A > a) S Cm:
(2))) We call the event that all where recall that 6. = / /log(8/8) (2m) and cn llg
above holds the "good event" E", which happens with probability at least 1 - 8.
By definition, we can show that the selection set of Conformal Alignment is equivalently S ti [m]: g(Xn+z) 2 F). where 7 = inf {t: FDR(t) S a). and
(DR = m 1+
Xx) 2 1, A sc D TH
1 (t) n +
+1 E (g (N++t) 2 1
(t)
By the DKW inequality above, with probability at least 1- 8.
1 nF (t) nen
1 1 (t) + ne,
< (DR(1)
n +
Fmn (t) + 6m
n n Fm (1) cm
holds simultaneously for all E R. On the other hand, the (realized) power is
m j= 1 [g (g(Xn+))) 2 F, Ants > cj
h+ m
Pow
Σn 1 1(Ant) c)
16

---

On the good event E", we have
(7) Cm P(A>)) +m
(F) + cm
S Pow S
P(A > C) cm
By the definition of f, we know that < T_ the good event E". The monotonicity of realized power as function of selection cutoff thus implies that
Pow 2 (T-
P+(T-) Em
2
m+ (T+)
P(A c) cm
F+ +
Pow S
m+
P(A c) cm
B Additional comparison with prior works
B.1 Comparison with selective prediction
For selective prediction, El-Yaniv et al. [11, 12] suggest considering the risk measure, defined (in our context)
E
Σ Jm
1{Ant) S Sc,j SS
=
E
Ejejm
10 E S}
This measure is known the marginal FDR (mFDR) in the multiple testing literature and there has been an exhaustive discussion of its comparison with FDR. For example, Benjamini and Hochberg [5] proposed FDR in favor of mFDR, since the latter is impossible to control in the global null case; Storey [45] also pointed out that mFDR cannot be used for controlling the joint behavior of the numerator (number of false selections) and the denominator (number of selections). On the other hand, to control the risk/mFDR, [12] proposes a method that searches for the cutoff over a grid of values, and then takes union bound achieve the overall error control, which might be conservative. In contrast, our method for FDR control does not involve taking union bounds, delivering sharp finite-sample control guarantees.
B.2 Comparison with conformalized selection
Our work instantiates conformalized selection [18] for the reliable deployment of foundation model outputs. We detail our contributions as follows. First, we justify that selecting reliable outputs with FDR control is practically reasonable criterion for downstream tasks. Second, we formulate the alignment problem
selective prediction problem that can be tackled with conformal selection, adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current application in job hiring and drug discovery requires numerical outputs and predictions; the task of aligning foundation model outputs, however, does not come with clear numerical output. To make conformal selection applicable our purpose, we formulate the alignment status numerical indicator that could be inferred with reference information, and then decide how train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Our third contribution is the implementation of the pipeline. this regard, we conduct extensive numerical experiments, finding that using lightweight prediction model such as logistic regression random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides concrete and easy-to-use pipeline for practical use.
C Additional implementation details
This section contains implementation details for Section 4 and 5.
17

---

C.1 Question answering
As is mentioned in Section 4, we adopt subset of sample size 8000 from the TriviaQA dataset [21]² and a subset of sample size 7700 from the CoQA dataset [39] For both TriviaQA and CoQA datasets, we follow the exact text preparation and prompt engineering procedure in [29], which we show below. To evaluate the correctness of generated outputs, we adopt the rouge-L score [28] that focuses on the longest common subsequence between two sequences. Following the practice in [24, 29], a generated answer defined to admissible when the rouge-L score is no less than 0.3.
TriviaQA. The TriviaQA dataset can be loaded from the Python module datasets and we use re. rc.oocontext split. Prompts in use are the same those in [29, 47] and take the following format.
Answer these questions:
Q: In Scotland bothy/bothie is a?
A: House
Q: [sample question]
A:
CoQA. The CoQA dataset is downloaded from https://domi oads cs stanford edu/nlp/ data/ cataa//cc///aa///eeaa/eeee coloo//coa---ooo coqa-dev-vi 0. json and the prompts are the same as those in [29, 24].
[The provided context paragraph] [additional question-answer pairs]
Q: [question]. A:
Answers are generated using the default configurations similar to [29]; in specific, we use num_ num_beams=1 sample=True, cop_p=1.00 O, cop_k=O, temperature=1.1 For each question, we generate M = 20 answers independently, with which we treat the first output as f(X;) and utilize the .M outputs to calculate the similarity- scores.
Given the generated outputs, we adopt the pipeline in [29] to calculate the uncertainty and confidence scores. The self-evaluation score _eva), (True), is obtained using the prompt in [22] as follows:
[story]
Question: [question]
Here are some brainstormed ideas: [few_shote examples]
Possible Answer: [generated answer] Is the possible answer:
(A) True
(B) False
The possible answer
The celf-evaluation score is then defined as the output probability associated with the generated answer "A)".
Language models. We use language models without fine-tuning for comparison in terms of the accuracy of the generated answers: OPT-13B and LLaMA-2-13B-chall The implemented OPT-13B model is from Hugging Face https: https://huggingfee implemented LLaMA-2-133 2-113--chal is from https:/ attps:///lamaa /ttp///aaaaaa///// all.ma.metaa.....amaam..aa.....a meta. com. We utilize an off- tie-the-ssfff DeBERTa-large model [13]] as the NLI classiferr to calculate similarities.
co/facebook/opt 13b and the
"nttpe://stanforddll. htt.:a/llppiitubb.ooooooooo
18

---

C.2 CXR report generation
We use the subfolders p10, p11, p12 from the MIMIC-CXR dataset [20] (these all the data use, including fine-tuning and subsequent experiments for Conformal Alignment) to limit the resources required for fine-tuning, The MIMIC-CXR dataset can be accessed from the PhysioNet project page betps://physionss org/ wwwtton/mmmmicceocc Health Data License 1.5.0.
...00.0. O/ under the PhysioNet Credentialed
In this application, we fine-tune a lisounlaagguae model. with the Vision Transformer google/vit- cese-patcch16-211 100-224-innnn in21k pre- rreitrained on ImageNet-22 - as the imgge encoder and GPT2 the text decoder. We largely follow the training procedure in [36]. In particular, each raw image is resized hold-out dataset with sample size of 43, 300
224 224 pixels. We then fine-tune the model for 10 epochs with batch size of 8, and other hyperparameters are set to default values. The training process takes about 10 hours
NVIDIA A100 GPU.
Once fine-tuned, the model is treated as fixed for generating reports. The data not used fine-tuning used as the training, calibration, and test data in our experiments. For report generation, we use the default configurations bea__he2gh do_ do_sample= e-Tuu)) and the input of the vision- language model consists of pixel values of images (resized to 224 224 pixels) and tokenized prompts (tokenized via the same GPT2 model). For each image, we generate A 10 reports independently. with which treat the first output as fX,) and utilize the M outputs to calculate the features (similarity-bans -based scores) for training the alignment predictor. To determine a generated report is admissible, we follow the practice in [36] and use the CheXbert model [44] to convert both the generated and reference reports into 14-dimensional vectors. For each coordinate, the entry falls into the categories "Blank", "Positive". "Negative" "Uncertain", with which further convert them to binary vectors with each entry indicating "Positive" or not. The alignment score A; E {0, defined to be the indicator of whether there binary vectors. Similar to the QA task, we modify the pipeline in [29] to calculate the uncertainty and confidence scores, in which we utilize an off shelf DeBERTa- large model [13] as the NLI classifier to calculate similarities.
least 12 matched coordinates between the two
D Additional experimental results for question answering
D.1 FDR/Power for TriviaQA dataset with alternative classifiers
We present the FDR/power plots when applying Conformal Alignment to the TriviaQA dataset and the alignment predictor is trained with random forests (Figure 8) and XGBoost (Figure 9), fixing ?1 0.2 and 72 0.5. These figures similar Figure (which uses logistic regression train the alignment predictor) in the main text. We observe similar messages as those in Section 4: the FDR always tightly controlled with satisfactory power. Regarding the choice [D), we similarly see that 500 high-qualin samples suffice for the powerful deployment of our method.
'uttps://hugginggacoooo httt::///uuggggggoooooooo/////// logoggl/vittbbaeeeeeeeeeee watpgg/lvvi--as--pttte 224--n21k
19

---

OPT-13B: |D|=100
OPT-13B: |D|=500
OPT-13B: |0]0
1.0
FDR and Power
FDR and Power
FDR and Power
0.5
0.5
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLaMA-2-1330 --h-hhhttt
|D|=100
LLaMA-2-13B-chen
[D|=500
LLaMA-2-13B-chat
[D|=2000
FDR and Power
1.0
FDR and Power
1.0
FDR and Power
1.0
FDR
FDR
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 8: FDR/power plots for conformal alignment with TriviaQA dataset and random forests as the base classifier. Details are otherwise the same Figure 3.
OPT-13B: |D|=100
OPT-13B: |D|=500
OPT-13B: 20000
1.0
1.0
1.0
FDR Power
FDR and Power
FDR and Power
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.5
0.0
0.5
1.0
0.0
1.0
LLaMA-2-13B-chatt
|D|=100
LLaMA-2-13B-chat
|D|=500
LLaMA-2-13B-chh
[D|=2000
1.0
1.0
1.0
FDR and Power
FDR
FDR and Power
FDR
FDR and Power
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 9: FDR/power plots for conformal alignment with TriviaQA dataset and XGBRF as the base classifier. Details are otherwise the same as Figure 3.
D.2 FDR/Power on CoQA dataset
In this part, we present a set of results parallel to the main text and Section D. when we apply Conformal Alignment to the CoQA dataset; details about the dataset are in Appendix C.I.
We present the FDR/power plots for the CoQA dataset when the alignment predictor is trained with logistic regression (Figure 10), random forest (Figure 11), and XGBoost (Figure 12), fixing 71 = 0.2 and 7/2 0.5. We again observe similar messages those in Section 4: the FDR remains tightly controlled, and sample size of (D) 500 suffices for the powerful deployment of our method.
We also recall the baseline, where asks the language model to evaluate its confidence (i.e., the a as it were the "probability" of alignment, i.c., Spaseline { [m] ofelr > 1 a}. In Figure 13,
Self Eval feature above), and threshold the obtained likelihood (g5ell) with a cutoff
20

---

present the results with the CoQA dataset, in which the baseline fails to control FDR for any given and also exhibits lower power than our proposed approach.
OPT-13B: |D|=100
OPT-13B: |D|=500
OPT-13B: [D|=2000
1.0
1.0
1.0
FDR and Power
FDR and Power
FDR and Power
0.5
0.5
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
(LaMA-2-13B-char -hhhhttt
|D|=100
LLaMA-2-13B-chatt 500
LLaMA-2-13B-cher |
|D|=2000
1.0
FDR and Power
FDR
FDR and Power
FDR
FDR and Power
FDR
Power
Power
Power
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 10: FDR/power plots for conformal alignment with CoQA dataset and logistic regression for training the alignment predictor. Details are otherwise the same as Figure 3.
OPT-13B: |D|=100
OPT-13B: 5D|5000
OPT-13B: [D]=2000
1.0
1.0
1.0
FDR Power
FDR and Power
FDR and Power
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLaMA-2-13B-chatt
|D|=100
LLaMA-2-13B-chat
|D|=500
LLaMA-2-13B-chat:
[D]=2000
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level FDR
Figure 11: FDR/power plots for conformal alignment with CoQA dataset and random forests as the base classifier. Details are otherwise the same as Figure 3.
21

---

OPT-13B: |D|=100
OPT-13B: |D|=500
OPT-13B: [D|=0000
1.0
FDR and Power
FDR and Power
FDR and Power
0.5
0.5
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLaMA-2-13B-chhr
|D|=100
LLaMA-2-13B-chmr
|D|=500
LLaMA-2-13B-chal
[D|=2000
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level FDR
Figure 12: FDR/power plots for conformal alignment with CoQA dataset and XGBRF as the base classifier. Details are otherwise the same as Figure
OPT-138
OPT-13B
LLaMA-2-13B-chat
LLaMA-2-13B-ohat
1.00
1.00
1.00
1.00
0.75
0.75
Power
0.50
0.50
0.50
0.50
0.25
0.25
0.25
Align Salf_Eval
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.25
0.50
0.75
0.0
0.5
1.0
0.25
0.50
0.75
target level of FDR
target level of FDR
target level of FDR
target level of FDR
Figure 13: Comparison of Conformal Alignment at FDR level a versus heuristic baseline applied to CoQA dataset. Details otherwise the same Figure 5.
D.3 FDR for individual features
Figure 14 presents the realized FDR at each FDR target level for the TriviaQA dataset when the alignment predictor is logistic regression over each individual feature.
OPT-13B
LLaMA-2-13B-chh
1.0
1.0
0.8
0.8
Sets
Lescal
BgVIC:
0.6
0.6
(c:(CC
FDR
Deg(C)
FigVIF)
0.4
0.4
Eco(E)
Deg(E)
EigV1
0.2
0.2
Ecc(h)
Deg(U]
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Target of FDR
Target level
Figure 14: The realized FDR each FDR target level for TriviaQA dataset when the alignment predictor is trained with logistic regression over each individual feature.
22

---

From Figure 14, we note that FDR is always controlled for each feature. However, FDR with more powerful LLM (LLaMA-2- 1-3---3- is tightly controlled while Self_Evar and Num_Sets with OPT-13B produce FDR and power both close to zero when a: is small as their predictive power is too low. We omit similar results when conducting the same experiment on the CoQA dataset.
D.4
Ablation studies
In this part, present and discuss ablation studies for choosing the data splitting schemes Y1 through experiments on the TriviaQA dataset, with the alignment predictor trained via logistic regression over all the features in the main text.
Varying (size of the tuning set). Recall that we use random subset of size [D]] for tuning hyperparameters [29] the features, and subset of size [ for training the alignment predictor. We start by studying the effect of ?1 on the performance with fixed %/2. This means the sample size for predictor training is fixed, but there is a tradeoff between the sample sizes used for hyperparameter tuning and the calibration step. Here, logistic regression is used to train the alignment predictor and the total number of "high-quality" data fixed [D] 2000.
We show the results when we vary 7 {0.2, 0.4, 0.6} for the TriviaQA dataset (Figure 15) and the CoQA dataset (Figure 16).
OPT-13B: v1=0.2
OPT-13B: y==0.4
OPT-13B: y1=0.6
1.0
1.0
1.0
FDR Power
FDR and Power
FDR and Power
0.5
0.5
0.5
FDR
FDR
FDR
Power
Power
Power
0.0
0.5
1.0
0.0
1.0
0.5
LLaMA-2-13B-chatt
=1=0.2
LLaMA-2-13B-chha
=04444
LLaMA-2-13B-chatt yı =0=6.6
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
FDR and Power
1.0
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Ablation study for the choice of 7/1 FDR/power plots for conformal alignment with TriviaQA dataset. We fix [D] = 2000, 72 0.3, and use logistic regression as the base classifier.
Figure 15:
23

---

OPT-13B: V==02
OPT-13B: y1=0,4
OPT-13B: V==0.6
FDR and Power
FDR and Power
1.0
FDR and Power
1.0
0.5
0.5
0.5
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLaMA-2-13B-cher
y1=0.2
LLaMA-2-13B- =1=0.4
LLaMA-2-13B-chaal #
v1=0.6
1.0
1.0
1.0
FDR Power
FDR
FDR and Power
FDR
FDR and Power
FDR
Power
Power
Power
0.5
0.5
0.5
0.0
0.0
0.0
0.5
1.0
0.0
1.0
0.5
Target level FDR
Target level of FDR
Target level of FDR
Figure 16: Ablation study for the choice of 7/1 FDR/power plots for conformal alignment with
TriviaQA dataset: [D] 2000, 72 0.3, logistic regression as the base classifier.
In Figure 16, we can see that with fixed [D] and when increases, the power slightly decreases due the decreasing sample size for training the alignment score predictor and BH procedure. Besides, we note that 7/1 = 0.2 is sufficient for tuning the parameters to calculate the features.
Varying 12 (size the predictor training set). We proceed to study the choice of 72, the proportion of reference data used for training the classifier. We vary the value of 72 while fixing the size of the reference set |D] 2000 and ?1 = 0.2 per our recommendation from the last part. Note that with [D] and Y1 fixed, varying 72 also reveals the effect of [Deal. We use logistic regression the base classifier. The results are shown for TriviaQA dataset in Figure 17 and CoQA dataset in Figure 18.
At values '/2 0.1 and 7/2 0.7, observe slight power loss compared with the middle two columns. The reason is that when 7/2 is close to either 0 or 1, there will be unbalanced sample sizes between ()2) and Deal, and the scarce of sample in each set will affect the performance, i.e. causing larger variance in FDDR/power or smaller power. In light of this, we suggest the choice of 72 [0.3, 0.5] to balance (22) and Deal-
OPT-138: ==001
OPT-138: y200.3
OPT-13B: y150.5
OPT-138: W=0.7
1.0
1.0
FDR and Power
FDR and Power
0.5
0.5
0.5
0.5
FDR
FDR
FDR
FDR
FDR
FDR
Power
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLaMMA2-11B-chhtt
y 1
LLMMM-2-13B-rhaa
===0.3
LLaMA-2-1 2-1tt-hhatt: =05
(LaMMA-2-13B-chat 0
FDR and Power
1.0
FDR and Power
1.0
Power
seed aaasss
Power
Power
Power
Power
Power
0.5
0.0
0.0
0.0
0.0
0.0
0.5
0.0
0.5
0.0
0.5
1.0
0.0
0.5
1.0
Target
Target of
level
Target level
Figure 17: Ablation study for the choice of 7/2 FDR/power plots for conformal alignment with
TriviaQA dataset, fixing [D] = 2000, = 0.2, and using logistic regression as the base classifier.
24

---

OPT-138: v=
OPT-138:
OPT-13B: Y2=0,5
CPT-138: y2=0.7
FDR and Power
1.0
FDR and Power
1.0
1.0
0.5
0.5
0.5
0.5
FDR
FDR
FDR
FDR
HDR
Power
Power
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
LLach-2-13B-chaft
LLaMA-2-13B-chal y.*
LLaMA-2-13B-chalt
*0.5
LLaMA-2-13B-chaa, 0.7
1.0
FDR and Power
FDR and Power
Power
Power
Power
Power
Power
0.5
0.5
0.5
FDR
0.0
0.0
0.0
0.0
0.0
0.5
1,0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Target level
Target
Tergel FDR
Terget
Figure 18: Ablation study for the choice of 72: FDR/power plots for conformal alignment with CoQA
dataset, fixing [D] = 2000, 71 = 0.2, and using logistic regression as the base classifier.
Question
Reference answer
Generated answer
Alignment scores
A
Which company produced the Hastings and Herald aircraft?
Handley ey-agee
Havilland
A = 0.083
A = Å; = 0.190
How many 'E' tiles are provided Scrabble game?
12
2
A A; 0.505
In the 1972 film Cabaret, Sally Bowles is working in which club?
KitKat
Kit Kat Klub
A, =1
An orrery, popular in the 13th and 19th centuries, was a model of what?
The Solar System
Solar system
A, = 0.814
A; =
Of which band Feargal Sharkey the lead singer until 1983?
THE UNDERTONES
The Undertones
Å; 0.938
Table I: Illustration on TriviaQA dataset with LLaMA-2- chat: selection threshold 0.388.
D.5
Real examples in QA
D.6 Empirical evaluation of feature importance
In Figure 5 and 7, quantify the importance of individual features in the alignment predictor via the ROC curve. In this section, we use the Shapley value as well as the model-based score to evaluate the individual feature importance more directly.
We used the XGBoost functionality to compute the importance scores Figure 19d. We also computed the Shapley value of individual features to measure their importance based on the prediction models we use in the manuscript. As the Shapley values are model-agmostic, we present results for three choices of g(X) with different base classifiers, including logistic regression, random forests, and XGBoostRF, in Figure 19a, 19b, and 19c. Although the exact order of features varies in four plots, there are features that have consistent dominating effects in all figures, e.g. (J) and EigV (J) based on the Jaccard similarity. In addition, NumSets as the feature alignment score predictors tends to exhibit an effect close to zero in all four figures. We should note that the aforementioned
25

---

Deg(J)
-0.14
Ecc(J) +0.08
-0.
BjgV(J)
+0.11
Lexical _im
Deg(C)
+0.11
Deg(J) +0.05
Deg(E)
+0.1
BigV(J)
+0.05
Ecc(J)
+0.07
SE
+0.04
Lexical_ _im
Eco(E)
+0.03
Eco(E)
+0.03
Self_Eva Eval
+0.03
EigV(C)
+0.03
Deg(C)
+0.02
NumSet
+0.03
Ecc(C)
+0.02
Self_Eval
+0.02
Deg(E)
+0.02
ElgV(E)
+0.01
EleV(C)
+0.02
Ecc(C)
++0
EigV(E)
+0.02
SE |+
NumSet
+0.01
0.00
0.05
0.10
0.15
0.00
0.05
0.10
0.15
0.20
(a) Shapley Values: logistic regression.
(b) Shapley Values: Random Forests
Ecc(J)
Eco(J)
-0.16
EigV(J)
EigV(J)
+0.07
Deg(C)
Deg(J)
+0.05
Deg(J)
Lexical_ Sim
+0.05
Eco(C)
Deg(C)
Deg(E)
Deg(E)
+0.03
EigV(C)
SE
+0.02
EigV(E)
Eco(E)
+0.02
Eco(E)
Eco(C)
+0.02
Lexical_Siie
Cig
SE
Self Eval
+0.01
NumSet
Eig)(E)
+0.01
Self Evall
NumSet
++0
0.00
0.05
0.10
0.15
0.00
(d) Feature importance for XGBRF extracted from sklearn.
(c) Shapley Values: XGBRF
Figure 19: Measures of Feature importance (Dataset: TriviaQA, LLM: LLaMA-2-13B-chat,
findings are also revealed in Figure 5 in the paper, where predicted scores with EigV (J), and Ecc (J) have higher power and that with NumSets is much less powerful. We thank the reviewer again for raising this question for more comparison of feature importance in the alignment score predictor.
D.7 Comments on FDR guarantee and comparison other frameworks
The major distinction between method and other applications of conformal prediction the foundation models is the type of guarantee we pursue. This is related to the following question: how the outputs of these algorithms in practice? In short, (a) For [36], it is unclear how to pick one answer from the multiple answers in a way that does not hinder validity. (b) Conformal actuality [32] makes the original outputs less specific, which can hinder downstream use.
To demonstrate (a), we conducted additional experiments more clear comparison. With TriviaQA dataset and the base model LLaMA-2 2-----2-1BB-haar we follow the procedure in [36] and have the following examples:
Example Question: In tennis, losing two sets 6-0 known double what? True answer: Bagel. Generated set: { bbagel').
26

---

Example 2: Question: What was the name of the brothel in The Best Little Whorehouse in Texas? True answer: Chicken ranch. Generated set: {'miss monas', 'chicken ranch', 'miss mona's', 'miss monas bordello').
As we can see in Example 1, the generated set is very informative and can be used directly. However, although the generated set in the second example has a high probability of covering the true answer, it contains very different answers and is potentially confusing in practice. In other words, whether "Chicken ranch" or "Miss Monas" is correct is unclear the user.
In addition, as is shown in Figure 20 (left panel), we present the proportion that the size of generated size is for different values of k E N, which shows that there are more than 40% of sets having no less than 4 distinct items. In addition, Figure 20 (right panel) shows that in nearly 30% of the questions, the admissible answers (with rougeL 0.3) take up less than 20% of the uncertain set, which poses a challenge in the direct deployment of answers in the uncertain set.
0.30
0.25
0.20
0.15
0.06
0.00
6
10
0.0
0.2
0.4
0.8
0.8
Size of [C]]
Proportion of Ag=
Figure 20: Additional histograms for Conformal Language Modeling: sizes of uncertainty sets and proportions admissible answers, where C₁ denotes the uncertainty set for question No.1 (Dataset: TriviaQA, LLM:
E
Additional experimental results for CXR report generation
E.1
Additional results with alternative classifiers
We also use random forests and XGBRF as alternative classifiers to train the alignment score predictor. Similar to Section 5, we fix 71 0.2 and 72 0.5. The results in Figure 21 for random forests, and Figure 22 for XGBoost. As (D) increases, we observe a slight increase as well as a decrease in variance in the power curve while FDR remains tightly controlled.
DDJ-100
|D|
[D]-2000
1.0
1.0
1.0
FDR and Power
FDR and Power
FDR and Power
FDR
FDR
FDR
Power
Power
Power
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 21: FDR/power plots for conformal alignment with CXR dataset when using random forests to train the alignment predictor. Details are otherwise the same in Figure 6.
27

---

DD|=100
|D|=500
[e|=2000
1.0
1,0
1.0
FDR and Power
0.8
FDR and Power
0.8
FDR and Power
0.8
FDR
FDR
FDR
Power
Power
Power
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 22: FDR/power plots for conformal alignment with CXR dataset when using XGBRF to train the alignment predictor. Details are otherwise the same in Figure 6.
E.2 Informativeness of individual features
Figure shows the plots of FDR in the same experiment with Figure 7. Recall that we use one feature at time train the alignment score predictor. While FDR is controlled for all features, Num_Sets and Ecc (C) tend to return empty selection sets when a is small, which results in zero FDR as well context of CXR report can be challenging for the Eccentricity score since the dimension embeddings much higher.
power. Besides, Ecc (E) is also relatively less powerful, which implies that the
1.0
SE
Num__ets Sets
0.8
Lexical _im
EigV(C)
Ecc(C)
0.6
FDR
Deg(C)
EigV(E)
0.4
Ecc(E)
Deg(E)
EigV(J)
0.2
Ecc(J)
Deg(J)
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Target level of FDR
Figure 23: FDR at each FDR level for CXR dataset when the alignment predictor is trained with
logistic regression over each individual feature, fixing [D] = 2000, 71 0.2, 722 = 0.5.
E.3 Ablation studies
We present ablation studies for the sample splitting hyperparameters 72.
Varying 7/ (size of the tuning set). With [D] 2000, ?/2 0.3 and logistic regression as the base classifier, we vary Y1 in {0.2, 0.4,0.4000 0.6) in Figure 24, in which both FDR and power exhibit larger variance when Y1 is close 1 due to limited sample size in Dr, U D Thus, recommend ?1 = 0.2 similar to the QA tasks.
Varying 2 (size of the predictor training set). With [D 2000, 71 0.2, and logistic regression as the base classifier, the proportion of the training set varies in {0.1, 0.3, 0.5, 0.7).
In Figure 25, the plots with 72 = 0.1, 0.3, 0.5 are comparable, but as is close to 1, implying that the Dest will produce p-values with a low resolution, the FDR and power are less stable. In this case, it reveals that there are different requirements in sample size for Drr and D To better understand
28

---

Y==0.2
Y1=0.4
V==0.6
1.0
1,0
1.0
FDR and Power
0.8
FDR and Power
0.8
FDR and Power
0.8
FDR
FDR
FDR
Power
Power
Power
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
Target level of FDR
Target level of FDR
Target level of FDR
Figure 24: Ablation study for 71 FDR/power plots for conformal alignment with CXR dataset when
fixing D
2000, 72
0.3 and using logistic regression train the alignment predictor.
12-0.1
12-03
Y2-0.5
FDRaadd Power
0.
FDR Power
0.8
FDR and Power
0.3
0.8
8.8
0.2
FDR
0.
FDR
0.
FDR
02
FDR
Power
1.0
0.
02
02
1.0
0.0
Target lovel of FDR
Target leval FDR
Targel level FDR
Target level of FDR
Figure 25: FDR/power plots for conformal alignment with CXR dataset:
2000, 71 0.2,
logistic regression as the base classifier.
the asymmetric roles of the training and calibration sets, we further decrease the size of the reference dataset and choose (D] = 100 to repeat the experiment.
Figure 26 further shows that too small (0.1) or too large (0.7) proportion of Dir are both not preferred, thus any value 7/2 E [0.3, 0.5) is suggested.
E.4 Real examples in report generation
To illustrate the role of the alignment reference and generated reports in Table 2. In this realization, following Algorithm 1, the selected set is obtained by S = {i Deal : A, 2 f} with = 0.592. For the first image, as highlighted in pink, the foundation model misreads the size of the cardiac silhouette. Accordingly, the alignment score predictor returns A, = 0.19 G, and the conformal alignment procedure successfully excludes this generated report. For the second image, the conditions of the cardio-mediastinal silhouette and hilar contours are correctly captured by the foundation model, and a correct assessment both pleural effusion and pneumothorax is provided.
predictor, present examples of CXR images with
X=70.1
K2P0.3
W2-0.6
k2007
1.0
1.0
1,0
FDR and Powes
FDR Pndittttt
EDR and Power
0.8
0.6
0.6
and
0.6
0.4
0.4
FDR
22
FDR
0.
Power
Power
Power
Power
0.0
0.0
0.0
0.0
02
044
0.5
3.8
1.0
0.0
0.2
3.4
0.6
0.8
1.0
0.0
02
0.4
0.6
0.8
1.0
0.0
02
0A
0.6 08
1.0
Target level FDR
Target lovel FDR
Targei level FDR
Target level of FDR
Figure 26: FDR/power plots for conformal alignment with CXR dataset: logistic regression as the base classifier.
[D] 100, ?1
0.2,
29

---

CXR image
Reference report
Generated report
In comparison with the study of substantial increase in opacifications diffusely involving both lungs. Cardiac silhouette remains within normal limita and there is evidence of pleural effusion. The appearance suggests diffuse pulmonary edema. However, in the appropriate clinical setting, widespread pneumonia or even ARDS could be considered.
In comparison with the study there is little overall change Again there is substantial enlargement the cardiac silhouette with diffuse bilateral pulmonary opacification consistent
there
with pulmonary edema. the appropriate clinical setting, superimposed pneumonia would have to be considered.
Alignment scores: A, 0
Å. 0.194 <
The lungs well inflated and clear. The cardiomedi astinal silhouette, hilar tours, and pleural surfaces normal. There is pleu ral effusion or pneumothorax Mild degenerative changes seen throughout the thoracic spine. IMPRESSION: acute cardiopulmonary process.
The cardiomediastinal and hi 1ar contours are within normal
limits. The lungs clear without focal consolidation, pleural effusion pneumoth orax. IMPRESSION: No acute cardiopulmonary process.
Alignment scores:
A = 1
Ā, = 0.875 7
Table 2: Illustraioon of conformal alignment with CXR examples: selection threshold 7 0.592.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims made in the abstract and introduction match theoretical and experi- mental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the technical conditions, under which the main theoretical results hold, are clearly stated in the main paper.
Guidelines:
30

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: please see Theorem 3.1, Proposition 3.3 and A.1. Proofs and technical details are presented in Section A.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: please see Section 4, and C. Reproducible codes can be found at https: //github. com/
ormal www.ooooooaaaaaloaaalaoreeeeeeee alignment.
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data access is stated in Section C and reproducible codes can be found at attps://githur com/yug http//yggrrr////// saay//con coof ormal-oaaaeeeooo al gnmentt.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: http://ccpullcgguuicieeeeoo
/CodeSubmis ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
32

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: training (VLM fine-tuning) details are shown in Section C and other LLMs (OPT-13B, LLaMA-2-13B-000 and GPT-2) are used without fine-tuning.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: error bars are shown in figures when applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: please see more details in Section C.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
33

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttur///////i....eeeee wttpiiss..pcpppppe cc/public Cuublic/Ethin eettee/Eessieieeeeeeeeee csGuidelines"
Answer:
[Yes]
Justification: the research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the broader impacts of this work, including the deployment of foundation models with misalignment error rate control in a wide range real-life applications, are discussed in both the abstract and introduction. To be concrete, it is crucial to ensure that LLM outputs are aligned with human values, especially in high-stake scenarios (e.g. medical decision-making
This paper provides a generic framework that identifies reliable LLM outputs and controls the misalignment error rate in a rigorous way. The proposed approach (conformal alignment) can be applied to a wide range of real-life scenarios, e.g. medical decision-making (radiology report generating) and conversational chatbot (question answering).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
34

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: the usage of LLMs and datasets (TriviaQA, CoQA, and MIMIC-CRR) are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, thcorswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
35

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "9GHQIYRUbw", "reviewer": "Reviewer_nRXb", "summary": "The paper proposes to use conformal prediction to determine when to trust model outputs. To this end the authors apply conformal risk control on across unit instead of focusing on a single unit. This is especially relevant in their X-ray example where the goal is to give a set of trusted documents to the medical professional to review. They show that their adaptation of conformal prediction is also sensible in terms of FDR and power.", "questions": "see above", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is the first to focus on conformal prediction across units compared to previous work that worked mainly on a unit level.\n- The paper proposes a new and practical way to use conformal prediction in foundation models for downstream tasks, especially when \n- The paper shows on extensive experiments the trade offs between power and FDR control which seems to be better than existing baselines.", "weaknesses": "- My main concern with this paper is that, at the end of the day it is just an application of conformal risk control to foundation models. No extensive new theory was needed, but rather just a twist to the setting. If the authors could clearly design the technical challenges faced it would be helpful in making a decision on this paper.\n- For the alignment score predictor, can the authors please elaborate on how this score was picked and why this was the best design choice? These seems to be randomly picked and a more systematic justification would be very welcome to better understand if this score can be improved or not\n- In practice for a medical professional, what is the target FDR rate? In all experiments the power is quite slow to increase at low FDR rate, however in practice i would have assumed that a medical professional would not want to look through documents where every second one is wrong? This puts into question the motivation as well as practicality of this paper which i would urge the authors to clarify.\n\ni am more than happy to change my score of the above have been addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments, and for appreciating the novelty and practicality of our framework, as well as our experimental results! \n\n- **Q1, Technical challenges**: We would like to clarify that the emphasis of the current work is the novel instantiation of conformal selection (we also note that this is a distinct framework from conformal risk control [1,2]) for reliable use of foundation model outputs. Compared with conformal selection, the major challenge lies in justifying the criteria, formulating the problem, adapting existing techniques to the current setting, and implementing the complete pipeline for practical use. We elaborate on the challenges in the following. \n\n    - The first challenge is justifying what criterion is practically useful. We argue that selecting reliable outputs (similar to the “prediction with abstention” idea) with FDR control is a practically reasonable criterion, since it ensures that most of the deployed outputs (such as the radiology reports that are handed over to doctors) are indeed reliable and useful. This contrasts our work with the works that apply conformal prediction to standard classification/regression tasks. Particularly in the context of LLM/foundation models, our proposal differs substantially from conformal language modeling [3] that generates multiple outputs for a unit, ensuring *at least* one of the outputs is aligned, and conformal factuality [4] that edits the original outputs to make it less specific/informative, so as to achieve guarantees on the correctness of the output for a single unit. More comparisons with the works connecting conformal prediction and LLM/foundation models can be found in Section 2.1 of the manuscript.\n\n    - The second challenge is formulating the alignment problem as a selective prediction problem that can be tackled with conformal selection, and adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current applications in job hiring and drug discovery apply to numerical outputs and predictions, rather than aligning foundation model outputs (which do not come with a clear numerical output). To make conformal selection applicable to our purpose, we need to formulate alignment as a numerical indicator that could be inferred with reference information, and then decide how to train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Indeed, how conformal prediction ideas may be used to calibrate LLMs/foundation models (beyond traditional numerical outputs) is a nontrivial problem that has attracted considerable recent interest; in this sense, we contribute to the discussion on this important issue. \n\n    - With the general recipe, the third important challenge and contribution is the practical instantiation of the pipeline, which features to use, what model to train for alignment, among other considerations. In this regard, we conduct extensive numerical experiments, finding that using a lightweight prediction model such as logistic regression or random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides a concrete and easy-to-use pipeline for practical use. Indeed, even beyond the current setting of selecting reliable outputs, we view the choice of features and prediction models as contributions that may be helpful for predicting the reliability of outputs for other purposes. \n\n\n- **Q2, Choice of alignment score predictor**: Thanks for the insightful question! To avoid any misunderstanding of what you refer to, we discuss the choice of (1) the alignment score $A=\\mathcal{A}(f(X),E)$, (2) the alignment predictor $g\\colon \\mathcal{X}\\to [0,1]$, and (3) the features $X$ that are used for prediction.  \n\n   - First, the choice of the alignment score $A=\\mathcal{A}(f(X),E)$ is context-dependent. In particular, our examples focus on the widely adopted definitions of ``reliability’’ of model outputs in the literature (see the references in the manuscript). For instance, in the X-ray report generation case, we use Chexbert [5] to generate the alignment score, which is widely used in the literature.\n\n   - Second, given features $X$ and the reference, the alignment predictor $g(X)$ should be as accurate as possible for the alignment score $A$ (see the power analysis in Proposition 3.3 that shows how the power of our procedure depends on the accuracy of $g(\\cdot)$). Indeed, the optimal $g(X)$ should be monotone in $P(A>c|X)$. In principle, one may use any machine learning model to train this predictor, while we have offered some practical suggestions in our manuscript. Through experiments, we find that relatively lightweight models such as logistic regression and random forests suffice for accurately identifying reliable outputs. This is consistent with other works on selective prediction for LLMs that use tree models [6]. \n\n   - Finally, for the choice of features $X$, the general principle is that they should be (1) informative and (2) light-weight (that is, they should be computationally efficient to acquire and train the model $g$ with). In particular,  we choose popular (heuristic) measures of uncertainty in the literature (see the references in the manuscript) as the features, since they shall be informative of the reliability of model outputs; we then train a mapping from the features via lightweight models. As such, the combination of the features is data-driven. \n\nPlease see the response to Q3 in the Official Comment. We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hMLCYHf1iO", "reviewer": "Reviewer_w89M", "summary": "This work applies recent work on conformal outlier detection [1] in the context of LLM alignment. The authors assume that one, given input $X$ and output $Y$ of an LLM, has access to an alignment scorer $A$ that can then produce an alignment score of $A(X, Y)$. They treat the null hypothesis of being unaligned as being $H_0: A(X, Y) \\leq c$, i.e., the alignment score being smaller than a threshold $c$. They describe the typical score function as $\\widehat{A}$, and the resulting conformal score being $\\widehat{A}(X)$ for a data point. Using the scores, they construct a conformal p-value that is superuniform when $A(X, Y) \\leq c$ and apply the Benajmini-Hochberg procedure on the conformal p-values to get FDR control.", "questions": "- What is $q$ in Prop. 3.3 --- is it some constant less than $\\alpha$?\n- What is the cost in power in directly estimating the asymptotic cutoff (and getting a PAC bound) vs. the conformal procedure?", "limitations": "Yes, the limitations are sufficiently addressed.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is theoretically rigorous and applies the framework of conformalized selection to \"alignment\" of LLMs, which is an application that is not directly explored in previous papers on conforamlized selection. The experiments show that with a small amount of data, one can derive powerful procedures with FDR control when using off the shelf open source models, and that's nice. It is interesting that a relatively \"weak\" (computationally) scoring model can still produce good results.", "weaknesses": "I think the paper, in some sense, is just an extra series of experiments for conformalized selection --- we already know it works well for many deep learning tasks from previous papers, and the experimental results are not particularly surprising, since it continues that trend. Many other conformal papers in the LLM space try to tackle the problem that's unique to LLMs, i.e., factuality, and try to investigate the fact that it is not already a pre-defined, structured task, and aim to apply conformal in a meaningful way. Thus, I think this paper does lack some novelty in its methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the rigor, power, and versatility of our framework! We address your comments in the following. \n\n- **Q1: Novelty**. Thank you for raising this question! We would like to argue that the motivation of this paper is exactly to address the unique challenges of LLMs—the reliability in adopting their outputs—with conformal selection techniques. In the process of applying conformal selection to this problem, there are nontrivial challenges including problem formulation, criteria justification, and practical instantiation of the framework (which is at least as important as the challenges other works address in applying conformal prediction to more traditional classification/regression tasks). We elaborate on the challenges and comparisons with other CP-LLM works below.\n\n\t\n\n    - **Unique challenge of LLM and “pre-defined tasks”**. \n        - We would like first to argue that the reliability of their outputs (including factuality) is a unique challenge of LLMs, which we aim to address in this work, similar to other related works. Instead of working with “pre-defined” tasks, a lot of our efforts are devoted to finding the appropriate criteria in the context of foundation model deployment, formulating this problem into a tractable form that can be tackled by conformal selection, arguing that it does provide practically meaningful guarantees that have not been previously considered, and developing (practical) techniques that make sure this framework leads to meaningful results. We elaborate on the challenges in the following. \n\n            (1) The first challenge is justifying what criterion is practically useful. We argue that selecting reliable outputs (similar to the “prediction with abstention” idea) with FDR control is a practically reasonable criterion, since it ensures that most of the deployed outputs (such as the radiology reports that are handed over to doctors) are indeed reliable and useful. This contrasts our work with the works that apply conformal prediction to standard classification/regression tasks. Particularly in the context of LLM/foundation models, our proposal differs substantially from conformal language modeling [3] that generates multiple outputs for a unit, ensuring *at least* one of the outputs is aligned, and conformal factuality [4] that edits the original outputs to make it less specific/informative, so as to achieve guarantees on the correctness of the output for a single unit. More comparisons with the works connecting conformal prediction and LLM/foundation models can be found in Section 2.1 of the manuscript.\n\n            (2) The second challenge is formulating the alignment problem as a selective prediction problem that can be tackled with conformal selection, and adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current applications in job hiring and drug discovery apply to numerical outputs and predictions, rather than aligning foundation model outputs (which do not come with a clear numerical output). To make conformal selection applicable to our purpose, we need to formulate alignment as a numerical indicator that could be inferred with reference information, and then decide how to train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Indeed, how conformal prediction ideas may be used to calibrate LLMs/foundation models (beyond traditional numerical outputs) is a nontrivial problem that has attracted considerable recent interest; in this sense, we contribute to the discussion on this important issue. \n\n            (3) With the general recipe, the third important challenge and contribution is the practical instantiation of the pipeline, which features to use, what model to train for alignment, among other considerations. In this regard, we conduct extensive numerical experiments, finding that using a lightweight prediction model such as logistic regression or random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides a concrete and easy-to-use pipeline for practical use. Indeed, even beyond the current setting of selecting reliable outputs, we view the choice of features and prediction models as contributions that may be helpful for predicting the reliability of outputs for other purposes. \n\n\n        - Second, we would like to emphasize that our framework applies to any ``alignment’’ criterion, including the important factuality issue that you have mentioned. The distinction from other works is the criterion we choose and the guarantee we deliver, which we view as an important contribution to the current active discussion in this area. For example, conformal language modeling [3] applies conformal prediction to a single unit, obtaining a less specific statement than the original generated response that is guaranteed to be correct (but possibly less informative); however, to ensure a population-wise guarantee, this may compromise originally informative and reliable outputs (by making them more vague). For another example, the method in [7] makes a selection when the CP set is a singleton; but here there are no theoretical guarantees on the outcome of *selected units*.  \n\nPlease see the response to the remaining questions in the Official Comment. We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hnqqQoecfQ", "reviewer": "Reviewer_FNNT", "summary": "This paper proposes a procedure similar to conformal prediction to select deployable units and control the FDR. Asymptotic result is provided to justify the correctness of the proposed algorithm. Real data experiments in QA and medical tasks are also provided to demonstrate the effectiveness of the proposed method.", "questions": "My rate is 4 for now because of the many issues in this paper. I still value the theoretical justifications and would like to raise my score if the authors are willing to clarify them.", "limitations": "No limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The mathematical formulation, theoretical justification, and basic experimental setups are all clearly and well written. The theoretical results are appreciated because they can provide more rigorous statistical guarantees on the proposed method.", "weaknesses": "While the theoretical results are appreciated, the major issue of this paper lies in its writing:\n\n(1) The word \"alignment\" in this paper is different from the very commonly used \"alignment\" in which LLMs are tuned to refuse to answer harmful questions. This wording issue can cause significant confusion when readers reading this paper. Please consider replacing the word with other choices to better describe the task performed in this paper, e.g., calibration?\n\n(2) The authors are suggested to better clarify their contribution compared to existing literature. In terms of the methodology, how is the algorithm different from the things in [5,16,17] in the reference list? For technical contribution, to set an example, in line 167, it is mentioned that \"The proof of Theorem 3.1 is adapted from [17], and we include it in Appendix A...\". Please consider summarize the adaptation and challenges when deriving Theorem 3.1.\n\n(3) While the title of this paper mentions \"foundation model\", it is not clear how the proposed algorithm is related to foundation models: it seems that the proposed method can also be used in other small-scale models. Alternatively, is there any other better methods for small-scale models than the proposed method which cannot be used for foundation models?\n\n(4) Please define \"power\" in line 172.\n\n(5) Line 255 \"More powerful model enables more powerful selection\": could you provide insights or interpretation on this observation?\n\n(6) Figure 5: XGBoost provides the functionality of computing the importance score for the features used in the regression. In addition to the individual factors considered in logistic regression in Figure 5, could the authors provide more results on how important each features is? The correlation among features may also makes the actual importance of the features when regression on them together.\n\n(7) Non-asymptotic result: Although Proposition 3.3 provides the asymptotic result on the power of the method, since the experiment includes \"How many high-quality samples are needed?\", the authors are also suggested to provide a discussion on the non-asymptotic results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the theoretical rigor and the writing of our paper! Please see below our detailed responses.\n- **Q1: Use of the term “alignment”**. Thank you for pointing out the potential confusion! In fact, our proposed method can also be used for “alignment” in the strict sense: if we define our alignment score to reflect whether an output is harmful, then applying our method is essentially tuning the LLM to refrain from deploying harmful responses with theoretical guarantees. In this sense, we are using the word “alignment” a bit loosely in our paper, since we allow “alignment” to correspond to other criteria depending on the context. To avoid potential confusion, we have included a disclaimer in the introduction explaining the difference in our updated manuscript. We welcome more discussion on the choice of terminology if you have further concerns! \n- **Q2: Clarifying Contributions**. We would like to clarify that the emphasis of the current work is the novel instantiation of conformal selection for reliable use of foundation model outputs. Compared with conformal selection, the major challenge lies in justifying the criteria, formulating the problem, adapting existing techniques to the current setting, and implementing the complete pipeline for practical use. We elaborate on the challenges in the following. \n\n    - The first challenge is justifying what criterion is practically useful. We argue that selecting reliable outputs (similar to the “prediction with abstention” idea) with FDR control is a practically reasonable criterion, since it ensures that most of the deployed outputs (such as the radiology reports that are handed over to doctors) are indeed reliable and useful. This contrasts our work with the works that apply conformal prediction to standard classification/regression tasks. Particularly in the context of LLM/foundation models, our proposal differs substantially from conformal language modeling [3] that generates multiple outputs for a unit, ensuring *at least* one of the outputs is aligned, and conformal factuality [4] that edits the original outputs to make it less specific/informative, so as to achieve guarantees on the correctness of the output for a single unit. More comparisons with the works connecting conformal prediction and LLM/foundation models can be found in Section 2.1 of the manuscript.\n\n    - The second challenge is formulating the alignment problem as a selective prediction problem that can be tackled with conformal selection, and adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current applications in job hiring and drug discovery apply to numerical outputs and predictions, rather than aligning foundation model outputs (which do not come with a clear numerical output). To make conformal selection applicable to our purpose, we need to formulate alignment as a numerical indicator that could be inferred with reference information, and then decide how to train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Indeed, how conformal prediction ideas may be used to calibrate LLMs/foundation models (beyond traditional numerical outputs) is a nontrivial problem that has attracted considerable recent interest; in this sense, we contribute to the discussion on this important issue. \n\n    - With the general recipe, the third important challenge and contribution is the practical instantiation of the pipeline, which features to use, what model to train for alignment, among other considerations. In this regard, we conduct extensive numerical experiments, finding that using a lightweight prediction model such as logistic regression or random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides a concrete and easy-to-use pipeline for practical use. Indeed, even beyond the current setting of selecting reliable outputs, we view the choice of features and prediction models as contributions that may be helpful for predicting the reliability of outputs for other purposes.\n\n\n- **Q3: Small model/foundation model**. You are absolutely correct that our method is not limited to foundation models, and can be generally applicable to problems of any size. The paper is titled “foundation model” because this is the primary question we find important and aim to address in this paper; in particular, substantial work is devoted to the instantiation of our framework for the alignment of foundation model outputs, e.g., the choice of criterion, building models for predicting the alignment score, among others. \n\n   For smaller base models, we might be able to consider more complicated models for training $g(\\cdot)$; for example, we may include all raw features for training $g$.This technique might be prohibitively expensive for foundation models.\n\n- **Q4: Definition of power**. Thank you for the suggestion! The definition of power was given in the equation between lines 96 and 97. To improve clarity, we have added a reminder of the definition of power in Proposition 3.3 in the revision.\n\nPlease see our response to the remaining question in the Official Comment.\n\nWe sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CqM3YmpdeZ", "reviewer": "Reviewer_Ctqb", "summary": "This paper introduces Conformal Alignment, a framework designed to ensure that outputs from foundation models align with human values before deployment in high-stakes tasks, such as radiology report generation. By training an alignment predictor with reference data, the framework guarantees that a user-specified fraction of selected units meet the alignment criterion. Demonstrated through applications in question answering and radiology report generation, this method effectively identifies trustworthy outputs with lightweight training over a moderate amount of reference data.", "questions": "1.\tCould you specify the new contributions of your work compared to [17]? It seems Theorem 3.1 and Algorithm 1 closely resemble those in [17], with the primary change being the replacement of the monotone nonconformity score V with an alignment score predictor. Can you elaborate on any additional novel aspects?\n\n2.\tCould you please explain the importance and implications of Proposition 3.3 in your framework?\n\n3.\tConsider evaluating your method on datasets with actual human ratings to better validate the claim that your framework ensures alignment with human values. \n\n4.\tCould you please extend the experimental comparisons to include other relevant methods, such as those in [16] and [17]?\n\n5.\tThis paper suggested |D|=500. However, if genuine human values are used to train the alignment predictor, this number of human annotations is not small for radiology report generation.  How do you plan to address this challenge?\n\n6.\tIn Conclusion, it is mentioned that the proposed solution is more relevant to downstream tasks than existing ideas. Could you please elaborate this?", "limitations": "Not exactly although some future works were mentioned", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper studied an important and relevant research problem about when to trust the output of foundation models before deploying them for downstream tasks.\n2.\tThis proposed solution is demonstrated by two applications: question and answering and radiology report generation.\n3.\tThe results show an achievement of strict error control through post-hoc adjustment/selection.", "weaknesses": "The paper's technical contribution seems modest, with some ambiguity in distinguishing its novelty from [17], a potentially overstated claim regarding model alignment with human values, and limited comparisons to relevant works.\n\n1.\tThe technical contribution of the paper appears incremental. As claimed, the main achievement lies in its strict error control through post-hoc adjustment/selection, rather than developing new training strategies to improve specific alignment metrics of the model. However, its theoretical foundation closely resembles that of [17]. For instance, Theorem 3.1 is a modification of a theorem in [17], and Algorithm 1 is also very similar to an algorithm in [17]. The paper lacks clarification on what constitutes the new contributions compared to [17]. It seems that this work merely replaces the monotone nonconformity score V in [17] with a predicted alignment score. Is that right? Additionally, the paper is not well-written, which hampers the understanding of its contributions.\n\n2.\tThe proposed Conformal Alignment claims to ensure that outputs from foundation models align with human values. However, the evaluation on the radiology report generation task does not support this claim. The alignment scores used to train the score predictor are based on the CheXbert score for 14-class classification, which does not strictly align with human ratings ([R1] Figure 3). The authors should consider using datasets with genuine human ratings, like those in [R1], for verification. Otherwise, this claim seems overstated.\n[R1] “Evaluating progress in automatic chest x-ray radiology report generation”. Patterns, 2023\n\n\n3.\tThe proposed method was not compared to any relevant works like [16] and [17], except for a heuristic baseline in Figure 4. Without such comparisons, it is difficult to position the proposed method within the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the importance and relevance of the research question, the applicability and rigor of our framework!\n- **Q1: Comparison with related works**. We would like to clarify that the emphasis of the current work is the novel instantiation of conformal selection for reliable use of foundation model outputs, achieving a practically meaningful guarantee that has not been studied in this thread. While we do not aim to improve the foundation model itself, there are unique challenges in instantiating conformal selection for the current setting.\nIn the following, we compare our work with (1) conformal selection [8, 9], and (2) other works that apply uncertainty quantification to LLMs (such as [3,4]). \n    - Compared with conformal selection, we formulate the current problem into a form that can be tackled by conformal selection, and propose practical techniques to make the application useful. More specifically, the major challenge lies in justifying the criteria, formulating the problem, adapting existing techniques to the current setting, and implementing the complete pipeline for practical use. Please see our response to [R1] for a more detailed explanation of these challenges and contributions.\n    - Compared with other works on conformal prediction and LLMs, the major distinction lies in what criterion we aim to achieve (and subsequently, how we achieve them). We argue selecting reliable outputs (similar to the “prediction with abstention” idea) with FDR control, which ensures that most of the deployed outputs (such as the radiology reports that are handed over to doctors) are indeed reliable and useful. This contrasts our work with the works that apply conformal prediction to standard classification/regression tasks. \n    - Particularly in the context of LLM/foundation models, our proposal differs substantially from conformal language modeling [3] that generates multiple outputs for a unit, ensuring *at least* one of the output is aligned, and conformal factuality [4] that edits the original outputs to make it less specific/informative, so as to achieve guarantees for a single unit. More comparisons with the works connecting conformal prediction and LLM/foundation models can be found in Section 2.1 of our manuscript.\n    We also appreciate the comment on the quality of paper writing. We have made efforts to clarify multiple points other reviewers have suggested. In addition, we would be more than happy to address any specific comments the reviewer may have. \n- **Q2: Evaluation with real human ratings**. Thank you for pointing out the potential discrepancy between CheXbert score and the real human rating! In our manuscript, the cheXbert score can be viewed as a proxy for the human rating. Our framework can similarly be applied to the alignment scores given by human evaluation; however, due to limited resources of such data, we shall leave this for future work. (We note that the data from [R1] are human annotations that compare generated reports from a given model against radiology reports in the CXR dataset we used. The total sample size is only 50; while it is possible to train a prediction model with such data (in practice, we only need the training and calibration data to be labeled), it is difficult for us to split out another fold of test data to complete the evaluation process with this dataset. Also, it is difficult to increase the available sample size due to its human annotation nature. Given this limitation, we were unable to directly use the resources in [R1] to validate our framework.) The use of CheXbert score is mainly to demonstrate the general applicability of our framework to any alignment criterion; thus, switching to real human ratings does not affect the validity., In the revision, we have modified our claim, acknowledging that there might be the discrepancy between the proxy and scores by real humans, and discuss human rating as an important future direction. \n- **Q3: Extending the experimental comparison to Jin and Candès [8, 9]**. As mentioned earlier, our current work is an instantiation of the conformal selection framework in the context of LLM alignment; that is, the method being implemented in our experiments is essentially the (adaptation of) the method proposed in [8]. Thus, we may not be able to compare with them in experiments. (However, we do note that this adaptation presents its unique challenges; for a detailed discussion on such challenges, please refer to our answer to [Q1 of Reviewer1]). We would be happy to compare with other candidate methods if you are referring to other methods.\n\nPlease see the response to the remaining questions in the Official Comment. We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!\n\n**[R1]** “Evaluating progress in automatic chest x-ray radiology report generation”. Patterns, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oi7dxARWch", "reviewer": "Reviewer_SzCp", "summary": "The paper proposes to adapt conformal guarantees to large language models. They build upon existing work to provide bounds on FDR error for each model output by conformalizing the distribution of an additional alignment predictor trained on labeled aligned data. Experiments are conducted on two radiological report generation and question answering using pretrained LLM models.", "questions": "- The term \"unit\" should be explicitly defined before using throughout the paper\n- How is alignment with human experts typically measured? How to generate ground-truth alignment scores? More discussion should be included\n- \"can be gravy\" --> \"can be grave\"\n- In line 114, what, specifically, is the \"selection bias\"?\n-  In line 119, please cite the \"Two works addressing the alignment of foundation models with CP ideas have been briefly discussed in Section 1.\"\n- In line 131, what is \"mild\" about the exchangeability condition and how does it differ from the conventional conformal prediction assumption?\n- In line 137, please provide the citations for the \"commonly-used alignment criteria:\n- How would this framework handle distribution shifts or in-context learning? \n- Can a reward model such as in RLHF be used as the alignment predictor? How does alignment relate to human preferences?\n- Did you try varying $M$ outputs during the training of the alignment predictor?", "limitations": "* How to select an appropriate alignment function? More discussion should be provided as it is the core component of the method\n* How to generalize beyond calibrating single tasks to multiple tasks? A user generally queries a foundational model for many different types of tasks, not only a single task. \n* How to handle dependence between inputs e.g. a chatbot's output depends on prior context and history. How would this be incorporated?\n* The paper would benefit from a dedicated limitations section.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper addresses an important topic, is well-written, and effectively contextualizes related work\n* The proposed method is easy drop-in for most systems and has wide utility in downstream applications\n* The guarantee offered is applicable to free text outputs and flexible in types of alignment functions", "weaknesses": "* The technical novelty of the method seems limited as it seems to be a direct application of prior work for LLM's https://arxiv.org/abs/2210.01408 (more discussion of the differences would be helpful in clarifying technical novelty)\n* There is a lack of comparison to other LLM conformal methods. Providing a few empirical comparisons against other work in uncertainty quantification and alignment would help demonstrate the advantages of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the importance of this topic, the writing quality, and the flexibility and versatility of our framework! We respond to your insightful questions in the following.\n\n- **Q1: Technical novelty**. Thank you for the question! While this work applies the conformal selection framework, this presents unique challenges in (1) justifying the criterion for reliability in LLMs/foundation models, (2) formulating the problem into a form that can be tackled by conformal selection, and (3) implementation techniques such as feature selection, alignment prediction training. Please refer to our response to [Q1 of R1] for a more detailed discussion.\n\n- **Q2: Comparison with other LLM conformal methods**.\n    - Thanks for this insightful question. It is worth differentiating the goals of existing LLM conformal papers. We mainly focus on two examples: [CLM, 3] and [conformal factuality, 4].\n        - [3] focuses on the generating process of LLM. In other words, CLM utilizes conformal risk control to determine the stop time for answer generating and for each question. It produces a set of answers that may contain the correct answer (under certain metrics) with a high probability. \n        - [4] focuses on modifying the generated answers to achieve factuality on average. \n        - In comparison, our approach focuses on the “black-box” scenario where we already have access to the inputs and outputs of foundation models, in which our task is to identify a subset of outputs that align with human-verified answers and can be used directly.\n\n    - The major distinction between our method and these existing works is the type of guarantee we pursue. This is related to the following question: how can we use the outputs of these algorithms in practice? In short, \n        (a) For CLM, it is unclear how to pick one answer from the multiple answers in a way that does not hinder validity.\n        (b) Conformal factuality makes the original outputs more vague, which can hinder downstream use. \n\n        - Note that point (b) has recently been noticed (after the submission of our paper) in the literature [12], hence we refer to certain recent works that demonstrate this point for simplicity. In contrast, we make no modification to the model outputs but instead filter out undesirable ones; thus, the quality of good outputs is not affected.\n\n        - In addition, to demonstrate (a), we conducted additional experiments for more clear comparison. With the TriviaQA dataset and the base model LLaMA-2-13B-chat, we follow the procedure in [3] and have the following examples:\n            - Example 1: Question: In tennis, losing two sets 6-0 is known as a double what? True answer: Bagel. Generated set: {‘bagel’}.\n            - Example 2: Question: What was the name of the brothel in The Best Little Whorehouse in Texas? True answer: Chicken ranch. Generated set: {'miss monas', 'chicken ranch', 'miss mona’s', 'miss monas bordello'}\n            As we can see in Example 1, the generated set is very informative and can be used directly. However, although the generated set in the second example has a high probability of covering the true answer, it contains very different answers and is potentially confusing in practice. In other words, whether “Chicken ranch” or “Miss Monas” is correct is unclear to the user.  \n            In addition, as is shown in the attached Figure 2 (left panel), we present the proportion that the size of generated size is k for different values of k, which shows that there are more than 40% of sets having no less than 4 distinct items. In addition, Figure 2 (right panel) shows that in nearly 30% of the questions, the admissible answers (with rougeL >= 0.3) take up less than 20% of the uncertain set, which poses a challenge in the direct deployment of answers in the uncertain set.\n\n- **Q3: Definition of “unit”**.\nThank you for raising this issue and we apologize for potential confusion in the usage of “unit”. We use each “unit” to broadly refer to a data point (a question in Q&A task, and a radiology instance in report generation). In our revision, we have clarified the definition of a “unit” in the introduction.\n- **Q4: Measurement of alignment**. Thank you for the thoughtful question! Indeed, an important problem is the acquisition of reference information to compute the true alignment score.  In general, the alignment criterion depends on the context and can be freely chosen by the practitioner. Currently, for consistency, our experiments with Q&A and radiology report generation follow existing definitions of “acceptable/reliable” outputs, such as factuality (correctness of answers) and consistency with human reports [3,4]. These alignment scores can be computed with existing datasets, yet we expect other notions of alignment that may suit other application contexts. If one is to deploy our method in a new context, they can use common criteria or let humans rate the alignment if feasible. We will add a discussion on this point in our updated manuscript. \n- **Q5: Wording**. Thank you for the detailed suggestions! We’ll modify the wording in our updated draft. Please consider all of them fixed!\n\n- **Q6: Exchangeability**. Our exchangeability condition needs the calibration and test data to be exchangeable, which is identical with the conventional conformal prediction assumption. \n- **Q7: Citations**. We have added the citations of [3,4] in line 119 and [5,10,11] in line 137.\n\nPlease see our response to the remaining questions in the Official Comment.\n\nWe sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!\n\n**Reference** Please see references in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback and constructive suggestions for improvement. Overall, the reviewers find our work addresses an important problem [R1, R3, R4, R5], is well-written [R1, R5], versatile to broad contexts and criteria [R1, R2, R5], offers rigorous and practically useful guarantees [R1, R3, R4], and offers extensive experimental results [R1, R2, R5]. 

A few stimulating questions raised by the reviewers have motivated us to conduct the following additional numerical experiments (please see the attached pdf file), which add to the empirical evidence and the paper:

- We extended our empirical analysis of feature importance. 
- We used real experiments to demonstrate the superiority of FDR control over other guarantees in the literature in terms of relevance to downstream tasks.

We also added more discussion in response to the reviewers’ important suggestions for clarification and further discussion, including 

- unique challenges and contributions relative to conformal selection
- factors that may affect the power of our methods, and non-asymptotic power analysis 
- potential extension to distribution shift, in-context learning, multi-task setting, and conversational settings with dependence
- potential extension to improve power and efficient data usage
- practical choice of the alignment score and predictor

Please see our point-by-point response to all questions raised by each reviewer’s comments below. 

**Reference**

1. Angelopoulos, Anastasios N., et al. "Learn then test: Calibrating predictive algorithms to achieve risk control." arXiv preprint arXiv:2110.01052 (2021).
2. Angelopoulos, Anastasios N., et al. "Conformal risk control." arXiv preprint arXiv:2208.02814 (2022).
3. Quach, Victor, et al. "Conformal language modeling." arXiv preprint arXiv:2306.10193 (2023).
4. Mohri, Christopher, and Tatsunori Hashimoto. "Language models with conformal factuality guarantees." arXiv preprint arXiv:2402.10978 (2024).
5. Smit, Akshay, et al. "CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using BERT." arXiv preprint arXiv:2004.09167 (2020).
6. Varshney, Neeraj, Swaroop Mishra, and Chitta Baral. "Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings." arXiv preprint arXiv:2203.00211 (2022).
7. Kumar, Bhawesh, et al. "Conformal prediction with large language models for multi-choice question answering." arXiv preprint arXiv:2305.18404 (2023).
8. Jin, Ying, and Emmanuel J. Candès. "Selection by prediction with conformal p-values." Journal of Machine Learning Research 24.244 (2023): 1-41.
9. Jin, Ying, and Emmanuel J. Candès. "Model-free selective inference under covariate shift via weighted conformal p-values." arXiv preprint arXiv:2307.09291 (2023).
10. Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. "Generating with confidence: Uncertainty quantification for black-box large language models." arXiv preprint arXiv:2305.19187 (2023).
11. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023.
12. Cherian, John J., Isaac Gibbs, and Emmanuel J. Candès. "Large language model validity via enhanced conformal prediction methods." arXiv preprint arXiv:2406.09714 (2024).

Meta-review: This work addresses an important problem on the emerging topic of foundation models (and more particularly related to large-language models). Despite the reviewers raised several important points, and some may not be fully addressed in the rebuttal (e.g., novelty with respect to prior works), I believe that the proposed work has some merits, as mentioned by the reviewers. More specifically, the task of conformal prediction has not been explored in LLMs, despite its paramount importance. Furthermore, the experiments are extensive, which add empirical support to the mathematical formulation and theoretical justifications presented in this work. Thus, I recommend the acceptance of this submission.
Decision: Accept (poster)