Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees

Authors: yu  gui
ying  jin
zhimei  ren
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 17:45:13

Abstract: Before deploying outputs from foundation models in high-stakes tasks, it is imperative to ensure that they align with human values.
For instance, in radiology report generation, reports generated by a vision-language model must align with human evaluations before their use in medical decision-making. This paper presents Conformal Alignment, a general framework for identifying units whose outputs meet a user-specified alignment criterion. It is guaranteed that on average, a prescribed fraction of selected units indeed meet the alignment criterion, regardless of the foundation model or the data distribution. Given any pre-trained model and new units with model-generated outputs, Conformal Alignment leverages a set of reference data with ground-truth alignment status to train an alignment predictor. It then selects new units whose predicted alignment scores surpass a data-dependent threshold, certifying their corresponding outputs as trustworthy. Through applications to question answering and radiology report generation, we demonstrate that our method is able to accurately identify units with trustworthy outputs via lightweight training over a moderate amount of reference data. En route, we investigate the informativeness of various features in alignment prediction and combine them with standard models to construct the alignment predictor.

Main paper: 
 introduction: 
 Large-scale, pre-trained foundation models are remarkably powerful in generating relevant and informative outputs of various forms for diverse downstream tasks, marking a new era of artificial intelligence [34, 57, 47] . However, it has been recognized that they can be prone to factual errors [43] , hallucinations [15, 51] , and bias [4] , among others. These issues raise prevalent societal concerns regarding the reliable use of foundation models in high-stakes scenarios [7, 6] . It remains a significant challenge to ensure that outputs from these highly complex models align with human values [36] .Towards uncertainty quantification of black-box models, conformal prediction [50, CP] offers a versatile, distribution-free solution. While CP has been applied to classification and regression tasks addressed by foundation models [25, 27, 46, 48, 54, 2] , its use to ensure the alignment of general-form outputs remains largely unexplored. Recently, [36] propose to generate multiple outputs for one input until it is guaranteed that at least one output meets a specific alignment criterion. Such a guarantee, however, may not be immediately practical, as users must still determine (manually) which output(s) is aligned. [32] modify the machine-generated outputs to ensure factuality with high probability, which in turn may sacrifice model power by making responses vague. 3 As such, determining guarantees that are practically meaningful for downstream tasks and developing methods to achieve them remain an active research area. This paper introduces a distinct type of guarantee: we aim to certify whether each model output is aligned or not, such that those certified are ensured to be mostly correct. This guarantee is directly relevant to downstream tasks, as it allows immediate and reliable use of the certified outputs without further modifications. On the other hand, we abstain from deploying model outputs for units (data points) that are not selected-intuitively, those are where the model lacks confidence (i.e., "knowing they do not know"), and they can be evaluated by human experts. Such a practice therefore provides an automated pipeline for the safe deployment of model-generated outputs.Present work. We present Conformal Alignment (CA), a general framework that determines when to trust foundation model outputs with finite-sample, distribution-free guarantees. The framework is based on CP ideas, but unlike standard CP that searches the sampling space to construct a prediction set that contains a desired output, we leverage the quantified confidence as an instrument to select units whose already-generated outputs are trusted. Given any model and any alignment criterion, Conformal Alignment ensures that a prescribed proportion of the selected units' model outputs indeed meet the alignment criterion. 4 Concretely, suppose m ∈ N + new units awaiting outputs. Given an error level α ∈ (0, 1), Conformal Alignment controls the false discovery rate (FDR) in selecting units with trustworthy outputs:EQUATIONwhere the expectation is taken over the randomness of the data. FDR control offers an interpretable measure of the quality of selected deployable units. Our method builds upon the Conformalized Selection framework [18, 17] , leveraging a holdout set of "high-quality" reference data to guide the selection with calibrated statistical confidence for trusted outputs. Like in CP, (1.1) holds in finite-sample as long as the holdout data are exchangeable with the new units.Our method inherits CP's (1) rigor, providing distribution-free, finite-sample error control, and (2) versatility, applicable to any model and any alignment criterion. In addition, by selecting rather than modifying outputs, our approach preserves the (3) informativeness of the original outputs, and remains (4) lightweight, e.g., avoiding the need to retrain large models.Demonstration of workflow. We illustrate the pipeline of Conformal Alignment via a use case in radiology report generation (detailed in Section 5) in Figure 1 : each prompt corresponds to an X-ray scan, for which a vision-language model generates a report. Our framework identifies a subset of generated reports that are reliable in the sense that at least, say 99%, of the selected reports are guaranteed to be aligned if they were to be compared with a human expert report.The workflow begins with (test) prompts D test , corresponding to X-ray scans X ∈ X . A foundation model f : X → Y generates a report f (X) ∈ Y for each scan. However, only high-quality reports should be handed over to doctors for clinical decisions. To achieve this, we leverage a set of X-ray scans that are independent of the model's training process, each with human expert reports available. These scans are randomly split into a training set D tr and a calibration set D cal . Conformal Alignment trains a predictor on D tr for predicting the alignment score that assess how likely a generated output aligns. Finally, new reports are selected if their predicted alignment scores exceed a data-driven threshold, which is delicately set with D cal such that the FDR is strictly controlled at the desired level.Practical relevance. We apply Conformal Alignment to two use cases-question answering and radiology report generation-to demonstrate its efficiency. With quite lightweight training of the alignment predictor (for example, training a tree-based model would suffice), we are able to accurately identify trustworthy outputs and select as many of them as possible. En route, we exhaustively investigate practical aspects such as the efficacy of various predictors in these applications and the amount of "high-quality" data needed for accurate selection to guide practical deployment. M I K 1 k X r O x Q D z e I K H x R K y n W q l V m t A Z F f d h o v q h i D U c M t l 6 B i S o Q R W a A 2 L 7 4 N R R J K Q C k 0 4 V q r v o F h 7 K Z a a E U 7 n h U G i a I z J F I 9 p 3 1 C B Q 6 q 8 d H H v H J 4 Z Z Q S D S J o S G i 7 U 7 x M p D p W a h b 7 p D L G e q N 9 e J v 7 l 9 R M d u F 7 K R J x o K s h y U Z B w q C O Y P Q 9 H T F K i + c w Q T C Q z tm i V O Q K i r Q 8 M 0 v t x e R J K Q C C M d K d R 0 7 B i / F E h j h N K u 4 i a I x J i M 8 o F 1 N B Q 6 p 8 t L J D Z l 1 q J W e 1 Y + k f g K s i f q 7 I 8 W h U u M w 0 J X 5 x m r W y 8 X / v G 4 C / Q s v Z S J O g A o y H d R P u A W R l Q d i 9 Z i k B P h Y E 0 w k 0 7 t a Z I g l J q B j q + g Q n N mp K J G E N k n E I 9 k J s K K c C d o E B p x 2 Y k l x G H D a D k Z X u d + + p 1 K x S N z B O K Z e i A e C 9 R n B o C X f 3 H N D D E O C e X q d + a k L 9 A F S k J i J L P P N q l 2 z J 7 D m i V O Q K i r Q 8 M 0 v t x e R J K Q C C M d K d R 0 7 B i / F E h j h N K u 4 i a I x J i M 8 o F 1 N B Q 6 p 8 t L J D 5 l 1 q J W e 1 Y + k L g H W R P 0 9 k e J Q q X E Y 6 M 7 8 Y j XU F r E 0 Q X 2 E 2 i F r B e J r u A M j d S 2 1 5 s h w y v O Z H a a t 7 M m w i 1 m C B r z v G 2 X X M f z f c / f o a 6 z 5 x + W v V 1 D y u b f d 6 n n u E O U y B i V t v 3 e 7 M Q 8 D S F C L p n W D c 9 N s J U x h Y J L y I v N V E P C + D X r Q c P Q i I W g W 9 n w h J x u G a V D u 7 E y L 0 I 6 V L 9 3 Z C z U u h 8 G p n K w s P 7 t D c S / v E a K 3 Y N W J q I k R Y j 4 a F A 3 l R R j O s i D d o Q C j r J v C O N K m F 0 p v 2 K K c T S p F U 0 I X 5 f S / 0 l t x / H K j n + + V z o 6 H s d R I B t k k 2 w T j + y T I 3 J G K q R K O L k j D + S J P F v 3 1 q P 1 Y r 2 O S i e s c c 8 a + Q H r 7 R M r 0 5 i d < / l a t e x i t > D testUse LLM report?Use LLM report?Use LLM report?< l a t e x i t s h a 1 _ b a s e 6 4 = " + g Z Q Q n e Y t L b m J P 7 4 X 2 Q n 7 7 k n 7 m s = " > A A A C A n i c d V D L S g N B E J z 1 G e N r 1 Z N 4 G Q y C p 2 V X 3 a g 3 U Q 8 e I 5 h E S E K Y n X T i 4 O y D m V 4 x L I s X f 8 W L B 0 W 8 + h X e / B s n D 0 F F C 4 Y p q r r p 7 g o S K T S 6 7 o c 1 M T k 1 P T N b m C v O L y w u L d s r q z U d p 4 p D l c c y V p c B 0 y B F B F U U K O E y U c D C Q E I 9 u D 4 Z + P U b U F r E 0 Q X 2 E 2 i F r B e J r u A M j d S 2 1 5 s h w y v O Z H a a t 7 M m w i 1 m C B r z v G 2 X X M f z f c / f o a 6 z 5 x + W v V 1 D y u b f d 6 n n u E O U y B i V t v 3 e 7 M Q 8 D S F C L p n W D c 9 N s J U x h Y J L y I v N V E P C + D X r Q c P Q i I W g W 9 n w h J x u G a V D u 7 E y L 0 I 6 V L 9 3 Z C z U u h 8 G p n K w s P 7 t D c S / v E a K 3 Y N W J q I k R Y j 4 a F A 3 l R R j O s i D d o Q C j r J v C O N K m F 0 p v 2 K K c T S p F U 0 I X 5 f S / 0 l t x / H K j n + + V z o 6 H s d R I B t k k 2 w T j + y T I 3 J G K q R K O L k j D + S J P F v 3 1 q P 1 Y r 2 O S ij p O F a E t E v N Y d U O s K W e S t o A B p 9 1 E U S x C T j v h + H 7 u d y Z U a R b L J 5 g m N B B 4 K N m A E Q x G 6 t t 2 x R 9 h 8 E V a 9 U k U w 2 W l b 5 f d m r u A s 0 6 8 n J R R j m b f / v K j m K S C S i A c a 9 3 z 3 A S C D C t g h N N Z y U 8 1 T T A Z 4 y H t G S q x o D r I F p f P n A u j R M 4 g V q Y k O A v 1 9 0 S G h d Z T E Z p O g W G k V 7 2 5 + J / X S 2 F w G 2 R M J i l Q S Z a L B i l 3 I H b m M T g R U 5 Q A n x q C i W L m V o e Mp K J G E N k n E I 9 k J s K K c C d o E B p x 2 Y k l x G H D a D k Z X u d + + p 1 K x S N z B O K Z e i A e C 9 R n B o C X f 3 H N D D E O C e X q d + a k L 9 A F S k J i J L P P N q l 2 z J 7 D m i V O Q K i r Q 8 M 0 v t x e R J K Q C C M d K d R 0 7 B i / F E h j h N K u 4 i a I x J i M 8 o F 1 N B Q 6 p 8 t L J D 5 l 1 q J W e 1 Y + k L g H W R P 0 9 k e J Q q X E Y 6 M 7 8 Y j Xm i V O Q K i r Q 8 M 0 v t x e R J K Q C C M d K d R 0 7 B i / F E h j h N K u 4 i a I x J i M 8 o F 1 N B Q 6 p 8 t L J D Z l 1 q J W e 1 Y + k f g K s i f q 7 I 8 W h U u M w 0 J X 5 x m r W y 8 X / v G 4 C / Q s v Z S J O g A o y H d R P u A W R l Q d i 9 Z i k B P h Y E 0 w k 0 7 t a Z I g l J q B j q + g Q n N m 
problem setup: 
 Suppose we have a pre-trained foundation model f : X → Y that maps a prompt to an output. Throughout, we view the model as given and fixed. Assume access to a set of holdout units D = (X i , E i ) n i=1 , where X i ∈ X is the input prompt and E i ∈ E is any reference that may be used in judging alignment. An alignment function A : Y × E → R takes as input the generated output f (X) and the reference E, and outputs an (true) alignment score A = A(f (X), E) [36] . In the radiology report generation example, A might be the similarity score between the machine-generated report f (X) and a human expert report E. We shall detail the choice of A in each use case. Note that for a unit without reference information, the alignment score of its generated output is unknown, and we do not seek to evaluate it, which often requires expert annotation or human judgment.For a test set D test = {X n+j } m j=1 , we aim to select a subset S ⊆ [m] := {1, . . . , m} such that most of their (unobserved) alignment scores exceed a pre-fixed threshold c ∈ R. The error is measured byFDR = E j∈[m] 1{A n+j ≤ c, j ∈ S} max(|S|, 1) ,where | • | denotes the cardinality of a set. In particular, we aim to enforce that FDR ≤ α for a pre-specified level α ∈ (0, 1). The FDR measures the averaged proportion of selected units that are not aligned, thereby directly quantifying the "cost" of deploying outputs in S. Such a measure (under different terminologies) appears to be considered empirically in selective prediction, although a rigorous control has been lacking therein (see e.g., [52, 49, 37] ). Besides FDR control, it is also desirable that as many units can be safely deployed, which translates into maximizing the power:EQUATIONIt should be emphasized that our framework prioritizes FDR control over power, in that we strictly enforce the former while optimizing the latter under the constraints. This is related to but slightly differs from the selection with controlled risk setting in selective prediction [11, 12] ; see Section B.1 for more details. Such a setup is motivated by scenarios such as medical decision-making and knowledge access, where the cost of committing a type-I error (i.e., selecting a non-aligned unit) can be grave, and should be of primary concern. In addition, we define power as the proportion among truly aligned units, instead of the number of selections-indeed, if a foundation model is not powerful in generating aligned outputs, to begin with, it is natural that only a few of the outputs shall be deployed.Uncertainty quantification and conformal prediction for foundation models. Conformal prediction [50, 26, 41] is a distribution-free framework for uncertainty quantification of generic prediction algorithms. This work adds to the growing literature that applies/extends CP to foundation models, which varies in definitions of uncertainty and alignment, types of guarantees, rules for decisionmaking, etc. Below, we summarize representative ideas and contrast them with the present work.A line of work applies CP to construct prediction sets that cover the outcome for a single test unit in classification or regression problems addressed by language models [25, 27, 46, 48, 54] . While desired, their reliability in downstream tasks is only empirically evaluated: we note that coverage of prediction sets does not carry over to the selected units [18, 19] . In contrast, this work applies to outputs of general forms, considers multiple units simultaneously, and offers guarantees relevant to downstream tasks. In particular, our methodology is built upon the series of papers in conformalized selection [18, 17] . While they aim to find large values of outcomes, we motivate and tailor the framework for the alignment of foundation models (a more detailed discussion is in Appendix B.2).Two works [36, 32] addressing the alignment of foundation models with CP ideas have been briefly discussed in Section 1. Our setup draws ideas from [36] , but the guarantees we provide differ from both. In particular, our method may be better suited to situations where it is desirable to avoid modifying the outputs or enlarging candidate sets which requires retraining the models.More generally, [24, 29, 16] investigate the metric of uncertainty for natural language outputs. This is orthogonal to the present work, as our method is compatible with any alignment criterion. Nevertheless, our work adds to the discussion on desirable uncertainty quantification guarantees.Selective prediction. Our framework is closely related to the idea of selective prediction, where one is allowed either to predict or to abstain from making a prediction [9, 11, 12, 33, 1] . In the context of large language models, [23, 53, 40, 55] have similarly considered the goal of teaching the model not to predict when the model is uncertain about its output, but theoretical guarantees on the alignment of the selected outputs are lacking. In contrast, this work rigorously formalizes the guarantee and provides an end-to-end framework that achieves it under a mild exchangeability condition. As mentioned earlier, the strict control of FDR is closely connected to the "selection with controlled risk" setting in selective prediction [11, 12] , which aims to control a slightly different error measure; we provide a detailed comparison of the error measures and methods in Appendix B.1.There is a rapidly growing literature on the alignment of foundation models, wherein topics include alignment evaluation [30, 42] , prompt design [22] , training with human feedback [3, 35, 38] , among others [8] . We rely on commonly-used alignment criteria in our applications [44, 29, 24] . However, we achieve strict error control via post-hoc adjustment/selection, instead of deriving training strategies to improve certain alignment metrics of the model. 
conformal alignment: 
 Given a set of data D with reference information, our procedure starts by splitting D into two disjoint sets: the training set D tr and the calibration set D cal . With a slight abuse of notation, we shall also use D tr and D cal to denote the indices of the units in the sets when the context is clear. We then fit a model g : X → R on D tr to predict the alignment score based on X (which may also involve information from f ). Next, we generate model outputs f (X i ) and compute the predicted alignment scoresA i = g(X i ) for every i ∈ [n + m]. We shall use (A i , A i ) i∈Dcal to determine the selection set.Following the framework of conformalized selection [18, 17] , we gather statistical evidence for trustworthy outputs via hypothesis testing, where the null hypothesis for j ∈ [m] isH j : A n+j ≤ c. (3.1)Rejecting H j then reflects evidence that the (true) alignment score of unit j is above the threshold c, and therefore the generated output f (X n+j ) is aligned. Under this framework, the task of selecting aligned units boils down to simultaneously testing the m hypotheses specified in (3.1) . For this purpose, we construct the conformal p-value: for any j ∈ [m],p j = 1 + i∈Dcal 1{A i ≤ c, A i ≥ A n+j } |D cal | + 1 . (3.2)It can be shown that when the test unit is exchangeable with the calibration units, the p-value defined above is valid in the sense that P(p j ≤ t, A n+j ≤ c) ≤ t for any t ∈ (0, 1) [18] . Intuitively, when a generated output is likely to be aligned, we expect A n+j to have a large magnitude, and p j to be small. As a result, we reject the null hypothesis-that is, declaring a sufficiently large alignment score-for a small p j , where the threshold for p-values is determined by the Benjamini-Hochberg (BH) procedure [5] . Concretely, let p (1) ≤ . . . p (m) denote the ordered statistics of the p-values; the rejection set of BH applied to the conformal p-values is S = {j ∈ [m] : p j ≤ αk * /m}, wherek * = max k ∈ [m] : p (k) ≤ αk m ,with the convention that max ∅ = 0. We describe the complete procedure in Algorithm 1, and establish its validity in Theorem 3.1. For notational simplicity, we denote Z i = (X i , E i ) for all i ∈ [n + m] (note that E i is not observable for i > n). Algorithm 1 Conformal Alignment Input: Pre-trained foundation model f ; alignment score function A; reference dataset D = (X i , E i ) n i=1 ; test dataset D test = (X n+j ) m j=1 ; algorithm for fitting alignment predictor G; align- ment level c; target FDR level α. 1: Compute the alignment score A i = A(f (X i ), E i ), ∀i ∈ D.A i ← g(X i ), ∀i ∈ D cal ∪ D test . 5: for j ∈ [m] do 6:Compute the conformal p-values p j according to Equation (3.2). 7: end for 8: Apply BH to the conformal p-values: S ← BH(p 1 . . . , p m ). Output: The selected units S. Theorem 3.1. Suppose that for any j ∈ [m], {Z n+j } ∪ {Z i } i∈Dcal are exchangeable conditional on {Z n+ℓ } ℓ̸ =j , i.e., for any permutation π of {1, . . . , n, n + j} and any {z 1 , . . . , z n , z n+j }, it holds thatP Z 1 = z 1 , . . . , Z n = z n , Z n+j = z n+j | {Z n+ℓ } ℓ̸ =j = P Z 1 = z π(1) , . . . , Z n = z π(n) , Z n+j = z π(n+j) | {Z n+ℓ } ℓ̸ =j .Suppose the predicted alignment score { A i } i∈Dcal∪Dtest have no ties almost surely and sup x |g(x)| ≤ M for some M > 0. Then for any α ∈ (0, 1), the output S from Algorithm 1 satisfies FDR≤ α.The proof of Theorem 3.1 is adapted from [18] , and we include it in Appendix A for completeness. Remark 3.2. The exchangeability condition required in Theorem 3.1 is satisfied when the samples in D cal ∪ D test are i.i.d., or when D cal is drawn from a set of samples without replacement, or when they are nodes on graphs in the transductive setting [14] . The assumption of no ties is without loss of generality, since one can always add a small random noise to break the ties.The following proposition characterizes the power (2.1), as well as the fraction of selected units that can be deployed with confidence, when the samples are i.i.d. The proof is in Appendix A.2. A finite-sample power analysis based on concentration inequalities is in Appendix A.3. Proposition 3.3. Under the same condition of Theorem 3.1, assume further that(X i , E i ) i∈[n+m] are i.i.d. Define H(t) = P(A ≤ c, g(X) ≥ t) and t(α) = sup{t : t P(H(g(X))≤t) ≤ α}.Suppose that there exists a sufficiently small ε > 0 such thatt P(H(g(X))≤t) < α for t ∈ [t(α) -ε, t(α)], then lim |Dcal|,m→∞ Power = P(H(g(X)) ≤ t(α) | A > c). lim |Dcal|,m→∞ 1 m m j=1 1{j ∈ S, A n+j > c} = P(H(g(X)) ≤ t(α), A > c).Figure 2 visualizes the asymptotic cutoff on g(X) beyond which will be selected by our method: it is the smallest value such that the red area on the right is less than α-proportion of red and blue areas combined. The area of the blue part on the right of the cutoff is thus the asymptotic fraction of selected units, whose proportion among the entire blue area is the asymptotic power. Intuitively, given a model (i.e., fixing areas for the red and blue), the power of Conformal Alignment depends on how well g discerns those A > c against those A ≤ c. Therefore, it is important to identify features that informs the alignment of the outputs; we will elaborate on this point and deliver practical recommendations in our experiments in Sections 4 and 5. In addition, the fraction of deployable units (blue) additionally depends on the model power, i.e., the fraction of units whose outputs are indeed aligned. These two factors both affect the fraction of units selected by our method (the blue area on the right of the cutoff), which will also be demonstrated in the experiments. In this section, we implement and evaluate Conformal Alignment in question-answering tasks, where we consider a conversational question answering dataset TriviaQA [21] and a closed-book reading comprehension dataset CoQA [39] . For each QA dataset, we use language models OPT-13B [57] and LLaMA-2-13B-chat [47] without finetuning to generate an answer f (X i ) via top-p sampling for each input X i following the default configuration. The alignment score A i measures how well the generated answer matches the true reference answers provided in both datasets. Following [24, 29] , we let A i ∈ {0, 1} indicate whether the rouge-L score [28] between the LLM-generated answer and reference answer is no less than 0.3, and set c = 0. See Appendix C.1 for more details.Dataset partition. Recall D is the reference dataset and D test is the test set we select from. We fix |D test | = 500 and split D as follows. Fixing γ 1 , γ 2 ∈ (0, 1), γ 1 + γ 2 < 1, we randomly sample ⌊(γ 1 + γ 2 ) • |D|⌋ instances without replacement from D as D tr for training the alignment predictor g. Within D tr , a random subset of size ⌊γ 1 • |D|⌋ is reserved for hyperparameter tuning in computing certain features to be introduced later, while the others are used in fitting g given the features. We then set the calibration set D cal = D\D tr . For the results presented in this section, γ 1 = 0.2, γ 2 = 0.5. Additional ablation studies for the choice of (γ 1 , γ 2 ) are summarized in Section 4.1.Alignment score predictor. With the training set D tr , according to Algorithm 1, our goal is to train an alignment score predictor g that maps X i to A i -the estimated probability for A i > c-based on available information such as the input, model parameters, and outputs. To this end, we train a classifier with binary label A i and the following features (as preparation, we additionally randomly generate 19 answers for each input, leading to M = 20 answers in total for each input):• Self-evaluation likelihood (Self_Eval). Following [22, 29] , we ask the language model itself to evaluate the correctness of its answer and use the likelihood, P(true), as a measure of confidence. • Input uncertainty scores (Lexical_Sim, Num_Sets, SE). Following [24] , we compute a set of features that measure the uncertainty of each LLM input through similarity among the M = 20 answers. The features include lexical similarity (Lexical_Sim), i.e., the rouge-L similarity among the answers. In addition, we use a natural language inference (NLI) classifier to categorize the M answers into semantic groups, and compute the number of semantic sets (Num_Sets) and semantic entropy (SE). Following [24, 29] , we use an off-the-shelf DeBERTa-large model [13] as the NLI predictor. • Output confidence scores (EigV(J/E/C), Deg(J/E/C), Ecc(J/E/C)). We also follow [29] to compute features that measure the so-called output confidence: with M generations, we compute the eigenvalues of the graph Laplacian (EigV), the pairwise distance of generations based on the degree matrix (Deg), and the Eccentricity (Ecc) which incorporates the embedding information of each generation. Note that each quantity is associated with a similarity measure; we follow the notations in [29] and use the suffix J/E/C to differentiate similarities based on the Jaccard metric, NLI prediction for the entailment class, and NLI prediction for the contradiction class, respectively.We use all above features to train the alignment predictor with standard ML models including logistic regression, random forests, and XGBoost. Existing works often use individual features in calibrating model confidence [22, 29, 24] ; in this regard, we evaluate these features by using each individual feature to train the predictor g and reporting the resulting power, which delivers additional insights upon their informativeness in predicting model alignment. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment results: 
 This section reports the results on the TriviaQA dataset, where a subset of 8000 questions are considered, with the alignment predictor trained via logistics regression; additional results when the predictor is trained via random forest and XGBoost are in Appendix D.1. Parallel results on the CoQA dataset with 7700 questions can be found in Appendix D.2. We also provide QA examples in Table 1 in Appendix D.5 to illustrate the performance of our method.Conformal Alignment strictly controls the FDR while heuristic baseline does not. Figure 3 shows the realized FDR and power for Conformal Alignment at target FDR levels α ∈ {0.05, 0.1, . . . , 0.95}, averaged over 500 independent experiments. The FDR curves demonstrate the finite-sample error control. It is worth noting that the FDR is tightly controlled; it becomes a constant for sufficiently large α under which all units can be selected without violating the FDR.To demonstrate the importance of a statistically rigorous treatment, we compare Conformal Alignment with a heuristic baseline, where one asks the language model to evaluate its confidence (i.e., the Self_Eval feature above), and threshold the obtained likelihood {q self j } with a cutoff of 1α as if it were the "probability" of alignment, i.e., S baseline = {j ∈ [m] : q self j ≥ 1 -α}. The resulting FDR and power for the TriviaQA dataset are in Figure 4 . The baseline fails to control FDR, showing that the self-evaluation score is over-confident. In addition, it is also less powerful than our procedure despite the higher error rate, implying that the self evaluation is less informative for predicting alignment. Comparing LLaMA-2-13B-chat and OPT-13B, we also observe that more powerful models may tend to be more (over-)confident in their output. More powerful model enables more powerful selection. The performance of Conformal Alignment also varies with the language models. We observe that more powerful LLMs can generate answers with higher accuracy which further leads to stronger signals in selection. For OPT-13B, the largest value of FDR is 0.5; we have nearly no discoveries and the power is close to zero when α = 0.05. In comparison, with LLaMA-2-13B-chat, FDR never exceeds 0.25, and the power is as high as 0.50 even at an FDR level of α = 0.05. As such, one may also use the FDR/power curve from Conformal Alignment as a measure to compare LLMs' performance.What feature is informative for alignment? In addition to the capability of LLMs, the features used to train g also play a vital role in the power of selection. To compare the informativeness of the aforementioned features for the alignment of outputs, we use one score at a time as the feature in training g and show Power/FDR curves in Figure 5 . When using the OPT-13B model, Self_Eval and Num_Setss are the least powerful, while Deg(J) and Ecc(J) are the most powerful in predicting alignment. Such a comparison is similar for LLaMA-2-13B-chat. We defer the FDR curves to Figure 14 in Appendix D.3 which remain strictly controlled. We include a more detailed analysis on feature importance via Shapley value in Appendix D.6.How many high-quality samples are needed? The size of high-quality samples |D| needed is important for applying Conformal Alignment. In practice, it is desired if a handful of data suffices to accurately identify aligned outputs. To study the effect of reference sample size |D| on the performance, in Figure 3 , we vary |D| in {100, 500, 2000} with fixed γ 1 = 0.2 and γ 2 = 0.5. In general, a few hundred high-quality samples suffice to achieve relatively high power, showing that Conformal Alignment does not require too expensive extra labeling. More specifically, we observe that when |D| increases, there is a slight improvement in power, especially when α is small; for large values of α, the power is robust to |D|. In addition, a larger labeled sample size reduces the variance in both FDR and Power and improves the stability in selection.Ablation study. In Appendix D.4, we present ablation studies for the choice of γ 1 and γ 2 in data splitting. In practice, larger values of γ 1 and γ 2 allow more samples for training the alignment predictor but decrease the resolution of the conformal p-values. With fixed |D| and γ 2 , we observe that the performance of our method is not sensitive to the choice of γ 1 ; in particular, a small value of γ 1 (e.g., 0.2) is sufficient for powerful selection. When fixing |D| and γ 1 = 0.2, we suggest setting γ 2 ∈ [0.3, 0.5] to balance the sample sizes for both the predictor training and the BH procedure.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: please see Section 4, 5 and C. Reproducible codes can be found at https: //github.com/yugjerry/conformal-alignment.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments for chest x-ray report generation: 
 In this section, we apply Conformal Alignment to chest X-ray report (CXR) generation. Following the pipeline in Figure 1 , we apply our method to (a subset of) the MIMIC-CXR dataset [20] . In practice, imagine a healthcare institute employs a foundation model to automatically generate radiology reports based on chest radiographs. Conformal Alignment can be used to determine which reports to trust and refer to doctors for medical decisions, and the rigorous guarantees it offers imply that 1α fraction of the approved reports indeed match the results if they were to be read by a human expert. We obtain the base foundation model f by fine-tuning an encoder-decoder model (pre-trained Vision-Transformer and GPT2) using a separate training set. For a generated report f (X i ) with a reference report written by a radiologist, the alignment score A i is defined as follows: we use CheXbert [44] to convert both reports to two 14-dimensional vectors of binary labels with each coordinate as the indicator of positive observations; we then define A i ∈ {0, 1} as the indicator of whether there are at least 12 matched labels, and set c = 0. We employ the same set of features and the same procedure in Section 4, except for Self_Eval which is not available from the current model. More details about the experimental setup are in Appendix C.2.We summarize the main findings from our experiments below; additional results with random forest and XGBoost predictors, as well as ablation studies for the choice of hyperparameters, are deferred to Appendix E. To illustrate the role of the alignment score predictor, we also present examples of CXR images with reference and generated reports in Table 2 of Appendix E.4.Tight FDR control. Figure 6 presents the FDR and power curves for Conformal Alignment where the alignment predictor is trained with all features using logistic regression, averaged over 500 independent runs of the procedure. Again, we observe that the FDR is tightly controlled at the target level; it remains constant for sufficiently large α since by then all units can be selected without violating the FDR. We also observe satisfactory power, although it is lower than that is the QA tasks since CXR generation can be a more challenging task, and the model in use is fine-tuned with a limited number of training samples. We shall expect even higher power for better tuned models. Hundreds of reference data suffice. Comparing subplots in Figure 6 , as the size of the reference data D increases, we observe a slight improvement in power. However, we again find that |D| = 500 already suffices for powerful deployment of Conformal Alignment. In practice, we expect this sample size also suffices for more powerful foundation models that are more carefully designed for this specific task.To study the informativeness of different features for alignment, we use one feature at a time to train the logistic regression classifier and compare the resulting power versus target FDR levels in Figure 7 from Conformal Alignment (curves showing FDR control are in Figure 23 of Appendix E.2). In the CXR task, features based on Jaccard similarity are still the most powerful while those based on NLI prediction exhibit slightly lower power. More complicated texts and more tokens in CXR reports could be potential reasons for the relatively poor performance of NLI prediction.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Data access is stated in Section C and reproducible codes can be found at https://github.com/yugjerry/conformal-alignment. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
discussion and limitations: 
 We have presented Conformal Alignment, a principled framework for ensuring the alignment of foundation model outputs with provable, distribution-free FDR control. Compared with standard conformal prediction, we provide a solution that may be more relevant to downstream tasks than existing ideas. We demonstrate that our framework is flexible, lightweight, and effective through concrete applications to question answering and radiology report generation tasks, where we also provide practical recommendations on sample size, data splitting, and feature engineering.The choice of alignment score in different applications remains an open problem. For example, in the radiology report generation example, as pointed out by [56] , there can be a discrepancy between the score given by CheXbert and that given by human evaluation; as such, it will be interesting to use our framework with actual human evaluation labels. More generally, interesting future directions include designing implementation tailored for other appropriate applications, tackling parallel or sequential outputs, improving power in data-scarce scenarios (e.g., with better base models and more informative features), extending to multi-task and in-context learning settings, and generalizing our framework to control other error notions beyond FDR that may be meaningful for downstream decisions.It is crucial to ensure that LLM outputs are aligned with human values, especially in high-stakes scenarios (e.g. medical decision-making). This paper provides a generic framework that identifies reliable LLM outputs and controls the misalignment error rate in a rigorous way. The proposed approach (conformal alignment) can be applied to a wide range of real-life scenarios, e.g. medical decision-making (radiology report generating) and conversational chatbot (question answering).A.1 Proof of Theorem 3.1Let V (x, a) = 2 M • 1{a > c} -g(x); define also V i = V (X i , A i ) and V i = V (X i , c) for every i ∈ [n + m]. By construction, V i = -A i ; V i equals to -A i when A i ≤ c and equals to 2 M -A i otherwise.The conformal p-value defined in (3.2) can be written asp j = 1 + i∈Dcal 1{A i ≤ c, A i ≥ A n+j } 1 + |D cal | = 1 + i∈Dcal 1{V i ≤ V n+j } 1 + |D cal | ,which is the p-value considered in [18] . It can also be checked that{V i } i∈Dcal ∪ { V n+j } j∈[m]have not ties almost surely. Applying Theorem 2.6 of [18] concludes the proof.The proof is an adaptation of Proposition 2.10 of [18] . We first note that the proof of Proposition 2.10 goes through with U = 1, which is the version we shall use in what follows.To start, as in the proof of Theorem 3.1, we takeV (x, a) = 2M • 1{a > c} -g(x). Let F (v) = P(V (X, A) ≤ v) for any v ∈ R. Then F (v) = P(V (X, A) ≤ v) = P(A ≤ c, g(X) ≥ -v) + P(A < c, 2M -g(X) ≤ v) = H(-v) + P(A < c, 2M -g(X) ≤ v).As a result, F (V (X, c)) = F (-g(X)) = H(g(X)), and for any t ∈ R, tP(F (V (X, c)) ≤ t) = t P(H(g(X)) ≤ t) .Invoking Proposition 2.10 of [18] , we have lim |Dcal|,m→∞Power = P(F (V (X, c)) ≤ t(α), A > c) P(A > c) = P(H(g(X)) ≤ t(α), A > c) P(A > c) = P(H(g(X)) ≤ t(α) | A > c). Similarly, lim |Dcal|,m→∞ m j=1 1{j∈S,An+j >c} m = lim |Dcal|,m→∞ Power • m j=1 1{A j > 0} m = P(H(g(X)) ≤ t(α) | A > c) • P(A > c) = P(H(g(X)) ≤ t(α), A > c).We have therefore completed the proof.In this part, we provide a finite-sample power analysis based on concentration inequalities. We defineF + (t) = P(g(X) ≥ t, A > c), F -(t) = P(g(X) ≥ t, A ≤ c), F (t) = P(g(X) ≥ t).The empirical version of these quantities are defined asF + m (t) = 1 m m j=1 1{g(X n+j ) ≥ t, A n+j > c), F + n (t) = 1 n n i=1 1{g(X i ) ≥ t, A i > c), F - m (t) = 1 m m j=1 1{g(X n+j ) ≥ t, A n+j ≤ c), F - n (t) = 1 n n i=1 1{g(X i ) ≥ t, A i ≤ c), F m (t) = 1 m m j=1 1{g(X n+j ) ≥ t), F n (t) = 1 n n i=1 1{g(X i ) ≥ t),We also define m + = m j=1 1{A n+j > c} as the number of truly aligned test outputs. Proposition A.1. Under the same condition of Theorem 3.1, assume further that (X i , E i ) i∈[n+m] are i.i.d. For any δ ∈ (0, 1), with probability at least 1δ,F + (τ -) -ϵ m P(A > c) + ϵ m ≤ m j=1 1{g(X n+j ) ≥ τ , A n+j > c} m j=1 1{A n+j > c} ≤ F + (τ + ) + ϵ m P(A > c) -ϵ m ,whereτ + := inf t : 1 n + 1 • 1 + nF + (t) + nϵ n F m (t) -ϵ m ≤ α , τ -:= inf t : 1 n + 1 • 1 + nF + (t) -nϵ n F m (t) + ϵ m ≤ α ,and ϵ m = log(8/δ/(2n)), ϵ n = log(8/δ)/(2m).Proof of Proposition A.1. Fix any constant δ ∈ (0, 1). Consider the function g(X, A)= M • 1{A > c} + g(X), where M > 2 sup x∈X |g(x)| < ∞ is a sufficiently large constant such that inf x∈X g(x, c + 1) > sup x∈X g(x, c). Therefore, we know that for any |t| < M/2, it holds thatF + (t) = P( g(X, A) ≥ M + t), F -(t) = P(t ≤ g(X, A) < M/2).And similar relations hold for the empirical versions of these quantities. Applying the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality [10, 31] with a union bound for the empirical c.d.f. of g(X, A) for both calibration and test data, we know that with probability at least 1δ,sup t F + m (t) -F + (t) ≤ ϵ m , sup t F - m (t) -F -(t) ≤ ϵ m , sup t F m (t) -F (t) ≤ ϵ m , sup t F + n (t) -F + (t) ≤ ϵ n , sup t F - n (t) -F -(t) ≤ ϵ n , and m + -P(A > c) ≤ ϵ m ,where recall that ϵ m = log(8/δ)/(2m) and ϵ n = log(8/δ)/(2n). We call the event that all above holds the "good event" E * , which happens with probability at least 1δ.By definition, we can show that the selection set of Conformal Alignment is equivalently S = {j ∈ [m] : g(X n+j ) ≥ τ }, where τ = inf t : FDR(t) ≤ α , andFDR(t) = m n + 1 • 1 + n i=1 1{g(X i ) ≥ t, A i ≤ c} m j=1 1{g(X n+j ) ≥ t} = 1 n + 1 • 1 + n • F + n (t) F m (t).By the DKW inequality above, with probability at least 1δ,1 n + 1 • 1 + nF + (t) -nϵ n F m (t) + ϵ m ≤ FDR(t) ≤ 1 n + 1 • 1 + nF + (t) + nϵ n F m (t) -ϵ mholds simultaneously for all t ∈ R. On the other hand, the (realized) power isPow := m j=1 1{g(X n+j ) ≥ τ , A n+j > c} m j=1 1{A n+j > c} = F + m ( τ ) m + .On the good event E * , we haveF + ( τ ) -ϵ m P(A > c) + ϵ m ≤ Pow ≤ F + ( τ ) + ϵ m P(A > c) -ϵ m .By the definition of τ , we know that τ -≤ τ ≤ τ + on the good event E * . The monotonicity of realized power as a function of selection cutoff τ thus implies thatPow ≥ F + m (τ -) m + ≥ F + (τ -) -ϵ m P(A > c) + ϵ m , Pow ≤ F + m (τ + ) m + ≤ F + (τ + ) + ϵ m P(A > c) -ϵ m .For selective prediction, El-Yaniv et al. [11, 12] suggest considering the risk measure, defined (in our context) as:R := E j∈[m] 1{A n+j ≤ c, j ∈ S} E j∈[m] 1{j ∈ S} .This measure is known as the marginal FDR (mFDR) in the multiple testing literature and there has been an exhaustive discussion of its comparison with FDR. For example, Benjamini and Hochberg [5] proposed FDR in favor of mFDR, since the latter is impossible to control in the global null case; Storey [45] also pointed out that mFDR cannot be used for controlling the joint behavior of the numerator (number of false selections) and the denominator (number of selections). On the other hand, to control the risk/mFDR, [12] proposes a method that searches for the cutoff over a grid of values, and then takes a union bound to achieve the overall error control, which might be conservative. In contrast, our method for FDR control does not involve taking union bounds, delivering sharp finite-sample control guarantees.Our work instantiates conformalized selection [18] for the reliable deployment of foundation model outputs. We detail our contributions as follows. First, we justify that selecting reliable outputs with FDR control is a practically reasonable criterion for downstream tasks. Second, we formulate the alignment problem as a selective prediction problem that can be tackled with conformal selection, adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current application in job hiring and drug discovery requires numerical outputs and predictions; the task of aligning foundation model outputs, however, does not come with a clear numerical output. To make conformal selection applicable to our purpose, we formulate the alignment status as a numerical indicator that could be inferred with reference information, and then decide how to train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Our third contribution is the implementation of the pipeline. In this regard, we conduct extensive numerical experiments, finding that using a lightweight prediction model such as logistic regression or random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides a concrete and easy-to-use pipeline for practical use.This section contains implementation details for Section 4 and 5.As is mentioned in Section 4, we adopt a subset of sample size 8000 from the TriviaQA dataset [21] 5 and a subset of sample size 7700 from the CoQA dataset [39]6 . For both TriviaQA and CoQA datasets, we follow the exact text preparation and prompt engineering procedure in [29] , which we show below. To evaluate the correctness of generated outputs, we adopt the rouge-L score [28] that focuses on the longest common subsequence between two sequences. Following the practice in [24, 29] , a generated answer is defined to be admissible when the rouge-L score is no less than 0.3.TriviaQA. The TriviaQA dataset can be loaded from the Python module datasets and we use the rc.nocontext split. Prompts in use are the same as those in [29, 47] and take the following format.Answer these questions: Q: In Scotland a bothy/bothie is a? A: House Q: [sample question] A:CoQA. The CoQA dataset is downloaded from https://downloads.cs.stanford.edu/nlp/ data/coqa/coqa-dev-v1.0.json and the prompts are the same as those in [29, 24] .Answers are generated using the default configurations similar to [29] ; in specific, we use num_beams=1, do_sample=True, top_p=1.0, top_k=0, temperature=1.0. For each question, we generate M = 20 answers independently, with which we treat the first output as f (X i ) and utilize the M outputs to calculate the similarity-base scores.Given the generated outputs, we adopt the pipeline in [29] to calculate the uncertainty and confidence scores. The self-evaluation score (Self_Eval), or P(True), is obtained using the same prompt in [22] as follows: The self-evaluation score is then defined as the output probability associated with the generated answer "A)".Language models. We use two language models without fine-tuning for comparison in terms of the accuracy of the generated answers: OPT-13B and LLaMA-2-13B-chat. The implemented OPT-13B model is from Hugging Face https://huggingface.co/facebook/opt-13b and the implemented LLaMA-2-13B-chat is from https://llama.meta.com. We utilize an off-the-shelf DeBERTa-large model [13] 7 as the NLI classifier to calculate similarities.We use the subfolders p10,p11,p12 from the MIMIC-CXR dataset [20] (these are all the data we use, including fine-tuning and subsequent experiments for Conformal Alignment) to limit the resources required for fine-tuning. The MIMIC-CXR dataset can be accessed from the PhysioNet project page https://physionet.org/content/mimic-cxr/2.0.0/ under the PhysioNet Credentialed Health Data License 1.5.0.In this application, we fine-tune a vision-language model, with the Vision Transformer google/vitbase-patch16-224-in21k pre-trained on ImageNet-21k8 as the image encoder and GPT2 as the text decoder. We largely follow the training procedure in [36] . In particular, each raw image is resized to 224 × 224 pixels. We then fine-tune the model on a hold-out dataset with a sample size of 43, 300 for 10 epochs with a batch size of 8, and other hyperparameters are set to default values. The training process takes about 10 hours on one NVIDIA A100 GPU.Once fine-tuned, the model is treated as fixed for generating reports. The data not used in fine-tuning are used as the training, calibration, and test data in our experiments. For report generation, we use the default configurations (max_length=512, do_sample=True) and the input of the visionlanguage model consists of pixel values of images (resized to 224×224 pixels) and tokenized prompts (tokenized via the same GPT2 model). For each image, we generate M = 10 reports independently, with which we treat the first output as f (X i ) and utilize the M outputs to calculate the features (similarity-based scores) for training the alignment predictor. To determine if a generated report is admissible, we follow the practice in [36] and use the CheXbert model [44] to convert both the generated and reference reports into 14-dimensional vectors. For each coordinate, the entry falls into the categories "Blank", "Positive", "Negative" or "Uncertain", with which we further convert them to binary vectors with each entry indicating "Positive" or not. The alignment score A i ∈ {0, 1} is defined to be the indicator of whether there are at least 12 matched coordinates between the two binary vectors. Similar to the QA task, we modify the pipeline in [29] to calculate the uncertainty and confidence scores, in which we utilize an off-the-shelf DeBERTa-large model [13] as the NLI classifier to calculate similarities.We present the FDR/power plots when applying Conformal Alignment to the TriviaQA dataset and the alignment predictor is trained with random forests (Figure 8 ) and XGBoost (Figure 9 ), fixing γ 1 = 0.2 and γ 2 = 0.5. These figures are similar to Figure 3 (which uses logistic regression to train the alignment predictor) in the main text. We observe similar messages as those in Section 4: the FDR is always tightly controlled with satisfactory power. Regarding the choice of |D|, we similarly see that 500 high-quality samples suffice for the powerful deployment of our method. In this part, we present a set of results parallel to the main text and Section D.1 when we apply Conformal Alignment to the CoQA dataset; details about the dataset are in Appendix C.1.We present the FDR/power plots for the CoQA dataset when the alignment predictor is trained with logistic regression (Figure 10 ), random forest (Figure 11 ), and XGBoost (Figure 12 ), fixing γ 1 = 0.2 and γ 2 = 0.5. We again observe similar messages as those in Section 4: the FDR remains tightly controlled, and a sample size of |D| = 500 suffices for the powerful deployment of our method.We also recall the baseline, where one asks the language model to evaluate its confidence (i.e., the Self_Eval feature above), and threshold the obtained likelihood {q self j } with a cutoff of 1α as if it were the "probability" of alignment, i.e., S baseline = {j ∈ [m] : q self j ≥ 1 -α}. In Figure 13 , we present the results with the CoQA dataset, in which the baseline fails to control FDR for any given α and also exhibits lower power than our proposed approach. 1.0 FDR LLaMA-2-13B-chat Self_Eval SE Num_Sets Lexical_Sim EigV(C) Ecc(C) Deg(C) EigV(E) Ecc(E) Deg(E) EigV(J) Ecc(J) Deg(J)Figure 14: The realized FDR at each FDR target level for TriviaQA dataset when the alignment predictor is trained with logistic regression over each individual feature.From Figure 14 , we note that FDR is always controlled for each feature. However, FDR with more powerful LLM (LLaMA-2-13B-chat) is tightly controlled while Self_Eval and Num_Sets with OPT-13B produce FDR and power both close to zero when α is small as their predictive power is too low. We omit similar results when conducting the same experiment on the CoQA dataset.In this part, we present and discuss ablation studies for choosing the data splitting schemes γ 1 , γ 2 through experiments on the TriviaQA dataset, with the alignment predictor trained via logistic regression over all the features in the main text.Varying γ 1 (size of the tuning set). Recall that we use a random subset of size ⌊γ 1 |D|⌋ for tuning hyperparameters [29] in the features, and a subset of size ⌊γ 2 |D|⌋ for training the alignment predictor. We start by studying the effect of γ 1 on the performance with fixed γ 2 . This means the sample size for predictor training is fixed, but there is a tradeoff between the sample sizes used for hyperparameter tuning and the calibration step. Here, logistic regression is used to train the alignment predictor and the total number of "high-quality" data is fixed at |D| = 2000.We show the results when we vary γ 1 in {0.2, 0.4, 0.6} for the TriviaQA dataset (Figure 15 ) and the CoQA dataset (Figure 16 ). In Figure 16 , we can see that with fixed |D| and γ 2 , when γ 1 increases, the power slightly decreases due to the decreasing sample size for training the alignment score predictor and BH procedure. Besides, we note that γ 1 = 0.2 is sufficient for tuning the parameters to calculate the features.Varying γ 2 (size of the predictor training set). We proceed to study the choice of γ 2 , the proportion of reference data used for training the classifier. We vary the value of γ 2 while fixing the size of the reference set |D| = 2000 and γ 1 = 0.2 per our recommendation from the last part. Note that with |D| and γ 1 fixed, varying γ 2 also reveals the effect of |D cal |. We use logistic regression as the base classifier. The results are shown for TriviaQA dataset in Figure 17 and CoQA dataset in Figure 18 .At values γ 2 = 0.1 and γ 2 = 0.7, we observe a slight power loss compared with the middle two columns. The reason is that when γ 2 is close to either 0 or 1, there will be unbalanced sample sizes between D(2) tr and D cal , and the scarce of sample in each set will affect the performance, i.e. causing larger variance in FDR/power or smaller power. In light of this, we suggest the choice of γ 2 ∈ [0.3, 0.5] to balance D We used the XGBoost functionality to compute the importance scores in Figure 19d . We also computed the Shapley value of individual features to measure their importance based on the prediction models we use in the manuscript. As the Shapley values are model-agnostic, we present results for three choices of g(X) with different base classifiers, including logistic regression, random forests, and XGBoostRF, in Figure 19a, 19b, and 19c . Although the exact order of features varies in four plots, there are features that have consistent dominating effects in all figures, e.g. Deg(J) and EigV(J) based on the Jaccard similarity. In addition, NumSets as the feature in alignment score predictors tends to exhibit an effect close to zero in all four figures. We should note that the aforementioned • Example 2: Question: What was the name of the brothel in The Best Little Whorehouse in Texas? True answer: Chicken ranch. Generated set: {'miss monas', 'chicken ranch', 'miss mona's', 'miss monas bordello'}.As we can see in Example 1, the generated set is very informative and can be used directly. However, although the generated set in the second example has a high probability of covering the true answer, it contains very different answers and is potentially confusing in practice. In other words, whether "Chicken ranch" or "Miss Monas" is correct is unclear to the user.In addition, as is shown in Figure 20 (left panel), we present the proportion that the size of generated size is k for different values of k ∈ N, which shows that there are more than 40% of sets having no less than 4 distinct items. In addition, Figure 20 (right panel) shows that in nearly 30% of the questions, the admissible answers (with rougeL ≥ 0.3) take up less than 20% of the uncertain set, which poses a challenge in the direct deployment of answers in the uncertain set. We also use random forests and XGBRF as alternative classifiers to train the alignment score predictor. Similar to Section 5, we fix γ 1 = 0.2 and γ 2 = 0.5. The results are in Figure 21 for random forests, and Figure 22 for XGBoost. As |D| increases, we observe a slight increase as well as a decrease in variance in the power curve while FDR remains tightly controlled. Figure 21 : FDR/power plots for conformal alignment with CXR dataset when using random forests to train the alignment predictor. Details are otherwise the same as in Figure 6 .Reference report Generated report Alignment scores:A i = 0 A i = 0.194 < τIn comparison with the study of _, there has been a substantial increase in opacifications diffusely involving both lungs. Cardiac silhouette remains within normal limits and there is no evidence of pleural effusion. The appearance suggests diffuse pulmonary edema. However, in the appropriate clinical setting, widespread pneumonia or even ARDS could be considered.In comparison with the study of _, there is little overall change. Again there is a substantial enlargement of the cardiac silhouette with diffuse bilateral pulmonary opacifications consistent with pulmonary edema. In the appropriate clinical setting, superimposed pneumonia would have to be considered. Guidelines:A i = 1 A i = 0.875 > τ• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: the research conducted in the paper conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: the broader impacts of this work, including the deployment of foundation models with misalignment error rate control in a wide range of real-life applications, are discussed in both the abstract and introduction. To be concrete, it is crucial to ensure that LLM outputs are aligned with human values, especially in high-stakes scenarios (e.g. medical decision-making). This paper provides a generic framework that identifies reliable LLM outputs and controls the misalignment error rate in a rigorous way. The proposed approach (conformal alignment) can be applied to a wide range of real-life scenarios, e.g. medical decision-making (radiology report generating) and conversational chatbot (question answering).Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.See Appendix D.7 for more detailed discussion on the types of guarantees.In this work, we use "alignment" to refer to a desired property of an output that may vary with the context.https://nlp.cs.washington.edu/triviaqa/https://stanfordnlp.github.io/coqa/https://github.com/microsoft/DeBERTahttps://huggingface.co/google/vit-base-patch16-224-in21k 
 
 Appendix:  


Decision: This work addresses an important problem on the emerging topic of foundation models (and more particularly related to large-language models). Despite the reviewers raised several important points, and some may not be fully addressed in the rebuttal (e.g., novelty with respect to prior works), I believe that the proposed work has some merits, as mentioned by the reviewers. More specifically, the task of conformal prediction has not been explored in LLMs, despite its paramount importance. Furthermore, the experiments are extensive, which add empirical support to the mathematical formulation and theoretical justifications presented in this work. Thus, I recommend the acceptance of this submission.

Reviews: review: paper_summary: paper_summary: The paper proposes to use conformal prediction to determine when to trust model outputs. To this end the authors apply conformal risk control on across unit instead of focusing on a single unit. This is especially relevant in their X-ray example where the goal is to give a set of trusted documents to the medical professional to review. They show that their adaptation of conformal prediction is also sensible in terms of FDR and power.
strength_weakness: strength_weakness: - My main concern with this paper is that, at the end of the day it is just an application of conformal risk control to foundation models. No extensive new theory was needed, but rather just a twist to the setting. If the authors could clearly design the technical challenges faced it would be helpful in making a decision on this paper.
- For the alignment score predictor, can the authors please elaborate on how this score was picked and why this was the best design choice? These seems to be randomly picked and a more systematic justification would be very welcome to better understand if this score can be improved or not
- In practice for a medical professional, what is the target FDR rate? In all experiments the power is quite slow to increase at low FDR rate, however in practice i would have assumed that a medical professional would not want to look through documents where every second one is wrong? This puts into question the motivation as well as practicality of this paper which i would urge the authors to clarify.

i am more than happy to change my score of the above have been addressed.
questions: questions: see above
limitations: limitations: yes
paper_summary: paper_summary: This work applies recent work on conformal outlier detection [1] in the context of LLM alignment. The authors assume that one, given input $X$ and output $Y$ of an LLM, has access to an alignment scorer $A$ that can then produce an alignment score of $A(X, Y)$. They treat the null hypothesis of being unaligned as being $H_0: A(X, Y) \leq c$, i.e., the alignment score being smaller than a threshold $c$. They describe the typical score function as $\widehat{A}$, and the resulting conformal score being $\widehat{A}(X)$ for a data point. Using the scores, they construct a conformal p-value that is superuniform when $A(X, Y) \leq c$ and apply the Benajmini-Hochberg procedure on the conformal p-values to get FDR control.
strength_weakness: strength_weakness: I think the paper, in some sense, is just an extra series of experiments for conformalized selection --- we already know it works well for many deep learning tasks from previous papers, and the experimental results are not particularly surprising, since it continues that trend. Many other conformal papers in the LLM space try to tackle the problem that's unique to LLMs, i.e., factuality, and try to investigate the fact that it is not already a pre-defined, structured task, and aim to apply conformal in a meaningful way. Thus, I think this paper does lack some novelty in its methods.
questions: questions: - What is $q$ in Prop. 3.3 --- is it some constant less than $\alpha$?
- What is the cost in power in directly estimating the asymptotic cutoff (and getting a PAC bound) vs. the conformal procedure?
limitations: limitations: Yes, the limitations are sufficiently addressed.
paper_summary: paper_summary: This paper proposes a procedure similar to conformal prediction to select deployable units and control the FDR. Asymptotic result is provided to justify the correctness of the proposed algorithm. Real data experiments in QA and medical tasks are also provided to demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: While the theoretical results are appreciated, the major issue of this paper lies in its writing:

(1) The word "alignment" in this paper is different from the very commonly used "alignment" in which LLMs are tuned to refuse to answer harmful questions. This wording issue can cause significant confusion when readers reading this paper. Please consider replacing the word with other choices to better describe the task performed in this paper, e.g., calibration?

(2) The authors are suggested to better clarify their contribution compared to existing literature. In terms of the methodology, how is the algorithm different from the things in [5,16,17] in the reference list? For technical contribution, to set an example, in line 167, it is mentioned that "The proof of Theorem 3.1 is adapted from [17], and we include it in Appendix A...". Please consider summarize the adaptation and challenges when deriving Theorem 3.1.

(3) While the title of this paper mentions "foundation model", it is not clear how the proposed algorithm is related to foundation models: it seems that the proposed method can also be used in other small-scale models. Alternatively, is there any other better methods for small-scale models than the proposed method which cannot be used for foundation models?

(4) Please define "power" in line 172.

(5) Line 255 "More powerful model enables more powerful selection": could you provide insights or interpretation on this observation?

(6) Figure 5: XGBoost provides the functionality of computing the importance score for the features used in the regression. In addition to the individual factors considered in logistic regression in Figure 5, could the authors provide more results on how important each features is? The correlation among features may also makes the actual importance of the features when regression on them together.

(7) Non-asymptotic result: Although Proposition 3.3 provides the asymptotic result on the power of the method, since the experiment includes "How many high-quality samples are needed?", the authors are also suggested to provide a discussion on the non-asymptotic results.
questions: questions: My rate is 4 for now because of the many issues in this paper. I still value the theoretical justifications and would like to raise my score if the authors are willing to clarify them.
limitations: limitations: No limitations.
paper_summary: paper_summary: This paper introduces Conformal Alignment, a framework designed to ensure that outputs from foundation models align with human values before deployment in high-stakes tasks, such as radiology report generation. By training an alignment predictor with reference data, the framework guarantees that a user-specified fraction of selected units meet the alignment criterion. Demonstrated through applications in question answering and radiology report generation, this method effectively identifies trustworthy outputs with lightweight training over a moderate amount of reference data.
strength_weakness: strength_weakness: The paper's technical contribution seems modest, with some ambiguity in distinguishing its novelty from [17], a potentially overstated claim regarding model alignment with human values, and limited comparisons to relevant works.

1.	The technical contribution of the paper appears incremental. As claimed, the main achievement lies in its strict error control through post-hoc adjustment/selection, rather than developing new training strategies to improve specific alignment metrics of the model. However, its theoretical foundation closely resembles that of [17]. For instance, Theorem 3.1 is a modification of a theorem in [17], and Algorithm 1 is also very similar to an algorithm in [17]. The paper lacks clarification on what constitutes the new contributions compared to [17]. It seems that this work merely replaces the monotone nonconformity score V in [17] with a predicted alignment score. Is that right? Additionally, the paper is not well-written, which hampers the understanding of its contributions.

2.	The proposed Conformal Alignment claims to ensure that outputs from foundation models align with human values. However, the evaluation on the radiology report generation task does not support this claim. The alignment scores used to train the score predictor are based on the CheXbert score for 14-class classification, which does not strictly align with human ratings ([R1] Figure 3). The authors should consider using datasets with genuine human ratings, like those in [R1], for verification. Otherwise, this claim seems overstated.
[R1] “Evaluating progress in automatic chest x-ray radiology report generation”. Patterns, 2023


3.	The proposed method was not compared to any relevant works like [16] and [17], except for a heuristic baseline in Figure 4. Without such comparisons, it is difficult to position the proposed method within the existing literature.
questions: questions: 1.	Could you specify the new contributions of your work compared to [17]? It seems Theorem 3.1 and Algorithm 1 closely resemble those in [17], with the primary change being the replacement of the monotone nonconformity score V with an alignment score predictor. Can you elaborate on any additional novel aspects?

2.	Could you please explain the importance and implications of Proposition 3.3 in your framework?

3.	Consider evaluating your method on datasets with actual human ratings to better validate the claim that your framework ensures alignment with human values. 

4.	Could you please extend the experimental comparisons to include other relevant methods, such as those in [16] and [17]?

5.	This paper suggested |D|=500. However, if genuine human values are used to train the alignment predictor, this number of human annotations is not small for radiology report generation.  How do you plan to address this challenge?

6.	In Conclusion, it is mentioned that the proposed solution is more relevant to downstream tasks than existing ideas. Could you please elaborate this?
limitations: limitations: Not exactly although some future works were mentioned
paper_summary: paper_summary: The paper proposes to adapt conformal guarantees to large language models. They build upon existing work to provide bounds on FDR error for each model output by conformalizing the distribution of an additional alignment predictor trained on labeled aligned data. Experiments are conducted on two radiological report generation and question answering using pretrained LLM models.
strength_weakness: strength_weakness: * The technical novelty of the method seems limited as it seems to be a direct application of prior work for LLM's https://arxiv.org/abs/2210.01408 (more discussion of the differences would be helpful in clarifying technical novelty)
* There is a lack of comparison to other LLM conformal methods. Providing a few empirical comparisons against other work in uncertainty quantification and alignment would help demonstrate the advantages of the proposed method.
questions: questions: - The term "unit" should be explicitly defined before using throughout the paper
- How is alignment with human experts typically measured? How to generate ground-truth alignment scores? More discussion should be included
- "can be gravy" --> "can be grave"
- In line 114, what, specifically, is the "selection bias"?
-  In line 119, please cite the "Two works addressing the alignment of foundation models with CP ideas have been briefly discussed in Section 1."
- In line 131, what is "mild" about the exchangeability condition and how does it differ from the conventional conformal prediction assumption?
- In line 137, please provide the citations for the "commonly-used alignment criteria:
- How would this framework handle distribution shifts or in-context learning? 
- Can a reward model such as in RLHF be used as the alignment predictor? How does alignment relate to human preferences?
- Did you try varying $M$ outputs during the training of the alignment predictor?
limitations: limitations: * How to select an appropriate alignment function? More discussion should be provided as it is the core component of the method
* How to generalize beyond calibrating single tasks to multiple tasks? A user generally queries a foundational model for many different types of tasks, not only a single task. 
* How to handle dependence between inputs e.g. a chatbot's output depends on prior context and history. How would this be incorporated?
* The paper would benefit from a dedicated limitations section.

Comments: title: Follow-up comment
Authors responses
Thanks for the response
Reply to Reviewer nRXb
Response
Response to Reviewer FNNT
Response to Reviewer w89M
Response to Reviewer SzCp
Response to Reviewer Ctqb
Response to Q3 of Reviewer nRXb
comment: Thank you so much for the follow-up! 

For your concern on evaluation metrics, it's worth mentioning that in $\mathcal{D}_{\rm train}$, both generated and human-written reports are given and the difference lies in how to define the alignment map $\mathcal{A}$ in practice. As the limitation of human annotations can be inevitable in some scenarios, we currently follow the practice in [1], where $\mathcal{A}$ is a proxy for human rating based on the CheXbert score. As is shown in the reference [2] provided by the reviewer, existing metrics can be imperfect in terms of the alignment with human ratings, thus it is a very interesting open question (but is also of independent interest) in the area. We will discuss this in our updated draft!

[1] Quach, Victor, et al. "Conformal language modeling." arXiv preprint arXiv:2306.10193 (2023).

[2] Yu, Feiyang, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca et al. "Evaluating progress in automatic chest x-ray radiology report generation." Patterns 4, no. 9 (2023).
Dear Authors,

Thank you for your clarification. I truly understand and appreciate your contribution from the application perspective, which is the reason for my original score suggestion. However, the theoretical part (including the algorithm), which is a main component of the proposed work, seems to have a large overlap with the existing work [17].  I could not see how the algorithm in [17] is extended to address the application challenges in this work, which impacts my understanding of the novelty of this paper. Meanwhile, due to commonly very limited genuine human rating data (instead of machine-generated CheXbert scores), the proposed method needs to carefully handle a small sample-size problem. I appreciate the authors' response and suggest incorporating this into the limitation of the current approach. Due to the above reasons, I keep my original score.
Thank you for the feedback. Indeed, we apply conformal selection to LLMs (similar to related works that apply conformal prediction to LLMs e.g., [3,4]). However, as we have mentioned in our previous rebuttal, the emphasis of the current work is the novel instantiation of conformal selection for reliable use of foundation model outputs, which is non-trivial. Compared with conformal selection, the major challenge lies in justifying the criteria, formulating the problem, adapting existing techniques to the current setting, and implementing the complete pipeline for practical use. We elaborate on the challenges in the following. 

- The first challenge is justifying what criterion is practically useful. We argue that selecting reliable outputs (similar to the “prediction with abstention” idea) with FDR control is a practically reasonable criterion since it ensures that most of the deployed outputs (such as the radiology reports that are handed over to doctors) are indeed reliable and useful. This contrasts our work with the works that apply conformal prediction to standard classification/regression tasks. Particularly in the context of LLM/foundation models, our proposal differs substantially from conformal language modeling [3] that generates multiple outputs for a unit, ensuring *at least* one of the outputs is aligned, and conformal factuality [4] that edits the original outputs to make it less specific/informative, so as to achieve guarantees on the correctness of the output for a single unit. More comparisons with the works connecting conformal prediction and LLM/foundation models can be found in Section 2.1 of the manuscript.

- The second challenge is formulating the alignment problem as a selective prediction problem that can be tackled with conformal selection, and adapting conformal selection to the current setting. Note that while conformal selection is an established method, its current applications in job hiring and drug discovery apply to numerical outputs and predictions, rather than aligning foundation model outputs (which do not come with a clear numerical output). To make conformal selection applicable to our purpose, we need to formulate alignment as a numerical indicator that could be inferred with reference information, and then decide how to train a prediction model for the alignment score, which can therefore be used in conjunction with the conformal selection framework. Indeed, how conformal prediction ideas may be used to calibrate LLMs/foundation models (beyond traditional numerical outputs) is a nontrivial problem that has attracted considerable recent interest; in this sense, we contribute to discussion on this important issue. 

- With the general recipe, the third important challenge and contribution is the practical instantiation of the pipeline, which features to use, what model to train for alignment, among other considerations. In this regard, we conduct extensive numerical experiments, finding that using a lightweight prediction model such as logistic regression or random forests with popular heuristic (uncalibrated) uncertainty measures is often sufficient for identifying reliable outputs. This provides a concrete and easy-to-use pipeline for practical use. Indeed, even beyond the current setting of selecting reliable outputs, we view the choice of features and prediction models as contributions that may be helpful for predicting the reliability of outputs for other purposes. 

Hope this clarifies the confusion!
If this concern (resemblance to [17]) cannot be cleared, I have to maintain my original score.
Dear authors,

Thank you for your feedback. Indeed, my major concern is the resemblance of this paper to [17]. Could you please clarify that? "For instance, Theorem 3.1 is a modification of a theorem in [17], and Algorithm 1 is also very similar to an algorithm in [17]. The paper lacks clarification on what constitutes the new contributions compared to [17]. It seems that this work merely replaces the monotone nonconformity score V in [17] with a predicted alignment score. Is that right?" Thanks.
Thank you for the detailed reply. My criticism of novelty is more from its relationship with prior work [8]. The alignment criterion utilized in the experiments in this paper can be rephrased in terms of binary classification (as the alignment labels are simply binary labels in all the experiments), but many experiments of this type have already been carried out in [8]. This paper reformulates many existing "alignment" criteria that have already been established for certain tasks/datasets into binary classification just by picking a threshold to determine "aligned" vs "not aligned", so this is not really novel. By referring to existing work utilizing conformal techniques to LLMs, I was commenting that a key difficulty is actually figuring out what is a meaningful alignment criterion to control --- this paper does not address this difficulty at all.

I do think the experimental results are somewhat interesting (particularly the usage of an efficient scorer), so I am sticking with my original score.
I thank the author for their detailed response and for addressing my comments. I will raise my score.
Dear reviewer, 

did authors address your concerns in their rebuttal? I would appreciate if you could review their responses and potentially engage in a discussion with them?

Best,
your AC.
I am happy with the above response and will increase my score to 6.
Thanks again for the clarifications and hope that the authors will add them into the final version of their paper.
Thank you for taking the time to read through our rebuttal and follow up on the discussion! 

- Firstly, thank you for pointing out the references [1, 2]; we will add them to the manuscript. other works focus on the **construction of prediction sets** for a single unit, while we focus on using conformal prediction to **select units**. Specifically, [2] constructs a prediction set whose candidate values obey FDR control (which is defined for each unit, while ours is for multiple selected units), and [1] constructs prediction sets for units that are selected by a given process. As such, the techniques we use are also quite distinct. 

- For Q1, we do apply conformal selection to LLMs (similar to related works that apply conformal prediction to LLMs e.g., [3,4]). Our previous rebuttal emphasizes that, not only is the application not straightforward (which involves defining the suitable guarantee and efficient instantiation to maintain power), but the application itself also reveals richer and interesting phenomena for understanding LLMs and uncertainty (e.g. how language model affects the power of selection, what features are informative of reliability, what LLMs’s uncertainty is more predictable). 

- For Q2, we would discuss both the predictors $X$ and the prediction model $g(X)$ in order to minimize the chance of our misunderstanding. To summarize, we offer theoretical justification for the optimal choice of g(X). However, in practice, the model $g$ has to be trained and the choice of features follows the standard in the literature; thus, they are indeed data-dependent. 
    - First, the features $X$ we choose are the state-of-the-art measures of uncertainty in the current literature, based on the fact that such measures have been observed to be informative of the reliability of LLM outputs. We totally agree that there is much space for new features, which we leave for future work. 
    - Second, the choice of the prediction model $g$ (based on classifiers) is guided by the asymptotic power [proposition 3.3], which is maximized when $g(X)$ is proportional to $P(A > c | X)$ almost surely, and thus we train $g$ to be classification models. This model $g$ is naturally data-dependent to maximize the power of detecting trustworthy outputs. 

- Finally, we totally agree that it is meaningful to conduct additional ablation studies to better understand the performance. In response, we conducted ablation studies when (i) features $X$ are corrupted with noise (**Table 1**), and (ii) the predictor $g(X)$ is corrupted with noise after being trained (**Table 2**).  We can see that in **Table 1**, FDR control is still valid as $g(\tilde X)$ is trained with labeled data although features are corrupted. In **Table 2**, corrupted $\tilde g(X)$ is no longer informative, thus FDR control no longer holds when the noise strength increases. The power in both cases is decreasing with higher noise strength. In summary, we find that model training with informative features is a crucial component in the pipeline. 

We truly hope our comments could clarify your questions and enhance your evaluation of the current paper. Please don’t hesitate to let us know if you have any further questions or concerns!

**Table 1: features X with entrywise noise N(0, noise strength)**

noise strength   |   0.0  |  2.5  |  5.0  |  7.5  |  10.0  |  12.5  |  15.0
:---------------:   |  ----- |-----  | ----- | ----- | ------ | ------  | ------
FDR                   |           0.2  |  0.2  |   0.2  | 0.2   |  0.19 |   0.2  |   0.18
Power                |            0.95 |  0.93  |  0.9 |  0.85  |  0.73  |  0.62 |   0.45

**Table 2: predicted score g(X) with noise N(0, noise strength)**

noise strength    |  0.0   | 0.2  |  0.4  |  0.6  |  0.8  |  1.0
:----------------:    |-----    |----- | ----- | ----- | ----- | -----
FDR                    | 0.2     | 0.2  | 0.2   | 0.21  | 0.24 |  0.27
Power                 |0.94    | 0.9   | 0.68 |  0.45 |  0.41 |  0.43


**References**

[1] Bao, Yajie, Yuyang Huo, Haojie Ren, and Changliang Zou. "Selective Conformal Inference with FCR Control." arXiv preprint arXiv:2301.00584 (2023).

[2] Fisch, Adam, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. "Conformal prediction sets with limited false positives." In International Conference on Machine Learning, pp. 6514-6532. PMLR, 2022.

[3] Quach, Victor, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. "Conformal language modeling." arXiv preprint arXiv:2306.10193 (2023).

[4] Mohri, Christopher, and Tatsunori Hashimoto. "Language models with conformal factuality guarantees." arXiv preprint arXiv:2402.10978 (2024).
I thank the authors for their rebuttal.
1. The rebuttal is hard to follow the key points. I had to read over it multiple times to understand most of the idea. At its core teh rebuttal is not arguing that the theory is novel but rather the application. Hence papers likes [Selective conformal inference with false coverage-statement rate control] or [Conformal Prediction Sets with Limited False Positives] are neither cited nor discussed. Can the authors explain if the proposed method in the paper is a simple application of these types of papers? Please be succinct as your previous review already exceeded the 6000 characters and as per rule i would not have to read the additional comments. I believe this sentiment is prelevant amongst most reviewers and hence i would appreciate a clear distinction between the methods that already exist in term of "what can previous methods already do" and "what can the proposed method do that the existing methods cannot"
2. So the authors agree that the selection is indeed data dependent and hence a pure heuristic. Did the authors conduct any ablation study in the case these predictors were badly chosen? i.e. corrupted with noise or have lower signal?
3. i agree with the authors.
I appreciate the authors in addressing my comments. It is now clearer to me the contribution of this paper, so I raised my score to 6.
- **Q5: More powerful model enables more powerful selection**. Thank you for raising this question for discussion. This statement can be illustrated from both empirical and theoretical perspectives. (1) In practice, the alignment scores are correlated with the model likelihood,  input uncertainty, and output confidence produced by the model (line 209), so with more capable models, the alignment scores could be better calibrated. Taking Q&A as an example, the alignment scores with LLaMA-13b could be in line with the true correctness of the output more faithfully, thus the selection power is higher as is shown in Figure 3. (2) From the theoretical perspective, when the alignment score predictor g(x) is more powerful, e.g. g(x) \approx A, by proposition 3.3, the asymptotic power can be maximized. In addition, [8] shows via Neyman-Pearson Lemma that the optimal predictor should be monotonic in P(A>c|X).

- **Q6: Feature importance**. Thank you for the thoughtful question! We would like to first clarify that the purpose of the ablation study on feature importance is mainly to provide information on the choice/usefulness of features that might be helpful for future related tasks, which is orthogonal to our main point of validity. 

    Following your suggestions, we used the XGBoost functionality to compute the importance scores (please see Figure 1(d) in the attached pdf). We also computed the Shapley value of each feature to measure their importance based on the prediction models we use in the manuscript. As the Shapley values are model-agnostic, we present results for three choices of g(X) with different base classifiers, including logistic regression, random forests, and XGBoostRF, in Figure 1(a)(b)(c). Although the exact order of features varies in four plots, there are features that have consistent dominating effects in all figures, e.g. Deg(J) and EigV(J) based on the Jaccard similarity. In addition, NumSets as the feature in alignment score predictors tends to exhibit an effect close to zero in all four figures. We should note that the aforementioned findings are also revealed in Figure 5 in the paper, where predicted scores with EigV(J), Deg(J), and Ecc(J) have higher power and that with NumSets is much less powerful. We thank the reviewer again for raising this question for more comparison of feature importance in the alignment score predictor.

- **Q7: non-asymptotic power**. Thank you for the suggestion! We will be adding a parallel non-asymptotic result on power in the revised manuscript. To derive the non-asymtotic result, we leverage the uniform non-asymptotic concentration inequalities, obtaining an error bound on the scale of $1/\sqrt{n_{\text{calib}}}$. 

We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!

**Reference** Please see references in the global rebuttal.
- **Q2: typo in Prop 3.3**.This is a typo and should be $\alpha$ instead. Please consider it fixed!

- **Q3: PAC bound**. If we resort to the pac bounds, we need a high probability upper bound on the FDP that is uniform over the rejection threshold $t$ (since $t$ is chosen in a data-driven manner and the FDR is not monotone in $t$), which amounts to an error of $\frac{c}{\sqrt{n_{\text{calib}}}}$ with a potentially large constant $c$. Thus, if we would like to achieve an FDR below $\alpha$ with high probability, we need to run the BH procedure at nominal level $\alpha - \frac{c}{\sqrt{n_{\text{calib}}}}$.  When $n_{\text{calib}}$ is a few hundred, the gap is on the scale of $0.1$, which may have a huge impact on the power of our method, as manifested in the experiments. 

We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!


**Reference** please see references in the global rebuttal.
- **Q8: Distribution shift and in-context learning**. Thank you for the thoughtful question! There are recent interests in handling distribution shifts in conformal selection, and theoretically, it is feasible to adjust for covariate shift [9] with a proper estimator for the density ratio. This leads to a new definition of p-values and selection procedure. For in-context learning, as long as a certain exchangeability structure is preserved (for example, when the in-context prompts are generated in an exchangeable way), our method is still able to deliver FDR control. 

- **Q9: RLHF**. Thank you for the thoughtful comments! 
    - From our limited understanding of RLHF, it focuses on improving the foundation model itself, where the reward function is mainly a stepping stone for better training. In theory, it is completely feasible to use the reward model in our framework since our method applies to any criterion and can use any predictor. However, to incorporate RLHF in our setting, it might not be clear what the “ground-truth” labels are. 
    - Our definition of alignment is remotely related, but differs from the usual definition of human preference. Usually, human preference uses pairwise comparison to overcome the difficulty of a numerical measure of quality. In contrast, we follow the widely adopted notion of “acceptable/reliable” outputs. An interesting future direction would be to allow for using preference outcomes in our framework, but we expect that it would need a different formulation of the problem. 

- **Q10: Varying M**. Thanks for this insightful question! The choice of M, the number of generations for each question/CXR image, is very important in estimating the uncertainty/confidence scores. More concretely, a large M could produce better-calibrated scores (e.g. the score is more informative to predict the alignment with human evaluations). The effect of M has been explored in [10] (Table 7, page 22), where they found that the uncertainty/confidence scores with M=3 already outperform white-box scores (e.g. likelihood) and semantic entropy. In addition, the results with M=10 are very close to those with M=20 and the benefit in increasing M is minor when M >= 20. Thus, we opt to use $M=20$ for the best performance in our work. In our revision, we will add more ablation studies with smaller and larger values of $M$, and we expect that the findings should be consistent with those in [10]. 

- **Q11: Multi-task**. The multi-task setting is indeed an important setting in  LLM. From the theoretical perspective, the current framework can be directly generalized to the following two scenarios: (1) each unit has the same set of tasks or (2) the set of tasks for all units is generated in an exchangeable manner. In both settings, directly applying our framework leads to FDR control. If the tasks differ yet units are exchangeable within each task (which is a very mild condition), one can apply our framework within each task, similar to Mondrian Conformal Prediction, and this leads to FDR control conditional on each task. From the empirical perspective, it would be an interesting open question to design an efficient predictor g(X) to measure the alignment of a multivariate target. In this case, we may need to incorporate information about the task into the features as well. 

- **Q12: Dependency**. Thank you for the insightful comment! The only requirement of our method is exchangeability among calibration and test units, so if the inputs are dependent but exchangeable, our theory still applies. In the chatbot example, if we view each conversation as a unit (and aim at selecting conversations), then we still have exchangeability across units. We acknowledge that the exchangeability structure might be broken if the units correspond to former and latter parts of a series of discussions. This issue could be understood as (i) temporal shift in an online setting, or (ii) covariate shift where the covariate is the prior context and history. For each perspective, there are potential remedies. First, if we view it as an online selection problem, we might use ideas in online multiple testing to hedge the FDR budget as data accumulates. Second, if we view it as a covariate shift problem, then conformal selection can be extended to this setting [9], and the problem boils down to estimating a weight function that adjusts for the covariate shift. Both perspectives can bring new techniques to be incorporated into our current framework, and we will add a discussion about this point in our updated draft. 

- **Q13: Limitation section**. Thank you for the constructive suggestion! In our updated manuscript, we will add discussions on the following points in a new limitation section: 
    - More broad definition of alignment and prediction such as RLHF
    - New ideas for handling distribution shift and in-context learning
    - Evaluating our method with more powerful foundation models and larger-scale datasets
- **Q4: Sample size/Power**. Thank you for this thoughtful question! We totally agree that the sample size is a critical consideration in practice, and that is why we discussed this issue via (i) theoretical analysis and (ii) numerical experiments. In the following, we first summarize the empirical and theoretical results we establish, and discuss further plans to address this challenge. 
    1. **Current results**. First, our experimental results show that, with reasonably powerful models (which are smaller than state-of-the-art, ultra-large models), we can achieve satisfactory power with a few hundred samples. When the number of such data is limited, our method honestly reflects the amount of confidence we have in the alignment status of the samples (if the power is low), as opposed to blindly deploying the outputs without guarantees. Second, as suggested by Proposition 3.3, the two key driving factors of the power are the predictive power of the foundation model and the quality of the alignment score predictor $g(\cdot)$. We expect that one could boost the power of the whole pipeline by improving the quality of the foundation model or working with more powerful foundation models, if this is feasible. 
    2. **Plan for addressing the challenge**. We totally agree that one important future direction is to develop methods that could improve the performance of the selection procedure when the training sample size is limited. We outline a few ideas in the following. 
        - *Improve base foundation model*. Since the power of selection improves with the power of models, we anticipate that if the foundation model itself is more powerful, fewer samples would be needed for training. 
        - *Find more informative features*. Another important factor is whether the features $X$ obtained from the foundation model are informative/predictive of the unknown alignment score. While we currently use existing ideas in the literature for consistency, it will be helpful to identify better features. 
        - *Improve data usage*. The current framework splits labeled data into calibration and training fold, which reduces the sample size in alignment predictor training. However, it might be possible to leverage ideas similar to full conformal prediction to make sure (almost) all labeled data can be used in training. This would greatly increase the computation complexity, yet might be worthwhile if the labeled data are quite limited.
        - *Improve method development*. We may also extend conformal selection to allow for more flexible and powerful selection procedures. For example, we may use the covariates in the calibration data to increase the effective training sample size while preserving the rigorous error control of the framework. We leave this for future methodological work.
    
    In summary, we have added a discussion on this problem and outline possible ideas for improvement in our revision.

- **Q5: elaborating on relevance**. The statement is made in comparison with other works that try to connect conformal inference with LLM/foundation models. The main point is how practitioners might use the outputs of these algorithms and what guarantees make sense. While these existing works provide uncertain quantification (similar to prediction sets in conformal prediction), the guarantees they provide usually do not account for downstream implications, i.e., how to use them in subsequent tasks, which is the focus of this paper. Among the representative ideas in the literature [4] applies conformal prediction to obtain a less specific statement than the original generated response that is guaranteed to be correct (but possibly less informative) — however, this might make originally powerful outputs more vague, which hinders downstream use. On the other hand, [3] samples multiple outputs until one of them is reliable with high probability. However, practitioners still have to decide which one is the “good” output, among multiple of them; if one only uses [3] when the set of outputs is a singleton, then the adopted ones suffer from high error (please refer to our answer to Q2 of R5 for new numerical evidence, which is also in Figure 2 of the attached pdf).  
    In contrast, by FDR control, we ensure that practitioners can directly use the outputs selected by our method in downstream tasks, because most of them (such as 90%) are correct. This is without any modification to the original output, which preserves the sharpness of the foundation models. 

We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!

**Reference** Please see references in the global rebuttal.
**Q3: FDR target rate**.Thank you for raising this important practical issue! In such high-stakes decisions, we would expect the practitioners to work with a relatively stringent FDR target level; however, we would still like to leave the judgment of an appropriate FDR level to practitioners. Meanwhile, we want to emphasize that the power of the whole pipeline mostly depends on that of the foundation model (please refer to Proposition 3.3 of our manuscript for an explicit characterization of the asymptotic power of our method). Since we enforce error control, its tradeoff with power is inevitable.  Indeed, if the foundation model is unreliable, then it is reasonable that few of its outputs should be deployed in practice (otherwise there will be many *unacceptable* outputs being used, without any alert). In this sense, we provide a rigorous way of distinguishing acceptable outputs against unacceptable ones, adding a layer of protection for the deployment of the foundation model in high-stakes tasks. Such a tradeoff is similar to original conformal prediction: there, if the prediction model (the foundation model in our context) is not accurate, then it necessitates large prediction sets to achieve a prescribed coverage probability. Finally, we anticipate that very large, state-of-the-art models should lead to higher power; while the models we use here are on par with works in the literature [3], we still expect much space for improvement in terms of the quality of the foundation model. 

We sincerely hope these comments help clarify your questions and improve your evaluation of our work. Please do not hesitate to let us know if you have any other comments/questions!