Efficient Conformal Prediction via Cascaded Inference with Expanded Admission

1 INTRODUCTION . The ability to provide precise performance guarantees is critical to many classification tasks ( Amodei et al. , 2016 ; Jiang et al. , 2012 ; 2018 ) . Yet , achieving perfect accuracy with only single guesses is often out of reach due to noise , limited data , insufficient modeling capacity , or other pitfalls . Nevertheless , in many applications , it can be more feasible and ultimately as useful to hedge predictions by having the classifier return a set of plausible options—one of which is likely to be correct . Consider the example of information retrieval ( IR ) for fact verification . Here the goal is to retrieve a snippet of text of some granularity ( e.g. , a sentence , paragraph , or article ) that can be used to verify a given claim . Large resources , such as Wikipedia , can contain millions of candidate snippets—many of which may independently be able to serve as viable evidence . A good retriever should make precise snippet suggestions , quickly—but do so without excessively sacrificing sensitivity ( i.e. , recall ) . Conformal prediction ( CP ) is a methodology for placing exactly that sort of bet on which candidates to retain ( Vovk et al. , 2005 ) . Concretely , suppose we have been given n examples , ( Xi , Yi ) ∈ X ×Y , i = 1 , . . . , n , as training data , that have been drawn exchangeably from an underlying distribution P . For instance , in our IR setting , X would be the claim in question , Y a viable piece of evidence that supports or refutes it , and Y a large corpus ( e.g. , Wikipedia ) . Next , let Xn+1 be a new exchangeable test example ( e.g. , a new claim to verify ) for which we would like to predict the paired y ∈ Y . The aim of conformal prediction is to construct a set of candidates Cn ( Xn+1 ) likely to contain Yn+1 ( e.g. , the relevant evidence ) with distribution-free marginal coverage at a tolerance level ∈ ( 0 , 1 ) : P ( Yn+1 ∈ Cn ( Xn+1 ) ) ≥ 1− ; for all distributions P. ( 1 ) The marginal probability above is taken over all the n+ 1 calibration and test points { ( Xi , Yi ) } n+1i=1 . A classifier is considered to be valid if the frequency of error , Yn+1 6∈ Cn ( Xn+1 ) , does not exceed . In our IR setting , this would mean including the correct snippet at least -fraction of the time . Not all valid classifiers , however , are particularly useful ( e.g. , a trivial classifier that merely returns all ∗Equal contribution ( author order decided randomly ) . 1Our code is available at https : //github.com/ajfisch/conformal-cascades . possible outputs ) . A classifier is considered to have good predictive efficiency if E [ |Cn ( Xn+1 ) | ] is small ( i.e. , |Y| ) . In our IR setting , this would mean not returning too many irrelevant articles—or in IR terms , maximizing precision while holding the level of recall at ≥ 1− ( assuming Y is a single answer ) . In practice , in domains where the number of outputs to choose from is large and the “ correct ” one is not necessarily unique , classifiers derived using conformal prediction can suffer dramatically from both poor predictive and computational efficiency ( Burnaev and Vovk , 2014 ; Vovk et al. , 2016 ; 2020 ) . Unfortunately , these two conditions tend to be compounding : large label spaces Y both ( 1 ) often place strict constraints on the set of tractable model classes available for consideration , and ( 2 ) frequently contain multiple clusters of labels that are difficult to discriminate between , especially for a low-capacity classifier . In this paper , we present two effective methods for improving the efficiency of conformal prediction for classification tasks with large output spaces Y , in which several y ∈ Y might be admissible—i.e. , acceptable for the purposes of our given task . First , in Section 4 we describe a generalization of Eq . 1 to an expanded admission criteria , where Cn ( Xn+1 ) is considered valid if it contains at least one admissible y with high probability . For example , in our IR setting , given the claim “ Michael Collins took part in the Apollo mission to the moon , ” any of the articles “ Apollo 11 , ” “ Michael Collins ( astronaut ) , ” or “ Apollo 11 ( 2019 film ) ” have enough information to independently support it ( see Figure 1 ) —and are therefore all admissible . When Yn+1 is not unique , forcing the classifier to hedge for the worst case , in which a specific realization of Yn+1 must be contained in Cn ( Xn+1 ) , is too strict and can lead to conservative predictions . We theoretically and empirically show that optimizing for an expanded admission criteria yields classifiers with significantly better predictive efficiency . Second , in Section 5 we present a technique for conformalizing prediction cascades to progressively filter the number of candidates with a sequence of increasingly complex classifiers . This allows us to balance predictive efficiency with computational efficiency during inference . Importantly , we also theoretically show that , in contrast to other similarly motivated pipelines , our method filters the output space in a manner that still guarantees marginal coverage . Figure 1 illustrates our combined approach . We demonstrate that , together , these two approaches serve as complementary pieces of the puzzle towards making CP more efficient . We empirically validate our approach on information retrieval for fact verification , open-domain question answering , and in-silico screening for drug discovery . Contributions . In summary , our main results are as follows : • A theoretical extension of validity ( Eq . 1 ) to allow for inferred admissible answers . • A principled framework for conformalizing computationally efficient prediction cascades . • Consistent empirical gains on three diverse tasks demonstrating up to 4.6× better predictive efficiency AUC ( measured across all ) when calibrating for expanded admission , with computation pruning factors of up to 1/m , where m is the number of models , when using prediction cascades . 2 RELATED WORK . Confident prediction . Methods for obtaining precise uncertainty estimates have received intense interest in recent years . A significant body of work is concerned with calibrating model confidence— measured as pθ ( ŷn+1|xn+1 ) —such that the true accuracy , yn+1 = ŷn+1 , is indeed equal to the estimated probability ( Niculescu-Mizil and Caruana , 2005 ; Gal and Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ; Lee et al. , 2018 ) . In theory , these estimates could be leveraged to create confident prediction sets Cn ( Xn+1 ) . Unlike CP , however , these methods are not guaranteed to be accurate , and often still suffer from miscalibration in practice—especially for modern neural networks ( Guo et al. , 2017 ; Ashukha et al. , 2020 ; Hirschfeld et al. , 2020 ) . Selective classification ( ElYaniv and Wiener , 2010 ; Geifman and El-Yaniv , 2017 ) , where models have the option to abstain from answering when not confident , is similar in motivation to Eq . 1 . In fact , it can be considered as a special case in which the classifier chooses to abstain unless |Cn ( Xn+1 ) | = 1 . Conformal prediction . As validity is already guaranteed by design in conformal prediction , most efforts in CP focus on improving various aspects of efficiency . Mondrian CP ( Vovk et al. , 2005 ) accounts for the fact that some classes are harder to model than others , and leverages class-conditional statistics . Similiarly , several recent studies have built towards conditional—as opposed to marginal— coverage through various adaptive approaches , such as conformalizing quantile functions or working with conditional distributions that vary with x ( see Cauchois et al. , 2020 ; Chernozhukov et al. , 2019 ; Kivaranovic et al. , 2020 ; Romano et al. , 2019 ; 2020 , inter alia ) . Cauchois et al . ( 2020 ) also directly model dependencies among y variables for use in multi-label prediction . Our method for expanded admission , on the other hand , aggregates statistics for equivalent single labels by example and across classes . Though we only provide marginal guarantees , the ideas expressed in those related works are complementary , and can be applied here as well . Inductive CP ( Papadopoulos , 2008 ) is also complementary extension that dramatically reduces the cost of computing Cn ( Xn+1 ) in the general case ; we make use of it here . Most similar to our work , trimmed ( Chen et al. , 2016 ) and discretized ( Chen et al. , 2018 ) CP trade predictive efficiency for computational efficiency in regression tasks , where the label space is infinite . A key distinction of our method is that we do not force the same trade-off : in fact , we empirically show that our conformalized cascades can at times result in better predictive efficiency alongside a pruned label space . Prediction cascades . The idea of balancing cost with accuracy by using multi-step inference has been explored extensively for many applications ( Charniak et al. , 2006 ; Deng and Rush , 2020 ; Fleuret and Geman , 2001 ; Jurafsky and Martin , 2000 ; Li et al. , 2015 ; Rush and Petrov , 2012 ) . Some of these methods use fixed rules with no performance guarantees , such as greedy pipelines where the top k predictions are passed on to the next level ( Chen et al. , 2017 ; Ferrucci et al. , 2010 ) . Closer to our work , Weiss and Taskar ( 2010 ) optimize their cascades for overall pruning efficiency , and not for top-1 prediction . While they also analyze error bounds for filtering , their guarantees are specific to linear classifiers with bounded L2 norm , whereas our conformalized approach only assumes data exchangeability . Furthermore , they assume a target filtering loss before training—our tolerance level is defined at test time , which allows for much greater flexibility . 3 BACKGROUND . We begin with a brief review of conformal prediction ( see Shafer and Vovk , 2008 ) . Here , and in the rest of the paper , upper-case letters ( X ) denote random variables ; lower-case letters ( x ) denote scalars , and script letters ( X ) denote sets , unless otherwise specified . Proofs are deferred to the appendix . At the core of conformal prediction is a simple statistical hypothesis test : for each candidate y ∈ Y we must either accept or reject the null hypothesis that ( Xn+1 = xn+1 , Yn+1 = y ) is a correct pairing . Formally , we rely on a nonconformity measure S ( ( xn+1 , y ) , D ) to serve as the test statistic , where a higher value of S reflects that ( xn+1 , y ) is less “ conforming ” to the distribution specified by dataset D. For instance , S could be computed as − log pθ ( y|x ) , where θ is a model fit to D. Definition 3.1 ( Nonconformity measure ) . Let Z : = X ×Y be the space of examples ( X , Y ) , and let Z ( ∗ ) : = ⋃ d≥1 ( X ×Y ) d be the space of datasets of examples D , of any size d ≥ 1 . A nonconformity measure S is then a measurable mapping S : Z × Z ( ∗ ) → R , that assigns a real-valued score to any example ( X , Y ) , indicating how different2 it is from a reference dataset D. Furthermore , in order to retain exchangeability , S is symmetric with respect to permutations of its input data . To be specific , exact or full CP takes D to be all of the examples seen so far , including the candidate ( xn+1 , y ) . Thus , the nonconformity measure S has to be re-trained each time . An alternative—which 2The definition of “ different ” here is intentionally vague , as any metric will technically work . we use in this paper w.l.o.g.—is the inductive or split CP variant ( Papadopoulos , 2008 ) which assumes that D is a proper training set , independent of any of the subsequent n+ 1 exchangeable examples used for CP . Dropping D for ease of notation , we denote the score for example ( X , Y ) as the random variable S ( X , Y ) . The degree of nonconformity can then be quantified using a p-value . Lemma 3.2 ( Smoothed p-value ) . Assume that the random variables V1 , . . . , Vn+1 are exchangeable . We define the smoothed empirical p-value pvalue ( Vn+1 , V1 : n ) as pvalue ( Vn+1 , V1 : n ) : = | { i ∈ [ 1 , n ] : Vi > Vn+1 } |+ τ · | { i ∈ [ 1 , n ] : Vi = Vn+1 } |+ 1 n+ 1 , ( 2 ) where τ ∼ U ( 0 , 1 ) . Then , for any ∈ ( 0 , 1 ) , we have P ( pvalue ( Vn+1 , V1 : n ) ≤ ) ≤ . To construct the final conformal prediction , the classifier uses the p-values to include all y for which the null hypothesis—i.e. , that the candidate pair ( xn+1 , y ) is conformal—is not rejected . Theorem 3.3 ( CP ; Vovk et al . ( 2005 ) , see also Lei et al . ( 2018 ) ) . Assume that the random variables ( Xi , Yi ) ∈ X × Y , i = 1 , . . . , n + 1 are exchangeable . For any nonconformity measure S , and ∈ ( 0 , 1 ) , define the conformal label set ( based on the first n samples ) at xn+1 ∈ X as Cn ( xn+1 ) : = { y ∈ Y : pvalue ( S ( xn+1 , y ) , S ( x1 : n , y1 : n ) ) > } . ( 3 ) Then Cn ( Xn+1 ) satisfies Eq . 1 , where P ( Yn+1 ∈ Cn ( Xn+1 ) ) ≥ 1− .

Summary: This paper presents two advances in conformal prediction, a field with information retrieval applications in which a set of candidate responses to a query is presented and the objective is to return a small set of responses with at least one of the responses being the correct response.  The first contribution is a method in which the possibility of several admissible responses is modeled (rather than there being just one response) with the system being calibrated against the odds that a particular response is the "most admissible", i.e. most conforming to the joint query/response distribution being learned from data. The second contribution is a cascaded prediction system in which simpler and less computationally expensive models are used for initial filtering and then more sophisticated and more expensive models are used downstream to further refine the response set. Rigorous statistical adjustments are used to account for multiple-hypothesis-test issues arising from using the cascading system.