Learning Optimal Conformal Classifiers

1 INTRODUCTION . In classification tasks , for input x , we approximate the posterior distribution over classes y ∈ [ K ] : = { 1 , . . . , K } , denoted πy ( x ) ≈ P ( Y = y|X = x ) . Following Bayes ’ decision rule , the single class with highest posterior probability is predicted for optimizing a 0-1 classification loss . This way , deep networks πθ , y ( x ) with parameters θ achieve impressive accuracy on held-out test sets . However , this does not guarantee safe deployment . Conformal prediction ( CP ) ( Vovk et al. , 2005 ) uses a posttraining calibration step to guarantee a user-specified coverage : by allowing to predict confidence sets C ( X ) ⊆ [ K ] , CP guarantees the true class Y to be included with confidence level α , i.e . P ( Y ∈ C ( X ) ) ≥ 1 − α when the calibration examples ( Xi , Yi ) , i ∈ Ical are drawn exchangeably from the test distribution . This is usually achieved in two steps : In the prediction step , so-called conformity scores ( w.r.t . to a class k ∈ [ K ] ) are computed to construct the confidence sets C ( X ) . During the calibration step , these conformity scores on the calibration set w.r.t . the true class Yi are ranked to determine a cut-off threshold τ for the predicted probabilities πθ ( x ) guaranteeing coverage 1−α . This is called marginal coverage as it holds only unconditionally , i.e. , the expectation is being taken not only w.r.t . ( X , Y ) but also over the distribution of all possible calibration sets , rather than w.r.t . the conditional distribution p ( Y |X ) . CP also outputs intuitive uncertainty estimates : larger confidence sets |C ( X ) | generally convey higher uncertainty . Although CP is agnostic to details of the underlying model πθ ( x ) , the obtained uncertainty estimates depend strongly on the model ’ s performance . If the underlying classifier is poor , CP results in too large and thus uninformative confidence sets . “ Uneven ” coverage is also a common issue , where lower coverage is achieved on more difficult classes . To address such problems , the threshold CP method of ( Sadinle et al. , 2019 ) explicitly minimizes inefficiency . Romano et al . ( 2020 ) and Cauchois et al . ( 2020 ) propose methods that perform favorably in terms of ( approximate ) conditional coverage . The adaptive prediction sets ( APS ) method of Romano et al . ( 2020 ) is further extended by Angelopoulos et al . ( 2021 ) to return smaller confidence sets . These various objectives are typically achieved by changing the definition of the conformity scores . In all cases , CP is used as a post-training calibration step . In contrast , our work does not focus on advancing CP itself , e.g. , through new conformity scores , but develops a novel training procedure for the classifier πθ . After training , any of the above CP methods can readily be applied . Indeed , while the flexibility of CP regarding the underlying model appears attractive , it is also a severe limitation : Learning the model parameters θ is not informed about the post-hoc “ conformalization ” , i.e. , they are are not tuned towards any specific objective such as reducing expected confidence set size ( inefficiency ) . During training , the model will typically be trained to minimize cross-entropy loss . At test time , in contrast , it is used to obtain a set predictor C ( X ) with specific properties such as low inefficiency . In concurrent work , Bellotti ( 2021 ) addresses this issue by learning a set predictor C ( X ) through thresholding logits : Classes with logits exceeding 1 are included in C ( X ) and training aims to minimize inefficiency while targeting coverage 1 − α . In experiments using linear models only , this approach is shown to decrease inefficiency . However , ( Bellotti , 2021 ) ignores the crucial calibration step of CP during training and does not allow to optimize losses beyond marginal coverage or inefficiency . In contrast , our work subsumes ( Bellotti , 2021 ) , but additionally considers the calibration step during training , which is crucial for further decreasing inefficiency . Furthermore , we aim to allow fine-grained control over class-conditional inefficiency or the composition of the confidence sets by allowing to optimize arbitrary losses defined on confidence sets . Our contributions can be summarized as follows : 1 . We propose conformal training ( ConfTr ) , a procedure allowing to train model and conformal wrapper end-to-end . This is achieved by developing smooth implementations of recent CP methods for use during training . On each mini-batch , ConfTr “ simulates ” conformalization , using half of the batch for calibration , and the other half for prediction and loss computation , c.f . Fig . 1 a© . After training , any existing CP method can provide a coverage guarantee . 2 . In experiments , using ConfTr for training consistently reduces the inefficiency of conformal predictors such as threshold CP ( THR ) ( Sadinle et al. , 2019 ) or APS ( Romano et al. , 2020 ) applied after training . We further improve over ( Bellotti , 2021 ) , illustrating the importance of the calibration step during training . 3 . Using carefully constructed losses , ConfTr allows to “ shape ” the confidence sets obtained at test time : We can reduce class-conditional inefficiency or “ coverage confusion ” , i.e. , the likelihood of two or more classes being included in the same confidence sets , c.f . Fig . 1 b© . Generally , in contrast to ( Bellotti , 2021 ) , ConfTr allows to optimize arbitrary losses on the confidence sets . Because ConfTr is agnostic to the CP method used at test time , our work is complementary to most related work , i.e. , any advancement in terms of CP is directly applicable to ConfTr . For example , this might include conditional or application-specific guarantees as in ( Sadinle et al. , 2016 ; Bates et al. , 2021 ) . Most importantly , ConfTr preserves the coverage guarantee obtained through CP . 2 DIFFERENTIABLE CONFORMAL PREDICTORS . We are interested in training the model πθ end-to-end with the conformal wrapper in order to allow fine-grained control over the confidence sets C ( X ) . Before developing differentiable CP methods for training in Sec . 2.2 , we review two recently proposed conformal predictors that we use at test time . These consist of two steps , see Sec . 2.1 : for prediction ( on the test set ) we need to define the confidence sets Cθ ( X ; τ ) which depend on the model parameters θ through the predictions πθ and where the threshold τ is determined during calibration on a held-out calibration set ( Xi , Yi ) , i ∈ Ical in order to obtain coverage . 2.1 CONFORMAL PREDICTORS . The threshold conformal predictor ( THR ) ( Sadinle et al. , 2019 ) constructs the confidence sets by thresholding probabilities : Cθ ( x ; τ ) : = { k : πθ , k ( x ) = : Eθ ( x , k ) ≥ τ } . Here , the subscript Cθ makes the dependence on the model πθ and its parameters θ explicit . During calibration , τ is computed as the α ( 1 + 1/|Ical| ) -quantile of the so-called conformity scores Eθ ( xi , yi ) = πθ , yi ( xi ) . The conformity scores indicate , for each example , the threshold that ensures coverage . Marginal coverage of ( 1 − α ) is guaranteed on a test example ( X , Y ) . In practice , THR can also be applied on logits ( THRL ) or log-probabilities ( THRLP ) instead of probabilities . Adaptive Prediction Sets ( APS ) ( Romano et al. , 2020 ) constructs confidence sets based on the ordered probabilities . Specifically , Cθ ( x ; τ ) : = { k : Eθ ( x , k ) ≤ τ } with : Eθ ( x , k ) : = πθ , y ( 1 ) ( x ) + . . .+ πθ , y ( k−1 ) ( x ) + Uπθ , y ( k ) ( x ) , ( 1 ) where πθ , y ( 1 ) ( x ) ≥ . . . ≥ πθ , y ( K ) ( x ) and U is a uniform random variable in [ 0 , 1 ] to break ties . Similar to THR , the conformity scores Eθ ( xi , yi ) w.r.t . the true classes yi are used for calibration , but the ( 1− α ) ( 1 + 1/|Ical| ) -quantile is required to ensure marginal coverage on test examples . Performance of CP is then measured using two metrics : ( empirical and marginal ) coverage ( Cover ) as well as inefficiency ( Ineff ) . Letting Itest be a test set of size |Itest| , these metrics are computed as Cover : = 1 |Itest| ∑ i∈Itest δ [ yi ∈ C ( xi ) ] and Ineff : = 1 |Itest| ∑ i∈Itest |C ( xi ) | , ( 2 ) where δ denotes an indicator function that is 1 when its argument is true and 0 otherwise . Due to the marginal coverage guarantee provided by CP ( c.f . ( Romano et al. , 2020 ) or App . C ) , the empirical coverage , when averaged across several calibration/test splits , is Cover ≈ 1− α . Thus , we concentrate on inefficiency as the main metric to compare across CP methods and models . With accuracy , we refer to the ( top-1 ) accuracy with respect to the arg max-predictions , i.e. , arg maxk πθ , k ( x ) , obtained by the underlying model π . As shown in Fig . 2 ( left ) , THR clearly outperforms THRL and APS w.r.t . inefficiency ( lower is better ) averaged across random Ical/Itest splits ( details in Sec . F ) . CP is intended to be used as a “ wrapper ” around πθ . “ Better ” CP methods generally result in lower inefficiency for a fixed model πθ . For example , following Fig . 2 ( left ) , regularized APS ( RAPS ) ( Angelopoulos et al. , 2021 ) recently showed how to improve inefficiency compared to APS by modifying the conformity score – without outperforming THR , however . Fine-grained control over inefficiency , e.g. , conditioned on the class or the composition of the C ( X ) is generally not possible . Integrating CP into the training procedure promises a higher degree of control , however , requires differentiable CP implementations , e.g. , for THR or APS . 1 : function PREDICT ( πθ ( x ) , τ ) 2 : compute Eθ ( x , k ) , k∈ [ K ] 3 : return Cθ ( x ; τ ) = { k : Eθ ( x , k ) ≥ τ } 1 : function CALIBRATE ( { ( πθ ( xi ) , yi } ni=1 , α ) 2 : compute Eθ ( xi , yi ) , i=1 , . . . , n 3 : return QUANTILE ( { Eθ ( xi , yi ) } , α ( 1 + 1/n ) ) 1 : function SMOOTHPRED ( πθ ( x ) , τ , T=1 ) 2 : return Cθ , k ( x ; τ ) =σ ( ( Eθ ( x , k ) −τ ) T ) , k∈ [ K ] 3 : function SMOOTHCAL ( { ( πθ ( xi ) , yi } ni=1 , α ) 4 : return SMOOTHQUANT ( { Eθ ( xi , yi ) } , α ( 1+ 1n ) ) 1 : function CONFORMALTRAINING ( α , λ=1 ) 2 : for mini-batch B do 3 : randomly split batch Bcal ] Bpred = B 4 : { “ On-the-fly ” calibration on Bcal : } 5 : τ = SMOOTHCAL ( { ( πθ ( xi ) , yi ) } i∈Bcal , α ) 6 : { Prediction only on i ∈ Bpred : } 7 : Cθ ( xi ; τ ) = SMOOTHPRED ( πθ ( xi ) , τ ) 8 : { Optional classification loss : } 9 : LB = 0 or ∑ i∈Bpred L ( Cθ ( xi ; τ ) , yi ) 10 : ΩB = ∑ i∈Bpred Ω ( Cθ ( xi ; τ ) ) 11 : ∆ =∇θ1/|Bpred| ( LB + λΩB ) 12 : update parameters θ using ∆ Algorithm 1 : Smooth CP and Conformal Training ( ConfTr ) : Top left : At test time , for THR , PREDICT computes the conformity scores Eθ ( x , k ) for each k∈ [ K ] and constructs the confidence sets Cθ ( x ; τ ) by thresholding with τ . CALIBRATE determines the threshold τ as the α ( 1 + 1/n ) quantile of the conformity scores w.r.t . the true classes yi on a calibration set { ( xi , yi ) } of size n : =|Ical| . THR and APS use different conformity scores . Right and bottom left : ConfTr calibrates on a part of each mini-batch , Bcal . Thereby , we obtain guaranteed coverage on the other part , Bpred ( in expectation across batches ) . Then , the inefficiency on Bpred is minimized to update the model parameters θ . Smooth implementations of calibration and prediction are used .

Summary: Conformal prediction methods generally operate via post-processing on black-box classifiers to produce a confidence set. In this paper, the authors integrate this post-processing step into the training procedure in order to reduce the inefficiency of conformal prediction while providing the same coverage guarantee. The proposed method uses soft thresholding and differentiable sorting to convert two popular conformal post-processing method classess (THR - Sadinle et al. and APS - Romano et al.) into differentiable modules, although they only use THRLP (the method from Sadinle et al. applied to log-probabilities) in the training process for the results (in Table 1).  Additionally, they propose the use of weighted inefficiency regularization (also termed the size loss) and a "configurable" classification loss that can yield desirable behavior w.r.t. class-conditional inefficiency and mis-coverage (or coverage confusion) respectively.  The proposed method is shown to outperform a competing method by Belloti on 5 public multi-class datasets (including CIFAR100) and a binary dataset (wineQuality).