Adversarially Robust Conformal Prediction

1 INTRODUCTION . Deep neural net classifiers have achieved tremendous accomplishments over the last several years . Nevertheless , the increased deployment of these algorithms in real-world applications , especially ones that can be life-threatening like autonomous driving , raises major concerns about their reliability ( Heaven , 2019 ) . To alleviate these issues , it is important to develop techniques that allow the users to assess the uncertainty in predictions obtained by complex classifiers , revealing their limitations . Conformal prediction ( Vovk et al. , 2005 ) is a simple yet powerful tool for generating prediction sets whose size reflects the prediction uncertainty . Specifically , suppose we are given n training examples { ( Xi , Yi ) } ni=1 with feature vector Xi ∈ Rd , discrete and unordered class label Yi∈ { 1 , 2 , . . . , L } = Y , and any learning algorithm that aims at predicting the unknown Yn+1 of a given test point Xn+1 . Under the assumption that the training and test examples are sampled exchangeably—e.g. , they may be drawn i.i.d.—from an unknown distribution PXY , conformal prediction algorithms construct a distribution-free prediction set C ( Xn+1 ) ⊆ Y guaranteed to contain the test label Yn+1 at any desired coverage probability 1− α ∈ ( 0 , 1 ) : P [ Yn+1 ∈ C ( Xn+1 ) ] ≥ 1− α . ( 1 ) For example , it is common to set the desired coverage level 1 − α to be 90 % or 95 % . Note that the coverage probability P [ Yn+1 ∈ C ( Xn+1 ) ] is marginal because it is taken over all the training examples { ( Xi , Yi ) } ni=1 and the test point Xn+1 . The key idea of conformal prediction is to fit a classifier on the training set and use this model to assign non-conformity scores for held-out data points . These scores reflect the prediction error of the underlying classifier , where , loosely speaking , a smaller prediction error would lead to the construction of smaller and more informative sets . However , the sets constructed by the vanilla conformal method may not have the right coverage when the training and test points violate the exchangeability assumption ( Cauchois et al. , 2020 ; Gibbs & Candès , 2021 ; Tibshirani et al. , 2019 ; Podkopaev & Ramdas , 2021 ; Guan & Tibshirani , 2019 ) , which is hardly satisfied by real data in practice as distribution shift happens frequently ( Koh et al. , 2021 ) . In particular , we consider the potential threat of adversarial attacks ( Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ; Carlini & Wagner , 2017 ) —carefully crafted human-imperceptible noise perturbations that drives the fitted model to err at test ( inference ) time . Such noise perturbations can introduce a large and arbitrary distribution shift that is extremely hard to estimate . In this setting , the prediction sets constructed by the vanilla conformal approach are often invalid , i.e. , do not satisfy ( 1 ) , as illustrated in Figure 1 . Following that figure , while this method achieves the desired 90 % coverage when applied to clean test data , the empirical coverage obtained when applying the same method on adversarial test examples falls dramatically below 90 % , to about 30 % . Therefore , it is the main motivation of this work to construct prediction sets that are robust to adversarial attacks . We formalize this requirement as follows : P [ Yn+1 ∈ Cδ ( X̃n+1 ) ] ≥ 1− α , ( 2 ) where X̃n+1 = Xn+1 + is the test adversarial example , and ‖ ‖2 ≤ δ is a norm-bounded adversarial perturbation . Note that we use the notation Cδ to distinguish between the new setting from the exchangeability case for which δ = 0 . Crucially , we require ( 2 ) to hold in finite samples , for any PXY , any adversarial perturbation of magnitude bounded by δ , and regardless of the choice or accuracy of the underlying classifier . At the same time , we wish Cδ to be as small as possible . To realize ( 2 ) with the desired coverage , we propose to combine randomized smoothing ( Duchi et al. , 2012 ; Cohen et al. , 2019 ; Salman et al. , 2019 ) with the vanilla conformal prediction procedure , and hence we name our technique Randomly Smoothed Conformal Prediction ( RSCP ) . Randomized smoothing allows us to bound the Lipschitz constant of any non-conformity score function by convolving it with the Gaussian distribution function . Leveraging this bound , we show how to modify conformal prediction and rigorously construct prediction sets that account for the adversarial perturbation . Figure 1 illustrates that our proposed RSCP approach successfully attains the desired coverage level whereas the vanilla conformal method fails . Observe that RSCP constructs slightly larger prediction sets , reflecting the increased uncertainty induced by the adversarial noise . The contributions of this paper are two-fold : ( i ) We propose , for the first time , a new conformal prediction method that can account for the potential adversarial threats during inference time . Our RSCP method , described in Section 3 , is model-agnostic in that it can work with any classifier and non-conformity score , scalable since the smoothing can be done by Monte Carlo integration with many i.i.d . Gaussian noise realizations , and robust against any ` 2-norm bounded adversarial attack . ( ii ) We prove that the prediction sets constructed by RSCP are valid in the sense of ( 2 ) , and , in Section 5 , support this theoretical result with numerical experiments on CIFAR10 , CIFAR100 , and ImageNet data sets . 2 CONFORMAL PREDICTION . Since the focus of this paper is on how to adapt the vanilla conformal prediction method to the adversarial setting , in this section , we give background on conformal prediction . While we focus here on classification problems , the method of conformal prediction can also be applied to regression tasks . The vanilla conformal prediction can be divided into two categories : the split conformal prediction and the full conformal prediction . The first one involves data splitting while the second one constructs valid sets without splitting the data but at the cost of a significant increase in computational complexity at test time ( Papadopoulos et al. , 2002 ; Papadopoulos , 2008 ; Vovk et al. , 2005 ) . To avoid prohibitive computation complexity , in this paper we focus only on split conformal prediction . This method starts by splitting the training data into two disjoint subsets : a proper training set Itr ⊆ { 1 , . . . , n } and a calibration set Ical = { 1 , . . . , n } \ Itr . Then , a classifier f̂ ( x ) ∈ [ 0 , 1 ] L is fit to the proper training set , estimating the conditional class probabilities P [ Y = y | X = x ] for all y ∈ Y . In the case of deep net classifiers , which is the focus of this work , this may be the output of the softmax layer . Next , we compute a non-conformity score Si = S ( Xi , Yi ) ∈ R for each calibration point { ( Xi , Yi ) } i∈Ical . This score expresses how well the model prediction f̂ ( X ) is aligned with the true label Y , where a lower score implies better alignment . For example , the score from Vovk et al . ( 2005 ) ; Lei et al . ( 2013 ) is given by S ( x , y ) = 1− f̂ ( x ) y , ( 3 ) where f̂ ( x ) y ∈ [ 0 , 1 ] is the yth entry in the vector f̂ ( x ) . Another example is the score proposed by Romano et al . ( 2020b ) , which can be expressed as S ( x , y ) = ∑ y′∈Y f̂ ( x ) y′ I { f̂ ( x ) y′ > f̂ ( x ) y } + f̂ ( x ) y · u , ( 4 ) where I is the indicator function and u is a random variable distributed uniformly over the segment [ 0 , 1 ] . We refer to the score from ( 3 ) as HPS as it was shown to construct homogeneous prediction sets . Analogously , we refer to ( 4 ) as APS since it tends to yield adaptive prediction sets that reflect better the underlying uncertainty across sub-populations ; see Romano et al . ( 2020b ) for more details . Given the desired coverage level 1−α , the prediction set for a new test point Xn+1 is formulated as C ( Xn+1 ) = { y ∈ Y : S ( Xn+1 , y ) ≤ Q1−α ( { Si } i∈Ical ) } , ( 5 ) where Q1−α ( { Si } i∈Ical ) : = the ( 1− α ) ( 1 + 1 1 + |Ical| ) th empirical quantile of { Si } i∈Ical ( 6 ) is the score positioned d ( n+ 1 ) ( 1− α ) e in the sorted array of calibration scores Si , i ∈ Ical . In plain words , in ( 5 ) we sweep over all possible labels y ∈ Y and include in C ( Xn+1 ) the ‘ guessed ’ labels y whose scores S ( Xn+1 , y ) are smaller than most of the calibration scores S ( Xi , Yi ) . Since the calibration and test points are drawn exchangeably from PXY and f̂ is fixed , the score S ( Xn+1 , y ) for the guess y = Yn+1 can fall anywhere in the sorted array of calibration scores with equal probability . This property guarantees that the prediction set ( 5 ) satisfies ( 1 ) ; see Vovk et al . ( 2005 ) . 3 RANDOMLY SMOOTHED CONFORMAL PREDICTION . In this section , we introduce our proposed RSCP framework for constructing prediction sets that are valid in the adversarial regime . Recall that an adversarial attack can lead to a significant distributional shift between the clean calibration points and the corrupted test example X̃n+1 = Xn+1 + , thus violating the fundamental exchangeability assumption of the split conformal procedure . Focusing on the guess y = Yn+1 , an effective attack would result in a larger non-conformity score for the corrupted test point S ( X̃n+1 , y ) compared to that of the clean input S ( Xn+1 , y ) . Therefore , a naive comparison of S ( X̃n+1 , y ) to the same threshold Q1−α from ( 6 ) , which neglects the increased uncertainty caused by the adversarial perturbation , will result in a prediction set that may not achieve the desired coverage , as already illustrated in Figure 1 . To address this , we should compare the test score to an inflated threshold , larger than Q1−α , which rigorously accounts for the effect of the adversarial noise . This is the core idea behind our proposal described in detail below ; see Figure 2 . 3.1 ADVERSARIALLY ROBUST CALIBRATION . Suppose we are given a non-conformity score function S̃ for which we can bound by how much its value could be increased due to the adversarial noise ‖ ‖2 ≤ δ added toXn+1 . Formally , we require the score S̃ to satisfy the following relation : S̃ ( X̃n+1 , y ) ≤ S̃ ( Xn+1 , y ) +Mδ , ∀y ∈ Y , ( 7 ) where Mδ ≥ 0 is a constant that is a function of δ , such that Mδ1 ≥ Mδ2 for δ1 ≥ δ2 and Mδ = 0 for δ = 0 . In essence , we would like ( 7 ) to hold for the smallest possible Mδ . We denote this score function by S̃ to emphasize that it must satisfy ( 7 ) , distinguishing it from existing non-conformity scores S , e.g. , ( 3 ) – ( 4 ) ; see Section 3.2 for a concrete and very general framework for designing S̃ for which the constant Mδ can be easily derived . Importantly , Mδ serves as a bridge between the observed score S̃ ( X̃n+1 , y ) and the unobserved one S̃ ( Xn+1 , y ) for a fixed y ∈ Y . Leveraging this property , we propose to construct a prediction set robust to a norm-bounded adversarial attack by applying the following decision rule : Cδ ( X̃n+1 ) = { y ∈ Y : S̃ ( X̃n+1 , y ) ≤ Q1−α ( { S̃i } i∈Ical ) +Mδ } , ( 8 ) where S̃i = S̃ ( Xi , Yi ) . In contrast to the vanilla split conformal approach ( 5 ) , the prediction set defined above is generated by comparing the test score to an inflated threshold Q1−α + Mδ , as illustrated in Figure 2 . Notice that the level of inflation is affected by the magnitude of the adversarial perturbation as well as the robustness of S̃ , i.e. , the value of Mδ . A larger perturbation implies larger inflation , and a more robust score S̃ implies smaller inflation . The theorem below states that the constructed prediction set ( 8 ) is guaranteed to contain the unknown target label Yn+1 with a probability of at least 1 − α , for any distribution PXY , sample size n , score function S̃ that satisfies ( 7 ) , and adversarial perturbation of magnitude δ generated by any attack algorithm . The proof of this and all other results can be found in Section S1 of the Supplementary Material . Theorem 1 . Assume that the samples { ( Xi , Yi ) } n+1i=1 are drawn exchangeably from some unknown distribution PXY . Let X̃n+1 = Xn+1 + be a clean test example Xn+1 with an additive corruption of an ` 2-norm bounded adversarial noise ‖ ‖2 ≤ δ . Then , the prediction set Cδ ( X̃n+1 ) defined in ( 8 ) satisfies P [ Yn+1 ∈ Cδ ( X̃n+1 ) ] ≥ 1− α . Before presenting our framework to construct scores that rigorously satisfy ( 7 ) , we pause to prove a lower bound on the coverage that the vanilla split conformal could attain in the adversarial setting . Theorem 2 . Under the assumptions of Theorem 1 , the prediction set C ( X̃n+1 ) defined in ( 5 ) , applied with S̃ in place of S , satisfies P [ Yn+1 ∈ C ( X̃n+1 ) ] ≥ τ , where τ = max { τ ′ : Qτ ′ ( { S̃i } i∈Ical ) ≤ Q1−α ( { S̃i } i∈Ical ) −Mδ } . ( 9 ) Note that τ can be simply computed by running a grid search on the sorted array of calibration scores . It is important to observe that in contrast to Theorem 1 , which guarantees at least 1−α coverage for any user-specified level α , the worst coverage level τ of the vanilla split conformal is not controlled explicitly , i.e. , τ is known only after looking at the training and calibration data . Observe also that , by construction , τ ≤ 1 − α ( see Supplementary Section S1 ) and equality is met in a special case where δ = 0 , i.e. , no attack is performed . In this special case , Mδ = 0 by definition , and both Theorem 1 and 2 converge to the classic coverage guarantee of the vanilla conformal algorithm .

Summary: Quantifying predictive uncertainty is critical for various real-world applications of ML models. Post-hoc ``wrapper-style'' procedures for uncertainty quantification, which can be built on top of any black-box model and thus do not require modifications of training algorithms, are of great value due to a, typically, high number of engineering tweaks used during the model development stage. For example, (split) conformal prediction modifies an underlying point prediction model, which outputs the top-ranked label only, into a set-valued predictor that instead outputs a set of labels. Under the i.i.d. (or more generally, exchangeability) assumption, the resulting sets are provably valid (in terms of coverage) with guarantees being marginal over calibration and test data.   Violation of the i.i.d. assumption invalidates the inference, and adaptations to some structured distribution shifts have been proposed recently in the literature. The current work focuses on a setting where adversarial examples might be present at the test stage. Vanilla conformal is based on considering a collection of candidate prediction sets, parameterized by a single parameter, which is tuned/calibrated using a held-out set for performing set-valued predictions on test data. The authors propose correction of vanilla conformal, which essentially boils down to inflating the threshold in a way that guarantees robustness against adversarial examples with bounded $\ell_2$-norm (using randomized smoothing), where inflation results in outputting larger the prediction sets.