Disrupting Deep Uncertainty Estimation Without Harming Accuracy

1 Introduction . Deep neural networks ( DNNs ) show great and improving performance in a wide variety of application domains including computer vision and natural language processing . Successful deployment of these models , however , is critically dependent on providing effective uncertainty estimation for their predictions , or employing some kind of selective prediction [ 8 ] . In the context of classification , practical and well-known uncertainty estimation techniques include : ( 1 ) the classification prediction ’ s softmax score [ 3 , 4 ] , which quantifies the embedding margin between an instance to the decision boundary ; ( 2 ) MC-Dropout [ 7 ] , which is argued to proxy Bayesian networks [ 22 , 20 , 23 ] ; ( 3 ) Deep Ensembles [ 18 ] , which have shown state-of-the-art results in various estimation settings ; and ( 4 ) SelectiveNet [ 9 ] , which learns to estimate uncertainty in a way that produces optimal risk for a specific desired coverage . In this paper we show that all these well-known uncertainty estimation techniques are vulnerable to a new and different type of attack that can completely destroy their usefulness and that works in both black-box ( where the attacker can only query the attacked model for predicted labels and has no knowledge of the model itself ) and white-box ( the attacker has complete knowledge about the attacked model ) settings . While standard adversarial attacks target model accuracy , the proposed attack is designed to keep accuracy performance intact and refrains from changing the original classification predictions made by the attacked model . We call our method ACE : Attack on Confidence Estimation . To demonstrate the relevance of our findings , we test ACE on modern architectures , such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as MobileNetV2 [ 25 ] , EfficientNet-B0 [ 28 ] , as well as on standard baselines such as ResNet50 [ 13 ] , DenseNet161 [ 14 ] and VGG16 [ 26 ] . Our attacks are focused on the task of ImageNet classification [ 5 ] . Figure 1 conceptually illustrates how ACE works . Consider a classifier for cats vs. dogs that uses its prediction ’ s softmax score as its uncertainty estimation measurement . An end user asks the model to classify several images , and output only the ones in which it has the most confidence . Since softmax quantifies the margin from an instance to the decision boundary , we visualize it on a 2D plane where the instances ’ distance to the decision boundary reflect their softmax score . In the example shown in Figure 1 , the classifier was mistaken about one image of a dog , classifying it as a cat , but fortunately its confidence in this prediction is the lowest among its predictions . A malicious attacker targeting the images in which the model has the most confidence would want to increase the confidence in the mislabeled instance by pushing it away from the decision boundary , and decrease the confidence in the correctly labeled instances by pushing them closer to the decision boundary . Our attack , however , can be accomplished without harming the model ’ s accuracy , and thus avoid raising suspicion about a possible attack . Even if the mislabeled dog is assigned more confidence than only a single correctly predicted image , ACE will undermine the model ’ s uncertainty estimation performance , since based on its estimation measurements , it can not offer its newly least-confident ( and correct ) instance without offering the now more-confident mislabeled dog . It will either have to exclude both from the predictions it returns , reducing its coverage and its usefulness to the user , or include both and return predictions with mistakes , increasing its risk . ACE , unlike adversarial attacks , creates damage even with small magnitudes of perturbations , befitting even an attacker with very limited resources ( see Section 5 ) . This benefit is inherent in our goal , which is different from standard adversarial attacks : adversarial attacks seek to move the attacked instance across the model ’ s decision boundary , and thus require the magnitude of the perturbations it causes to be sufficiently large to allow this . Our objective is to simply move the correctly and incorrectly predicted instances , relative to one another within the boundaries , which requires a less powerful “ nudge ” . Figure 2 exemplifies such an attack on EfficientNet , showing how the model would be more confident of binoculars being a tank than a correctly-labeled tank . Figure 6 shows the Risk-Coverage ( RC ) curve ( see Section 3 for an explanation of RC curves ) for EfficientNet under our attack , and that for the strongest attack tested ( in terms of perturbation magnitude , which is still a very small amount ) , approximately 20 % of the model ’ s most confident predictions were wrong . Uncertainty attacks are in general conceptually different than adversarial attacks . Consider a scenario of a debt landing company using machine learning to decide whether to approve or disapprove loans to its clients . An interesting example is : lendbuzz.com . Such a company operates by granting loans to highly probable non-defaulting applicants achieving the highest confidence according to their predictive model . The choice of rejection threshold must depend on uncertainty estimation ( in accordance with the company ’ s risk tolerance ) . A malicious attacker might attempt to influence this company to grant loans to the maximal amount of defaulting applicants to increase its odds to go bankrupt . Aware of this risk , the company tries to defend itself in various ways . Two obvious options are : ( 1 ) Monitor their model ’ s accuracy . if it drops significantly it might point to an adversarial attack . Since normal adversarial attacks harms accuracy significantly , it will alarm such monitors . ACE does not harm accuracy , and therefore will not . ( 2 ) Subscribe to AI firewall services like those given by professional parties such as robustintelligence.com . Since standard adversarial attacks generate instances that cross the decision boundary , they require a relatively large epsilon that is easier to detect relative to those required by ACE . Even if the standard adversarial attack bypassed detection , the crossing adversarial instance they generated may have a very high uncertainty estimate due to its proximity to the decision boundary . While a similar attack was already considered in the context of OOD detection with Dirichletbased uncertainty models [ 15 ] , the present paper is the first to offer an algorithm to attack only the uncertainty estimation of any model employing any of the most common uncertainty estimation techniques in both black-box and white-box settings . 2 Related work . The most common and well-known approach to estimate uncertainty in DNNs for classification is using softmax at the last layer . Its values correspond to the distance from the decision boundary , with high values indicating high confidence in the prediction . The first ones to suggest this approach using DNNs were [ 3 , 4 ] . Lakshminarayanan et al . [ 18 ] showed that an ensemble consisting of several trained DNN models gives reliable uncertainty estimates . For classification tasks , the estimates are obtained by calculating a mean softmax score over the ensemble . This approach was shown to outperform the softmax response on CIFAR-10 , CIFAR-100 , SVHN and ImageNet [ 11 ] . Motivated by Bayesian reasoning , Gal and Ghahramani [ 7 ] proposed Monte-Carlo dropout ( MCDropout ) as a way of estimating uncertainty in DNNs . At test time , for regression tasks this technique estimates uncertainty using variance statistics on several dropout-enabled forward passes . In classification , the mean softmax score of these passes is calculated , and then a predictive entropy score is calculated to be used as the final uncertainty estimate . Geifman and El-Yaniv [ 9 ] introduced SelectiveNet , an architecture with a suitable loss function and optimization procedure that trains to minimize the risk for a specific required coverage . The SelectiveNet architecture can utilize any DNN as its backbone , and constructs three prediction heads on top of it – one of which is used solely for training , and the other two for prediction and rejection . SelectiveNet incorporates a soft “ reject option ” into its training process , allowing it to invest all its resources to achieve an optimal risk for its specific coverage objective . It was shown to achieve state-of-the-art results on several datasets for deep selective classification across many different coverage rates . The area under the risk-coverage curve ( AURC ) metric was defined in [ 10 ] for quantifying the quality of uncertainty scores . This metric is motivated from a selective classification perspective : given a model , and using any uncertainty score for that model , an RC curve can be generated to indicate what the resulting risk ( y-axis ) would be for any percentage of coverage ( x-axis ) . AURC is then defined to be the area under this curve . More details and an example are provided in Section 3 . Turning to adversarial attacks , one of the first adversarial examples was presented by Szegedy et al . [ 27 ] . They showed how small , barely perceptible perturbations of the input to a CNN , crafted by using the box-constrained L-BFGS ( Limited-Memory Broyden-Fletcher-Goldfarb-Shanno ) algorithm , could alter the DNN ’ s classification . Goodfellow et al . [ 12 ] developed a more efficient way to craft adversarial examples , called the fast gradient sign method ( FGSM ) , requiring only a single backward pass of the network . Formally , with being a fixed magnitude for the image perturbation , y being the true label and a given loss function , the FGSM adversarial example is simply x̃ = x+ · sign ( ∇xloss ( f ( x ) , y ) ) . Liu et al . [ 19 ] demonstrated that crafting transferable adversarial examples ( examples fooling multiple models , possibly of different architectures ) for black-box attacks is especially challenging when targeting specific labels . The authors showed that using an ensemble results in highly transferable adversarial examples . The knowledge required for this method to create non-targeted attacks is only the ground-truth label and the attacked model ’ s predicted label for targeted attacks . 3 Problem setup . Let X be the image space and Y be the label space . Let P ( X , Y ) be an unknown distribution over X × Y . A model f is a prediction function f : X → Y , and its predicted label for an image x is denoted by ŷf ( x ) . The model ’ s true risk w.r.t . P is R ( f |P ) = EP ( X , Y ) [ ` ( f ( x ) , y ) ] , where ` : Y × Y → R+ is a given loss function , for example , 0/1 loss for classification . Given a labeled training set Sm = { ( xi , yi ) } mi=1 ⊆ ( X × Y ) , sampled i.i.d . from P ( X , Y ) , the empirical risk of the model f is r̂ ( f |Sm ) , 1m ∑m i=1 ` ( f ( xi ) , yi ) . For a given model f , following [ 10 ] , we define a confidence score function κ ( x , ŷ|f ) , where x ∈ X , and ŷ ∈ Y is the model ’ s prediction for x . The function κ should quantify confidence in the prediction of ŷ for the input x , based on signals from the model f . This function should induce a partial order over points in X , and is not required to distinguish between points with the same score . For example , for any classification model f with softmax at its last layer , the vanilla confidence score is its softmax response values : κ ( x , ŷ|f ) , f ( x ) ŷ . Note that we are not concerned with the values of the confidence function ; moreover , they do not have to possess any probabilistic interpretation . They can be any value in R. An optimal κ for a given classification model f and 0/1 loss should hold for every two instances with their ground truth , ( x1 , y1 ) ∼ P ( X , Y ) and ( x2 , y2 ) ∼ P ( X , Y ) : κ ( x1 , ŷf ( x1 ) |f ) ≤ κ ( x2 , ŷf ( x2 ) |f ) ⇐⇒ PrP [ ŷf ( x2 ) 6= y2 ] ≤ PrP [ ŷf ( x1 ) 6= y1 ] so that κ truly is monotone with respect to the loss . In practice , DNN end users often calibrate a threshold over the κ used ( whether it is a softmax one or an MC-dropout one , etc . ) to get the desired coverage over the inference inputs exceeding this threshold , and give special treatment to the rest ( such as consulting an expert or abstaining entirely from giving a prediction ) . A natural way to evaluate the performance of a κ in such settings is from a selective model perspective . A selective model f [ 6 , 2 ] uses a selection function g : X → { 0 , 1 } to serve as a binary selector for f , enabling it to abstain from giving predictions for certain inputs . g can be defined by a threshold θ on the values of a κ function such that gθ ( x|κ , f ) = 1 [ κ ( x , ŷf ( x ) |f ) > θ ] . The performance of a selective model is measured using coverage and risk , where coverage , defined as φ ( f , g ) = EP [ g ( x ) ] , is the probability mass of the non-rejected instances in X . The selective risk of the selective model ( f , g ) is defined as R ( f , g ) , EP [ ` ( f ( x ) , y ) g ( x ) ] φ ( f , g ) . These quantities can be evaluated empirically over a finite labeled set Sm , with the empirical coverage defined as φ̂ ( f , g|Sm ) = 1m ∑m i=1 g ( xi ) , and the empirical selective risk defined as r̂ ( f , g|Sm ) , 1 m ∑m i=1 ` ( f ( xi ) , yi ) g ( xi ) φ̂ ( f , g|Sm ) . The risk-coverage curve ( RC curve ) is a curve showing the selective risk as a function of coverage , measured on some chosen test set . Figure 3 shows an example of such a curve . The RC curve could be used to evaluate a confidence score function κ over all its possible thresholds ( such that the threshold is determined to produce the matching coverage in the graph ) . A useful scalar to quantify κ ’ s performance is to consider all possible coverages , by calculating the area under its RC curve ( AURC ) [ 11 ] , which is simply the mean selective risk .

Summary: The paper presents an interesting method to create adversarial examples that do not change the class label, but rather flips the confidence in that label.  In other words, the accuracy of the classification system is not changed.  Rather, the idea is that the attack will force a system with well calibrated confidence to become confidently wrong and unconfidently correct rather than the desired behavior of being unconfidently wrong and confidently correct. The overall approach is basically the same as a standard label adversarial attack (i.e., follow the gradient), but at a lower magnitude as not to change the label.  The other minor difference is to determine the direction of the gradient depending on the ground truth label of the test sample.  That is increase confidence if the classification label prediction is incorrect and reduce the confidence otherwise. The paper does include extensive experimentation over various neural network models including uncertainty-aware models such as ensembles, MC dropout and SelectiveNet.