Disrupting Deep Uncertainty Estimation Without Harming Accuracy
Authors: Ido Galil
Ran El-Yaniv
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty estimation
adversarial attacks
deep neural networks
selective prediction
attacking uncertainty estimation

Abstract: Deep neural networks (DNNs) have proven to be powerful predictors and are widely used for various tasks. Credible uncertainty estimation of their predictions, however, is crucial for their deployment in many risk-sensitive applications. In this paper we present a novel and simple attack, which unlike adversarial attacks, does not cause incorrect predictions but instead cripples the network's capacity for uncertainty estimation. The result is that after the attack, the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. We present two versions of the attack. The first scenario focuses on a black-box regime (where the attacker has no knowledge of the target network) and the second scenario attacks a white-box setting. The proposed attack is only required to be of minuscule magnitude for its perturbations to cause severe uncertainty estimation damage, with larger magnitudes resulting in completely unusable uncertainty estimations.
We demonstrate successful attacks on three of the most popular uncertainty estimation methods: the vanilla softmax score, Deep Ensembles and MC-Dropout. Additionally, we show an attack on SelectiveNet, the selective classification architecture. We test the proposed attack on several contemporary architectures such as MobileNetV2 and EfficientNetB0, all trained to classify ImageNet.

Disrupting Deep Uncertainty Estimation Without Harming Accuracy
Ido Galil
Ran El-Yaniv Technion, Deci.AI rani technion, ac i1
Technion
idogalil igggmaii........ com
Abstract
Deep neural networks (DNNs) have proven to be powerful predictors and widely used for various tasks. Credible uncertainty estimation their predictions, however, is crucial for their deployment in many risk-sensitive applications. In this paper we present a novel and simple attack, which unlike adversarial attacks, does not cause incorrect predictions but instead cripples the network's capacity for uncertainty estimation. The result is that after the attack, the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. We present two versions of the attack. The first scenario focuses on black-box regime (where the attacker has no knowledge of the target network) and the second scenario attacks white-box setting. The proposed attack is only required to be of minuscule magnitude for its perturbations to cause severe uncertainty estimation damage, with larger magnitudes resulting in completely unusable uncertainty estimations. We demonstrate successful attacks on three the most popular uncertainty estimation methods: the vanilla softmax score, Deep Ensembles and MC-Dropout. Additionally, we show an attack on SelectiveNet, the selective classification architecture. We test the proposed attack on several contemporary architectures such as MobileNetV2 and EfficientNetBO, all trained to classify ImageNet.
Introduction
Deep neural networks (DNNs) show great and improving performance in wide variety of application domains including computer vision and natural language processing. Successful deployment of these models, however, is critically dependent providing effective uncertainty estimation for their predictions, or employing some kind of selective prediction
g
In the context of classification, practical and well-known uncertainty estimation techniques include: (I) the classification prediction's softmax score [3] 4], which quantifies the embedding margin between an instance to the decision boundary; (2) MC-Dropout [7]. which is argued to proxy
Bayesian networks [2]] 20 23]; (3) Deep Ensembles I8], which have shown state-of-the- results in various estimation settings; and (4) SelectiveNer [9] which learns to estimate uncertainty in way that produces optimal risk for a specific desired coverage.
In this paper we show that all these well koll- uncertainty estimation techniques are vulnerable to new and different type attack that can completely destroy their usefulness and that works in both black-box (where the attacker can only query the attacked model for predicted labels and has no knowledge of the model itself) and white-hox (the attacker has complete knowledge about the attacked model) settings. While standard adversarial attacks target model accuracy, the proposed attack is designed to keep accuracy performance intact and refrains from changing the original classification predictions made by the attacked model. We call our method ACE: Attack on Confidence Estimation. To demonstrate the relevance of our findings, we test ACE on modern architectures, such
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

as Mob 25 EfficientNet-BO 28] as well as standard baselines such as ResNet50 [3]. 1enseNet161 14 and VGGI6 [26]. Our attacks are focused on the task of ImageNet classification
S Figure T conceptually illustrates how ACE works. Consider a classifier for cats vs. dogs that
Cats
Decision Boundary
Mislabeled Instance
Dogs
Figure 1: The intuition driving ACE on a classifier of cats and dogs, when the uncertainty measure used is softmax. Correct predictions are moved closer to the decision boundary without crossing it, thus increasing their uncertainty (purple arrows), and incorrect predictions are moved further away from the decision boundary to decrease their uncertainty (green arrow).
uses its prediction's 's softmax score as its uncertainty estimation measurement. An end user asks the model to classify several images, and output only the ones in which it has the most confidence. Since softmax quantifies the margin from an instance the decision boundary, we visualize it on a 2D plane where the instances' distance to the decision boundary reflect their softmax score. In the example shown Figure [] the classifier was mistaken about one image a dog, classifying it as a cat, but fortunately its confidence in this prediction is the lowest among its predictions. A malicious attacker targeting the images in which the model has the most confidence would want to increase the confidence in the mislabeled instance by pushing away from the decision boundary, and decrease the confidence in the correctly labeled instances by pushing them closer to the decision boundary.
Our attack, however, can be accomplished without harming the model's accuracy, and thus avoid raising suspicion about possible attack. Even if the mislabeled dog assigned more confidence than only a single correctly predicted image, ACE will undermine the model's uncertainty estimation performance, since based on its estimation measurements, it cannot offer its newly least-confident (and correct) instance without offering the now more-confident mislabeled dog. It will either have to exclude both from the predictions it returns, reducing its coverage and its usefulness to the user, or include both and return predictions with mistakes, increasing its risk.
ACE, unlike adversarial attacks, creates damage even with small magnitudes of perturbations, befitting even an attacker with very limited resources (see Section [5). This benefit is inherent in our goal, which is different from standard adversarial attacks: adversarial attacks seek move the attacked instance across the model's 's decision boundary, and thus require the magnitude of the perturbations causes to be sufficiently large to allow this. Our objective to simply move the correctly and incorrectly predicted instances, relative to one another within the boundaries, which requires a less powerful "nudge". Figure 2 exemplifies such an attack on EfficientNet, showing how the model would be more confident of binoculars being tank than correctly-labeled tank. Figure shows the Risk- Lakk-Coeraage (RC) curve (see Section 3 for an explanation of RC curves) for EfficientNet under our attack, and that for the strongest attack tested (in terms of perturbation magnitude, which is still very small amount), approximately 20% of the model's most confident predictions were wrong.
Uncertainty attacks are in general conceptually different than adversarial attacks. Consider scenario of a debt landing company using machine learning to decide whether to approve or disapprove loans to its clients. An interesting example is: lendbuzz.com. Such a company operates by granting loans to highly probable non-defaulting applicants achieving the highest confidence according to their predictive model. The choice of rejection threshold must depend on uncertainty estimation (in accordance with the company's 's risk tolerance). A malicious attacker might attempt to influence this company to grant loans to the maximal amount of defaulting applicants to increase its odds to go

---

Correctly predicted: Tenk Confidence: 0,9392
Correctly predicted: Tank Confidence: 0.036
+ 0.005 x
Model confident of incorrect instances
of correct instances
+ 0.003
Incorrectly predicted:
Incorrectly predicted: Confidence: 0.908
Confidence: 0.1
Demonstration of attacking EfficientNet with softmax as the uncertainty estimation method. Left: Two images being predicted to be tanks. The top one is correctly labeled and the model has high confidence in label. The bottom image is of binoculars incorrectly labeled tank, with the model having low confidence in its prediction. Right: After adding perturbations to the original images, the model has lower confidence its (correct) tank label than the wrongly labeled binoculars.
Figure 2:
bankrupt. Aware of this risk, the company tries to defend itself in various ways. Two obvious options are: (1) Monitor their model's accuracy. drops significantly it might point to an adversarial attack. Since normal adversarial attacks harms accuracy significantly, it will alarm such monitors. ACE does not harm accuracy, and therefore will not. (2) Subscribe to AI firewall services like those given Since standard adversarial attacks generate instances that cross the decision boundary, they require relatively large epsilon that is easier to detect relative those required by ACE. Even if the standard adversarial attack bypassed detection, the crossing adversarial instance they generated may have very high uncertainty estimate due to its proximity to the decision boundary.
by professional parties such robustintelligenee.om
While a similar attack was already considered in the context of OOD detection with Dirichlet- based uncertainty models [15], the present paper the first offer an algorithm to attack only the uncertainty estimation of any model employing any of the most common uncertainty estimation techniques in both black-hox and white-box settings.
Related work
The common and well-known approach estimate uncertainty in DNNs for classification is using softmax at the last layer. Its values correspond to the distance from the decision boundary, with high values indicating high confidence in the prediction. The first ones suggest this approach using DNNs were
Lakshminarayanan al. [8 showed that ensemble consisting of several trained DNN models gives reliable uncertainty estimates. For classification tasks, the estimates are obtained by calculating softmax
the ensemble. This approach shown outperform the softmax response on CIFAR-10, CIFAR-100, SVHN and ImageNet
Motivated by Bayesian reasoning, Gal and Ghahramani proposed Monte-Carke dropout (MC- Dropout) as a way estimating uncertainty DNNs. At test time, for regression tasks this technique estimates uncertainty using variance statistics on several dropout-enabled forward passes. In classification, the mean softmax score of these passes calculated, and then predictive entropy score is calculated to be used as the final uncertainty estimate.
Geifman and EI- Yaniv introduced SelectiveNet, an architecture with a suitable loss function and optimization procedure that trains to minimize the risk for a specific required coverage. The SelectiveNet architecture can utilize any DNN its backbone, and constructs three prediction heads on top of it one of which is used solely for training, and the other two for prediction and rejection.

---

SelectiveNet incorporates soft "reject option" into its training process, allowing it to invest all its resources to achieve an optimal risk for its specific coverage objective. was shown to achieve state-of-the-art results on several datasets for deep selective classification across many different coverage rates.
The area under the risk-coverage curve (AURC) metric was defined in [10] for quantifying the quality of uncertainty scores. This metric is motivated from selective classification perspective: given a model, and using any uncertainty score for that model, an RC curve can be generated to indicate what the resulting risk (y-axis) would be for any percentage of coverage (x-axis). AURC is then defined to be the area under this curve. More details and an example are provided in Section B
Turning to adversarial attacks, one of the first adversarial examples was presented by Szegedy et al. 277 They showed how small, barely perceptible perturbations of the input to a CNN, crafted by using the box-constrained L-BFGS (Limited-Memor algorithm, could alter the DNN's classification. Goodfellow et al. [12] developed a more efficient way to craft adversarial examples, called the fast gradient sign method (FGSM), requiring only a single backward pass of the network. Formally, with being a fixed magnitude for the image perturbation, y being the true label and given loss function, the FGSM adversarial example is simply T- = x + e sign( V (f v)).
Liu al. [19] demonstrated that crafting transferable adversarial examples (examples fooling multiple models, possibly of different architectures) for black-box -ox attacks is especially challenging when targeting specific labels. The authors showed that using an ensemble results in highly transferable adversarial examples. The knowledge required for this method to create non-targeted attacks is only the ground-truth label and the attacked model's predicted label for targeted attacks.
3 Problem setup
Let X be the image space and y be the label space. Let P(X. V) be an unknown distribution over X' x y. A model f is a prediction function f : X - V, and its predicted label for image x is denoted by و The model's true risk w.r.t. P is R(f(P) = E ( (x) y)]. where y x y + IR+ is a given loss function, for example, 0/1 loss for classification. Given a labeled training set Sm (Ir; y1) m C (X's ))), sampled i.i.d. from P)X,)),, the empirical risk of the == ((f(xz).....
model f f() A L En
m
For a given model f. following [10. we define a confidence score function <(x, fl.). where x E X, and y is the model's prediction for The function should quantify confidence in the prediction of ÿ for the input æ, based on signals from the model f. This function should induce a partial order over points in X, and is not required to distinguish between points with the same score. For example, for any classification model f with softmax at its last layer, the vanilla confidence score is its softmax response values: st(x, flf) - f(x)g- Note that we are not concerned with the values of the confidence function; moreover, they do not have to possess any probabilistic interpretation. They can be any value in I.. An optimal x for a given classification model f and 0/1 loss should hold for every two instances with their ground truth, (II, v1) P(X, y) and (x2, y2) )))
x(21, وy (4i) I[s) S (12))1)()22)1)) - Prplür( # y2l S Prp (5(())) y
so that x truly is monotone with respect to the loss.
In practice DNN end users often calibrate threshold over the k used (whether it is a softmax one or an MC-dropout one, etc.) to get the desired coverage over the inference inputs exceeding this threshold, and give special treatment to the rest (such as consulting an expert or abstaining entirely from giving a prediction). A natural way to evaluate the performance of k in such settings is from selective model perspective. A selective model f [ uses selection function g : X' - {0. 1} to serve as a binary selector for S, enabling it to abstain from giving predictions for certain inputs. g can be defined by threshold 0 on the values of a K function such that (x f)- = 1 [[[( Ūy (x) >
0]. The performance of a selective model is measured using coverage and risk, where coverage, defined as o(f.g) the non-rejected instances in X. The selective risk of the selective model (f.g) is defined as
Ep [x]]( is the probability mass of
R(J.9) A Er((((()))))(t)) These quantities can be evaluated empirically over finite labeled
&(..)
set Sm. with the empirical coverage defined as e(f.g) 1 Em g(zz), and the empirical

---

selective risk defined as f(f.g) ()) & = 2 ((()))))))))))))
The risk-coverage curve (RC curve)
((.
ResNet18
AURC
6.53
002
0.30
0.0
022
0.1 1.0
Coverage
Figure 3: The RC curve ResNet18 trained on CIFAR- 10, measured on the test set. The risk is calculated using 0/1 loss (meaning the model has about 95% accuracy for 1.0 coverage); the k used was softmax-response. The value of the risk at each point of coverage corresponds to the selective risk of the model when rejecting inputs that are not covered at that coverage slice. e.g.. the selective risk for coverage 0.8 is about 0.5%, meaning that an end user setting matching threshold would enjoy model accuracy of 99.5% on the 80% of images the model would not reject.
curve showing the selective risk as a function of coverage, measured on some chosen test set. [igure]sshoe an example of such a curve. The RC curve could be used to evaluate confidence score function к over all its possible thresholds (such that the threshold is determined to produce the matching coverage in the graph). A useful scalar to quantify k's performance is to consider all possible coverages, by calculating the area under its RC curve (AURC) [1I which simply the mean selective risk.
4
How to attack uncertainty estimation
Attacking the uncertainty estimation "capacity" of a model is essentially sabotaging the partial order induced by its confidence score к, for example, by forcing к to assign low confidence scores to correct predictions, and high confidence scores incorrect predictions (an more refined attack, discussed Section is to completely reverse the partial order, which could be useful when the loss used is not 0/1). An adversarial example that specifically targets uncertainty estimation by finding minimal perturbation for an input æ, such that Ø = x + c would cause ii to produce bad (in terms of selective risk) partial order its inputs without changing the model's accuracy. The most harmful attack on x for a model f should hold for every two adversarial instances with their ground truth:
be crafted
(( S K( (F(2))) = (ri(ūr(6)) Ayl S (r + v2].
Figure shows such hypothetical worse-case RC curve scenario, in which all of its incorrectly predicted instances are assigned more confidence than its correctly predicted ones. An end user setting threshold to get coverage of 0.22 over inputs would suffer 100% error rate. An RC curve of our proposed attack showing similar actual damage can be observed Figure
The restriction of not changing the model's 's accuracy allows an attacker only harm the uncertainty estimation without attracting suspicion by causing a higher error rate for the model. This constraint underscores the fact that it is not necessary to change the model's predictions in order harm its capacity to estimate uncertainty. This restriction could be lifted based on the goal of the attack (if, for example, the attacker does not mind that the attacked model's accuracy is being reduced).
This type of attack also enjoys two benefits unique it and not shared with common adversarial attacks: Firstly, the perturbations needed for this type of attack are inherently smaller than those needed for most adversarial attacks. Perturbations with too large magnitudes could cause the crafted restriction).
example to cross the decision boundary and thus harm the model's 's accuracy (violating Secondly, its success not binary: The goal of most adversarial attacks to push an input over the decision boundary. If the perturbation magnitude insufficient to do that, the attack has failed. In
5

---

EfficientNet80
AURC $58.09
0.6
AURC 69
Coverage
Figure 4: The blue curve shows the hypothetical worse possible RC curve of EfficientNetBO trained on ImageNet, compared to its actual empirically measured RC curve (red curve).
our approach, the attack is partially successful even it causes a single incorrect prediction to get a higher confidence score than correct one, with the magnitude of the perturbations only contributing to the potency of the attack (creating a worse partial order and creating a larger confidence gap, numerically, between the correct and incorrect samples).
Algorithm 1 Attack on Confidence Estimation
function ACE(f, K, v, y. e, Edecay: max_ maar_tterations)
2:
< sign( V =i 0f (x)
b k. is calculated with f on the label predicted by f
3:
for < mar itterations do
4:
0y (x) = then
D fi correct, decrease confidence for x b f incorrect, increase confidence for x
5:
f x 6-
6:
else
7:
I x + 1
8:
if 05 (x( then return &c
9:
label is unchanged so accuracy will be unharmed
10:
else
11:
f- Edecng
D insufficient max Algorithm'ss 's arguments: fis the attacked model, f is the proxy for function, x is the input, y is the true label, is the initial magnitude of the perturbation, Edecay is the rate at which decreases when the attack changes the input's prediction and max iiterations the number of decrease steps.
return x
max_iterations jittrrrttrtonssss & c too big
the confidence scoring
Algorithm I summarizes the general template for our proposed attack, dubbed 'ACE' (Attack on Confidence Estimation). In black-box settings, the proxy for the attacked model f an ensemble of models not consisting of / itself (with ūs ()(x) requiring query to f to obtain its label for the input r)), and white-bon settings, f f. The derivative of x w.r.t. x, the case of the softmax score, is the derivative of the y prediction component of f w.r.t. to the input æ, where y = (x). We show in our white-box experiments that attacking the softmax score specifically is sometimes preferable, even the k used is not the softmax score, which is why we target the softmax score in black- box settings. We speculate that this is due to the fact that softmax allows a direct "handle" on the prediction margin in cases where does not (for example, in Section 5.4 we show that targeting the selector head of SelectiveNet that is used for rejection is less effective than targeting its softmax score).
While the scope of this paper is classification, future work could modify ACE for regression tasks, in which the uncertainty is often measured by the variance of several outputs made by the model (e.g.. variance outputs provided by an ensemble of DNNs). A simple conversion of this algorithm could, for example, define any instance with a loss above some threshold (such as the median loss on some validation set) as an "incorrect prediction" and loss values below the threshold as "correct predictions". The k being attacked would be the variance of the outputs, causing increase in

---

Confidence for incorrect predictions Confidence for correct predictions
Confidence incorrect predictions after ACE Confidence for correct predictions after ACE
15
2.5
10
1.5
0.4
Figure 5: Leftt: histograms of an EfficientNet's confidence scores (with &. being softmax-response for its correct predictions and incorrect predictions. Right: histograms for the confidence scores after using ACE (c- = 0.005, white-box) to attack EfficientNet's uncertainty estimation performance.
variance for the inputs for which the model had a low loss, and a decrease in variance for inputs for which the model had a high loss.
A weakness of ACE for conducting large scale attacks is that knowledge of the ground truths for the instances attacks is required, which might be difficult to obtain. Some attackers, however, could have good proxy of the ground truths, such as different model capable of making accurate predictions. Alternatively, the attacker might not care at all about the ground truths and would like only to increase the confidence for certain types of labels and decrease the confidence for other types regardless of their truthfulness (i.e., causing a loan company to have a very high confidence for the label approving the granting of loan and very low confidence for the label disapproving loans, thus increasing the chance for it to go bankrupt).
5 Experiments
We evaluate the potency of ACE using several metries: AURC (x (x103), Negative Log-likelihood (NLL) and Brier score [ commonly used for uncertainty estimation in DNNs, and defined in Appendix[A [ We test several architectures on the ImageNet (5] validation set (which contains 50,000 images), implemented and pretrained by PyTorch (4]. with the exception of EfficientNetBO, which was implemented and pretrained by (29]. We test ACE in black-box settings using three values for e: 0.0005, 0,005 and 0.05 (with RGB intensity values between 0 and 1). We follow [19] in using an ensemble to craft the adversarial examples, in our case consisting of ten ResNet50 models trained on IetgeNet. For "easier" white-box settings test ACE with even smaller values: 0.00005, 0.0005 and 0.005. To the best of our knowledge, some of our tested c values were never used by standard adversarial attacks. Due to ACE refraining from changing the originally predicted labels and decreasing iteratively if needed, the effective values selected by ACE are smaller and noted in the results (as "Effective e"). In all of our experiments, we the hyperparameters arbitrarily: Edenny = 0.5, max sau__lleraionns = 15. We did not try to optimize these, and we expect the algorithm to reach better results with smaller <decay if we also increase mar maar_ileration (since it would find bigger& values that could be used without changing the predictions).
5.1 Attacking softmax uncertainty
Table shows the results of using ACE for different values of under black-box settings. Note that for larger values of e, the effective e is about half the size, meaning that even fewer resources are sufficient for a very harmful attack. Under attack by ACE with 0.05, the AURC is 3.5 6.5 times worse, and both the NL. and Brier score have worsened by about 100%, meaning that the uncertainty estimation of the model is devastated while the accuracy is left unharmed just as ACE intended. The RC curves included in Appendix B.3 provide an important insight the metrics cannot: while on average, larger values harm uncertainty estimation more, sometimes for larger coverages, smaller is more harmful. For example, the selective risk for VGG16 above coverage 0.6 is higher for c 0.005 than it for 0.05 (indicated by the purple curve being under the blue curve). This artifact may either be inherent to such black-box attacks, or it may be due to many attacked instances losing their "adversarial trajectory" by taking overly large steps, and could be fixed with an iterative version of ACE further discussed in Section 6) 6 an RC curve for EfficientNerB00 with ACE under white-box settings (the legend presents the different values for Results for the even more potent white-box setting and its RC curves are included in Appendix B. In Appendix B.2 we
7

---

EfficientNet80
AURC 553.3
0.0005
0.8
0.0
Risk
AURC 2915
0.4
AURC 8
AURC 69.
0.0 0.2
0.4
06
0.8
10
Coverage
Figure 6: The RC curves of EfficientNet while under white-box attack by different values of The colored numbers next to each matching colored curve correspond to the area under the curve (AURC) x 1000. For attacks with 0.005, end user asking the model only for its 20% most confident predictions will get almost 100% wrong predictions.
evaluate ACE with adversarially trained model. These results suggest that standard adversarial training does not provide significant robustness against ACE.
Table 1: Results of ACE under black-box settings various architectures pretrained on ImageNet with softmax as the confidence score
Softmax score
Effective
AURC
NLL
Brier Score
Accuracy
69.9
0.963
0.336
76.01
ResNe150
0.0005
0,00046
141.9
1.308
0.474
76.01
0.005
0.003353
436.9
2.615
0.652
76.01
0.05
0.020622
456.6
2.494
0.656
76.01
69.1
0.958
0.322
77.67
EfficientNerBO
0.0005
0.000495
78.8
0,342
77.67
77.67
0.05
0.029023
300.9
1,775
0.585
77.67
89.7
1.147
0.386
71.85
Mobilenet V2
0.0005
0.00n493
104.3
1.217
0,416
71.85
236.9
0.05
0.025108
317.8
2.148
0.628
71.85
DenseNet161
0.0005
0.000493
81
1.022
0.358
77.15
0.05
0.026584
359.7
2.1
0.585
77.15
0.0005
0.000493
93.7
1.135
0.395
73.48
VGG16
0.005
0.004296
207.8
1.634
0.544
73.48
0.05
0.026088
298.2
1.985
0.602
73.48
5.2 Attacking deep ensemble uncertainty
Table 2: Results of ACE under black-box settings on ensemble of size consisting of ResNet50 models trained on ImageNet.
ResNet50 Ensemble
Effective
AURC
Brier score
Accuracy
0
63.8
0.888
0.317
77.61
Foreign Proxy
0.0005
0.00049
0.005
0.00432
182.1
1.356
0.196
77.61
0.05
0.02865
290.8
1.836
0.594
77.61
63.8
0.317
77.61
ResNet50 Proxy
B.000S
0.00049
82.7
0.966
77.61
0.005
0.00396
279.1
1,582
0.563
77.61
0.05
0.02342
381
0.633
77.61
Table 2 shows the results using ACE under black-box settings. We evaluate two different kinds of ensemble proxies to attack ensemble of ResNet50: (1) ensemble proxy of foreign architectures

---

(EfficientNet, MobileNetV2 and VGG), (2) an ensemble proxy consisting of ResNet50 models matching the victim models. While ACE harms the ensemble in both scenarios, it clearly stronger when using a matching ensemble. In the resulting RC curve, which is included in Appendix C.2 we can make the same observation we did when attacking softmax: for large coverages, a smaller é causes greater damage to the selective risk (using a ResNet50 proxy for example, the selective risk for any coverage above 0.45 is slightly higher for e = 0.005 than is for e = 0.05). For white-box settings, we test ACE using ensembles of different sizes consisting of ResNet50 models trained on ImageNet. Appendix shows the results of these experiments, from which we observe that for 0.005, the AURC degrades about eightfold. An interesting observation is that the bigger the ensemble, the more resilient is to ACE, and even the smallest ensemble is more resilient than a single ResNet50 model (in both black-box and white-box settings). Note that since the scope of this paper does not include explicit attempts to enhance these techniques' resiliency to our attack, we have not used adversarial training for the ensemble was suggested by [88] (which was found to improve uncertainty estimation). We did not want to give a perceived advantage to Deep Ensembles over other techniques in this respect, and as the results show, Deep Ensembles more resilient even without adversarial training.
5.3 Attacking Monte Carlo dropout
Table 3:
Results for MC-Dropout of ACE under black-box settings on models pretrained on ImageNet with predictive entropy over several dropout-enables passes producing the confidence score.
MC- MC-Dropout
Effective
AURC
NL.I
Brier Score
Accuracy
94.5
1.143
0.386
71.79
MobileNet V2 30 posses
0.0005
0.00049
104.4
1.207
0.413
71.79
0.005
0.00427
212.6
1.71
0.575
71.84
0.05
0.02514
275.2
2,145
0.634
71.79
95.1
1.148
0.387
71.58
MobileNet V2 10 passes
0.0005
0.00048
104.9
1.21
0.413
71.72
0.005
0.01427
213.2
1.713
0.573
71.7
0.05
0,02514
275.3
2.146
0.631
71.61
86.1
1.056
0.366
73,22
VGG16 30 posses
0.0005
0.00049
95
1.117
0.392
73.35
0.005
0.0043
188.9
1.573
0.541
73.34
0.05
0.02611
261.1
1.978
0.613
73.28
86.5
1.066
0.368
73.18
VGG16
0.0005
0.00048
95.9
1.123
0.392
73.13
10 passes
0.005
0,00429
189
1.578
0.537
73.08
0.05
0.02608
261.5
1,984
0.609
73.04
We test ACE MC- MC-Dropout using predictive entropy its confidence score. We first experiment with ACE under white-box settings and test two methods of attack: an indirect method, in which ACE targets the softmax score of a single dropout-disabled forward pass, and direct method, in which ACE targets the predictive entropy produced from N dropout-enabled forward passes. In both methods, the model's confidence in the prediction is caleulated by predictive entropy over N forward passes with dropout enabled. From the results presented in Appendix D.I. we can observe that while for small values the direct method is slightly more harmful, for bigger values the indirect softmax method is significantly more harmful and on par with results of attacking those models when their k softmax instead of being calculated by MC-Dropout. These results lead us to conclude that for most purposes, an indirect softmax attack could be more harmful than a direct attack on the predictive entropy used by MC-Dropout. Using this knowledge, we attack MC-Dropout in black-box settings identically to how we attacked softmax: by using ensemble as a proxy and attacking its softmax score, without using dropout at all. Observing Table for the results, we notice that for black-box settings, MC-Dropout seems somewhat more resilient compared to using softmax for these same models. The number of forward passes made by the models do not seem to affect ACE's effectiveness. The RC curves for the black-box settings are provided in Appendix D.2
5.4 Attacking SelectiveNet
Finally, we test ACE under white-box settings attacking SelectiveNet using two different architectures for its backbone, namely ResNet18 and VGG16, both trained on CIFAR-10 [66]. The confidence score is given by the selector head of SelectiveNet. which was trained to be used for specific

---

coverage. Since the model is trained to give predictions with an optimal selective risk for the coverage for which they were optimized. the selective risk that coverage is the most relevant metric by which to measure the performance of SelectiveNet model. The paper that introduced SelectiveNet [ emphasized the need to calibrate SelectiveNet's trained coverage to fit the desired empirical coverage. For fairness and to reach flawless calibration, we train the models to fit a trained coverage of 0.7, but evaluate the selective risk according to the model's s true empirical coverage on the test set (as observed without any attack). For example, the empirical coverage of SelectiveNet with ResNet18 as its backbone on the test set is 0.648, and the selective risk we show Table is the model's 's risk for that exact coverage. As additional metrics, we also provide the AURC, NLL and Brier score (even though they do not reflect the true goal of the network in this context). We consider a direct
Table 4: Results of ACE on SelectiveNet when used with ResNet18 8 and VGGI6 as backbone, and trained on CIFAR-10. The empirical coverages (perfectly calibrated) for ResNet18 and VGG16 are 0.648 and 0.778, respectively. The selector is always used as the confidence score. A direct ACE attack targets the selector, and an indirect attack targets the softmax score.
Effective e
Selective Risk
AURC
NLL
Brier Score
Accuracy
D
0
0,0012
5.04
0:204
0.083
94.83
ResNett8 Hackhome
0,00005
0.000005
0.0015
5.56
0.085
94.81
Diroct (sclector head)
0,0005
0.00049
0,0059
12,16
0.283
0.102
94.83
0.005
0.00577
0.0747
132.08
0.547
0.151
94.83
0.0012
5.04
0.204
0.083
94.83
ResNet18 Backbone
0,00005
0.00005
0.0015
5.5
0.215
0.087
94.83
Indirect (soltmax score)
0,0005
0.00049
0.00566
11.69
0.314
0.114
94.83
0.005
0,00385
0,0796
150.07
0.648
0.162
94,83
0.0067
7,46
0:259
0.105
93.54
VGG16 Backbons
0.00005
0.00005
0.0075
7,87
0,267
0.107
93.54
Direct (selector head)
0,0005
0.000.49
0.0163
12.57
0.34
0.128
93.54
0.005
0.004402
0.0772
116.1
0.703
0.175
0
0.0067
7.46
0.259
0.105
93.54
VGG16 Backhone
0,00005
0,00005
0.0075
7,86
0.268
0.108
93.54
Indirect (softruxx score)
0.0005
0.000449
0.0163
12.52
0:35
0.134
93.54
0.005
0.00404
0.0829
122.46
0.761
0.181
93.54
attack on the model's selector head and an indirect attack via the model's softmax score. In both cases the final confidence score is measured using the selector head. As seen in the results in Table 4 an indirect attack by softmax seems more harmful than direct one. For example, for SelectiveNet with a VGG16 backbone, the error over the not "rejected" instances is 0.67%, but it is 7.72% for directly (selector) crafted adversarial instances (c 0.005) and 8.29% for indirectly (softmax) crafted adversarial instances (c = 0.005). For context, we include in Appendix E ACE results on standard (not selective) architectures trained on CIFAR-10 with the confidence score being the softmax score.
6 Concluding Remarks
We have presented a novel attack on uncertainty estimation, capable of causing significant damage with even small values of e (compared to standard adversarial attacks) and without changing the model's 's original predictions. This attack severely harms some of the most popular techniques for estimating uncertainty. We can think of several ways in which the attack could be further improved: First is optimizing the perturbations in iterative process, similar to how BIM [17] or PGD (21) take FGSM steps iteratively. This may not only increase ACE's potency, but may also fix the artifacts of larger values of leading to a lower selective risk than smaller values (observed in Sections 5.1 (52). This might also have the added benefit of using more of the e resources that were allocated to the attack, thus increasing the effective Low ratios of Effective such as observed from our results mean unspent resources for the attack.
The results of this attack also teach us that some methods are more resilient than others. We hope that this work will encourage efforts to examine the resiliency of other and new uncertainty estimation methods, as well as creating defense mechanisms or detection techniques for them. Finally, in the case of none-0/1 loss functions, an attack on uncertainty estimation might need to completely reverse the partial order observed by the model, which would capture the gaps in uncertainty between two correct instances or between two incorrect instances. Such an attack could be used for regression tasks as well, more naturally than suggested simple conversion of ACE, discussed in Section
10

---

Acknowledgments
This research was partially supported by the Israel Science Foundation, grant No. 710/18.
References
[1] G. W. Brier. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1):1, Jan. 1950.
[2] C. K. Chow. An optimum character recognition system using decision functions. IRE Transac- tions on Electronic Computers, ECC6(4):247-254, 1957.
[3] L. P. Cordella, C. De Stefano, F. Tortorella, and M. Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks, 6(5): 140:1140-, 1995.
[4] C. De Stefano, C. Sansone, and M. Vento. To reject or not reject: that is the question-an answer in case of neural classifiers. IEEK Transactions on Systems, Man, and Cybernetics, Part (Applications and Reviews), 30(1):84 94, 2000.
[5] J. Deng, W. Dong, R. Socher, L..J. Li, i K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009,
[6] R. EI- Yaniv and Y. Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
[7] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncer- tainty in deep learning. 2016.
[8] Y. Geifman and R. Elniv. Selective classification for deep neural networks. Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 4885-4894, 2017.
[9] Y. Geifman and R. Yaniv. Selectivenet: A deep neural network with integrated reject option. CoRR, abs/1901.09 091192 2019.
[10] Y. Geifman, G. Uziel, and R. El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2018.
[11] Y. Geifman, G. Uziel, and R. Yaniv. Boosting uncertainty estimation for deep neural classifiers. CoRR, 0001180050000,,, 2018.
[12] J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[14] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. CoRR, Abs/1608.06999. 2016.
[15] A.-K. Kopetzki, B. Charpentier, Zügner, S. Giri, and Günnemann. Evaluating robustness of predictive uncertainty estimation: Are bascheet-based models reliable?, 2020.
[16] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[17] A. Kurakin, Goodfellow, and Bengio. Adversarial examples in the physical world, 2017.
[18] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. 2017.
[19] Y. Liu, X. Chen, C. Liu, and Song. Delving into transferable adversarial examples and black-box attacks. CoRR, abs//6611.02770 2016.

---

[20] D. J. C. MacKay. A practical bayesian framework for backpropagation networks. Neural Comput., ((3):4884477 May 1992.
[21] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
[22] N. Tishby, E. Levin, and S.A. Solla. Consistent inference of probabilities in layered networks: Predictions and generalizations. pages 403-409 vol. 2, 1989.
[23] R. M. Neal. Bayesian Learning for Neural Network. PhD thesis, Dept. of Computer Science, University of Toronto, 1994.
[24] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L.. Fang. Bai, and S. Chintala. Pytorch: imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[25] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR, abs/1801 001.8081., 2018.
[26] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
[27] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks, 2014.
[28] M. Tan and Q V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/ hass/1905.10 1946, 2019.
[29] R. Wightman.
Pytorch image models. monorch-image-modeee 2019.
https://ggthub..co//uutggaaa
12


Reviews:
[{"review_id": "q2ZBBrsefwd", "reviewer": "Reviewer_PoYC", "summary": "Contribute an attack where the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. Attacks contributed in the white-box and black-box regime, where the black-box regime here constitutes \"the attacker has no knowledge of the target network\". Demonstrate minimal perturbations successfully attack existing uncertainty estimation methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: Appears to be an original problem which is targeted, with related work fairly considered. \n\n*Quality*: Appears to be a complete work. \n\n*Clarity*:  Paper is fairly clear. I appreciate the inclusion of Algorithm 1, I don't see any significant details missing. \n\n*Significance*: My overriding concern is regarding the practicality of the given attack. I understand that \"suspicion arousal\" has been briefly mentioned, but I would implore the authors to provide an (extended) elaboration regarding this point, possibly with an intuitive, hypothetical example of where this attack would be uniquely relevant, and existing attacks would not be. In what scenario are we interested in attacking, or defending against attacks, where the accuracy isn't reduced but the uncertainty estimates are? From another perspective, if we do not currently have an adequate solution for defending against attacks which reduce accuracy, why should we be concerned with attacks which do not reduce accuracy but harm uncertainty assessment?\n\nDespite the above concern, I still see the paper as clearly a complete, original work. I would feel more strongly about the paper if the authors wrote up a detailed, convincing account of the practicality of their algorithm, beyond the brief mention of \"suspicion arousal\" currently included. \n\n**Post-Rebuttal**: The authors answered my concerns with clarity, so I am raising my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations, but not  potential negative societal impact. For what it's worth, I don't see a unique potential negative societal impact here. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3 hours"}, "extra_text": {}}, {"review_id": "G2voq23DVrB", "reviewer": "Reviewer_omcp", "summary": "This paper presents a novel uncertainty estimation attack technique. Unlike most existing adversarial attacks, which cause incorrect prediction results, this novel attack technique only cripples the network’s capacity for uncertainty estimation. The result is that, after the attack, the DNN is more confident of its incorrect predictions than its correct ones without having its accuracy reduced. Also, as this proposed attack can be accomplished without harming the model’s accuracy, and thus avoid raising suspicion about a possible attack.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The topic is quite interesting and well motivated. Though lots of attack and defend techniques have been explored, they mainly focus on altering the predictions of the models. This paper explores the attack techniques which only change the uncertainty estimation without causing incorrect prediction results. This topic is worth exploring as the uncertainly is also important for deep neural networks besides the predictions.\n\n2. The experimental part is impressing. The paper conducts attacking on different models, settings and uses several metrics to quantify the consequences of the proposed attacking technique.\n\nCons:\n\nSome of the claims in the paper are not supported. Such as:\n\n1. The algorithm could easily be modified for regression: while no regression experiments have been conducted in the paper to demonstrate the generability of the proposed attacking technique.\n\n2. The proposed attack technique severely harms some of the most popular techniques for estimating uncertainty: no experiments have been conducted to support this claim.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4huOu3zGzY", "reviewer": "Reviewer_UycQ", "summary": "The paper shows that one can manipulate the uncertainty estimation of a neural network without harming its accuracy by perturbing the inputs. To this end, the paper proposes a greedy heuristic to decrease the perturbation size of FGSM until the adversarial example does not flip the original prediction. Experimental results show that the proposed method indeed can increase several uncertainty estimation metrics (i.e., AURC, NLL, and Brier Score) of several types of neural networks regarding the uncertainty, e.g., softmax-based or MC dropout, without reducing their accuracy. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper tackles a new problem setup of attacking the calibration of neural networks without changing the accuracy, and I agree that there might be a practical scenario   that this threat model is more applicable than only considering adversarial examples which can also reduce the accuracy. Nevertheless, I feel the technical contributions of the paper are not significant enough to meet the NeurIPS bar upon the literature of adversarial attacks, as one might argue that the proposed method is simply a rescaled version of FGSM. While the observation here may re-confirm the high brittleness of neural networks around its decision boundary, but further insight beyond this seems to require more non-trivial works as far as I can see: the current experimental results only demonstrate that the proposed attack works, which is not that surprising, but lack on further analysis on it. It would be also nice if the paper could provide more concrete cases when the current threat model is more important than the standard adversarial examples.\n\n- Table 1-4: the paper could also report the results when the attack is done by standard FGSM or PGD with the comparable size of $\\epsilon$ or with allowing a larger value, to show that the standard method cannot achieve what ACE (the proposed method) can achieve.\n- I also suggest the paper to discuss [Ovadia et al., 2019] in the text, as it also address the limited ability of neural networks on uncertainty estimation under distribution shifts.\n- Is the proposed attack method still applicable to adversarially robust networks?\n- It would be nice if the paper could provide a more clear comparison of ACE with the standard adversarial attack: for example, how the attack objective of prediction confidence different from the standard cross-entropy? What if one applies ACE with the cross-entropy loss?\n- What would happens if one performs adversarial training using the samples found by ACE? Will the resulting network gets better robustness/calibration/generalization?\n\n[Ovadia et al., 2019] Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift, NeurIPS 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I could not find some clear statements that discuss their limitations or negative societal impacts in the text, so I suggest the authors to add the respective paragraphs in the revision.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple heuristic for manipulating the uncertainty estimation of a neural network without harming its accuracy by perturbing the inputs. Two reviewers (omcp and UycQ) suggest reject, while one reviewer (PoYC) does acceptance. In the discussion period, the authors provide additional experimental results and the positive reviewer (PoYC) further raised the score. Although Reviewer UycQ also appreciated that point, but still concerned that the paper needs more convincing arguments (or detailed experimental supports) on it, and Reviewer omcp also supported the concern. AC (and all reviewers) also agrees that the contribution of this work would be better conveyed if more extensive evaluations with simple modification of various existing solutions are provided. For example, one may consider DeepFool [Moosavi-Dezfooli et al., 2016] as an alternative baseline of ACE, as it is for estimating the margin of a given clean sample so that it can be also used to craft a perturbation that does not flip the classifier prediction. Although AC thinks that the paper is still an original work, AC tends to recommend reject given the current status.
Decision: Accept (Poster)