Disrupting Deep Uncertainty Estimation Without Harming Accuracy

Abstract: Deep neural networks (DNNs) have proven to be powerful predictors and are widely used for various tasks. Credible uncertainty estimation of their predictions, however, is crucial for their deployment in many risk-sensitive applications. In this paper we present a novel and simple attack, which unlike adversarial attacks, does not cause incorrect predictions but instead cripples the network's capacity for uncertainty estimation. The result is that after the attack, the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. We present two versions of the attack. The first scenario focuses on a black-box regime (where the attacker has no knowledge of the target network) and the second scenario attacks a white-box setting. The proposed attack is only required to be of minuscule magnitude for its perturbations to cause severe uncertainty estimation damage, with larger magnitudes resulting in completely unusable uncertainty estimations.We demonstrate successful attacks on three of the most popular uncertainty estimation methods: the vanilla softmax score, Deep Ensembles and MC-Dropout. Additionally, we show an attack on SelectiveNet, the selective classification architecture. We test the proposed attack on several contemporary architectures such as MobileNetV2 and EfficientNetB0, all trained to classify ImageNet.


Reviews and author responses:

 I appreciate for the response and clarification made by authors.

After reading the authors' response and also other reviews, I would like to keep my original score.

 I appreciate for the detailed response made by authors, and the additional experiments on attacking an adversarially robust network - the results are indeed interesting, as it implies that the current adversarial training may not be enough to calibrate the confidences between a clean sample and its near decision boundary, although it seems a more precise conclusion requires an extensive additional experimental support. 

Overall, I would like to keep my score at this time, as I am still not fully convinced for my major concern of the lack of technical contribution even after the response. For example, in case of my question on Table 1-4 of comparing ACE with FGSM or PGD, the response did not address the case of PGD, which is of more concern to compare the technical effectiveness of ACE. The question on the ablation of using the cross-entropy loss for ACE is also addressed without empirical results, while I still suspect this ablation would just work similarly with ACE in practice. Apart from that, using the "target confidence" as an attack objective would not be a completely new variant of the standard cross-entropy based attack in the context of adversarial attack, which is I still consider the method as a rescaled version of PGD where the scaling factor is line-searched to not exceed the decision boundary. I agree with authors that, nevertheless, a more refined attack method would much strengthen the technical significance of the paper.

I also feel that there should be a more detailed discussion with [Ovadia et al., 2019] than the current response to fully claim the significance of empirical observations - once we agree that data shift can deteriorate uncertainty estimation [Ovadia et al., 2019], the observation that ACE-samples can deteriorate uncertainty estimation may not be that surprising for some readers, as ACE is itself an instance of data shift.

 We further investigated your question regarding adversarial training and we have more comprehensive insights. Our latest response to Reviewer PoYC include results comparing the (black-box) ACE attack of EfficientNetB0 that was adversarially trained with AdvProp to the ACE attack of EfficientNetB0 without adversarial training.
For your convenience, we attach the results below:

|                        | $\epsilon$ | Effective $\epsilon$ | AURC  |
|------------------------|------------|----------------------|-------|
| EfficientNetB0 AdvProp | 0          | 0.00000              | 73.7  |
|                        | 0.0005     | 0.00050              | 78.2  |
|                        | 0.005      | 0.00474              | 127.8 |
|                        | 0.05       | 0.03304              | 292.7 |

|                | $\epsilon$ | Effective $\epsilon$ | AURC  |
|----------------|------------|----------------------|-------|
| EfficientNetB0 | 0          | 0                    | 69.1  |
|                | 0.0005     | 0.00049487           | 78.8  |
|                | 0.005      | 0.004458629          | 185.1 |
|                | 0.05       | 0.02902293           | 300.9 |

These results for EfficientNetB0 (and for other models) indicate that, unlike our initial hypothesis, standard adversarial training does not increase resiliency to ACE. These results will be added to the paper.

 Thanks for your constructive feedback.
(1) You make a valid point. We have conducted an additional black-box attack with an ensemble proxy consisting of various architectures dissimilar to ResNet (the ensemble includes EfficientNet, MobileNetV2, VGG, and DenseNet). The results below show that ACE is harmful to uncertainty estimation in this context as well:

|                   | $\epsilon$ | Effective $\epsilon$ | AURC  |
|-------------------|------------|----------------------|-------|
| Ensemble | 0          | 0                    | 63.8  |
|                   | 0.0005     | 0.02919              | 74.2  |
|                   | 0.005      | 0.00434              | 177.8 |
|                   | 0.05       | 0.00049              | 290.2 |

(2) We have finished testing on several adversarially trained networks. On almost all of these, ACE performance seems identical to its performance on non-robust models, meaning the adversarial training has contributed nothing to the model's resiliency against ACE. We include below the results for a black-box ACE setting against an EfficientNetB0 trained with AdvProp (we've included ACE against a non-adversarially trained EfficientNetB0 as well for comparison):

|                        | $\epsilon$ | Effective $\epsilon$ | AURC  |
|------------------------|------------|----------------------|-------|
| EfficientNetB0 AdvProp | 0          | 0.00000              | 73.7  |
|                        | 0.0005     | 0.00050              | 78.2  |
|                        | 0.005      | 0.00474              | 127.8 |
|                        | 0.05       | 0.03304              | 292.7 |

|                | $\epsilon$ | Effective $\epsilon$ | AURC  |
|----------------|------------|----------------------|-------|
| EfficientNetB0 | 0          | 0                    | 69.1  |
|                | 0.0005     | 0.00049487           | 78.8  |
|                | 0.005      | 0.004458629          | 185.1 |
|                | 0.05       | 0.02902293           | 300.9 |

Notice that the AURC values for each epsilon are very similar to the AURC values obtained without adversarial training.
ACE performance on adversarially trained models will indeed strengthen the motivation greatly, and we will of course include the full results in the revised paper.

Contribute an attack where the DNN is more confident of its incorrect predictions than about its correct ones without having its accuracy reduced. Attacks contributed in the white-box and black-box regime, where the black-box regime here constitutes "the attacker has no knowledge of the target network". Demonstrate minimal perturbations successfully attack existing uncertainty estimation methods.   *Originality*: Appears to be an original problem which is targeted, with related work fairly considered. 

*Quality*: Appears to be a complete work. 

*Clarity*:  Paper is fairly clear. I appreciate the inclusion of Algorithm 1, I don't see any significant details missing. 

*Significance*: My overriding concern is regarding the practicality of the given attack. I understand that "suspicion arousal" has been briefly mentioned, but I would implore the authors to provide an (extended) elaboration regarding this point, possibly with an intuitive, hypothetical example of where this attack would be uniquely relevant, and existing attacks would not be. In what scenario are we interested in attacking, or defending against attacks, where the accuracy isn't reduced but the uncertainty estimates are? From another perspective, if we do not currently have an adequate solution for defending against attacks which reduce accuracy, why should we be concerned with attacks which do not reduce accuracy but harm uncertainty assessment?

Despite the above concern, I still see the paper as clearly a complete, original work. I would feel more strongly about the paper if the authors wrote up a detailed, convincing account of the practicality of their algorithm, beyond the brief mention of "suspicion arousal" currently included. 

**Post-Rebuttal**: The authors answered my concerns with clarity, so I am raising my score.  The authors adequately addressed the limitations, but not  potential negative societal impact. For what it's worth, I don't see a unique potential negative societal impact here. 

 Great! Glad to see the results correspond to expectation. I would suggest including these results in the paper to demonstrate that this attack really is one that existing defense strategies do not account for. I will raise my score. 

 I agree with the authors response regarding practicality, and would suggest the authors include said motivation in the revision. 

Continuing the thread of the motivating example, I believe there are still aspects which the authors have not considered. 

(1): The black-box evaluation in the current paper is weak. For example, in Table 2, results are presented where "we use one ensemble consisting of five ResNet50 models to attack another ensemble also consisting of five ResNet50 models, all trained separately on ImageNet." This implies the only difference between the ensemble used for the attack and the ensemble used for testing is random seed. When attacking a debt landing company, the difference between the classifier used for generating the attack, and the classifier used by the company would very rarely be solely random seed, unless the attacker had additional information, however that would constitute a violation of the black-box threat model. 

(2): The company could also defend itself with an existing defense technique, such as adversarial training, which the authors did not consider in the evaluation. While the attack does not target what adversarial training intends to defend itself against, evidence has not been provided in the main paper that the uncertainty attack works against defended models. Evidence that demonstrates that if the target attempts to defend themselves to standard adversarial attacks, they will remain vulnerable to the uncertainty attack, would be convincing. While it can be expected, defenses, such as adversarial training, are fundamentally different than the models used for evaluation in the submitted paper.

I do see preliminary results regarding (2), at least with regards to adversarial training, were provided in response to another reviewer. I would encourage the authors to include said results in revising their work, particularly as the debt company in the motivating example, if they felt they were being targeted, likely would employ defense techniques existing in the literature. 

 Thank you for your comment and detailed suggestions. 

*"It would be also nice if the paper could provide more concrete cases when the current threat model is more important than the standard adversarial examples"*

Let's consider a scenario of a debt landing company using Machine Learning to decide whether to approve or disapprove loans to its clients. An interesting example is:
https://lendbuzz.com/
Such a company operates by granting loans to highly probable non-defaulting applicants achieving the highest confidence according to their predictive model. The choice of rejection threshold must depend on uncertainty estimation (in accordance with the company's risk tolerance).
A malicious attacker might attempt to influence this company to grant loans to the maximal amount of defaulting applicants to increase its odds to go bankrupt. Aware of this risk, the company tries to defend itself in various ways. Two obvious options are:
1. Monitor their model's accuracy. if it drops significantly it might point to an adversarial attack. Since normal adversarial attacks harms accuracy significantly, it will alarm such monitors.
2. Subscribe to AI firewall services like those given by professional parties such as https://www.robustintelligence.com/ .
Since standard adversarial attacks generate instances that cross the decision boundary, they require a relatively large epsilon that is easier to detect relative to those required by ACE.

Even if the standard adversarial attack bypassed detection, the crossing adversarial instance they generated has a very high uncertainty estimate due to its proximity to the decision boundary. 

ACE, on the other hand, is more harmful in this context because (1) it typically uses a considerably smaller epsilon and is harder to detect; (2) accuracy monitoring is no longer relevant (since ACE does not change accuracy); and (3) ACE will give higher confidence to riskier applicants.

*"I feel the technical contributions of the paper are not significant enough to meet the NeurIPS bar upon the literature of adversarial attacks, as one might argue that the proposed method is simply a rescaled version of FGSM."*

While we agree that our proposed technique is simple, we think it is an important first step introducing a complex setting of attacking uncertainty estimation. For example, we briefly suggested (in Sections 4 and 6) a more refined attack for future work that reverses the partial order induced by the attacked model. Such an attack would not only have to deal with independent instances, but will also have to consider the entire ordering of all instances.

*"Table 1-4: the paper could also report the results when the attack is done by standard FGSM or PGD with the comparable size of  or with allowing a larger value, to show that the standard method cannot achieve what ACE (the proposed method) can achieve."*

This is a good suggestion, and we will add such a comparison to the revised paper. We have conducted a quick preliminary experiment with FGSM. The results are significantly inferior to ACE. 

*"I also suggest the paper to discuss [Ovadia et al., 2019] in the text, as it also address the limited ability of neural networks on uncertainty estimation under distribution shifts."*

Thanks for the reference, which we overlooked. This paper is indeed very related to ours. Our observation that better resiliency is achieved with deep ensembles may connect to that paper's results showing ensembles to be relatively well performing under a more general (non-adversarial) distribution shift. This will be added to the paper.

*"Is the proposed attack method still applicable to adversarially robust networks?"*

We interpret "adversarially robust networks" as models that were trained with adversarial examples.
This is a good question. Following your question, we experimented this direction and our preliminary results so far suggest that ACE performance is just as good for small and medium values of epsilon (from the range we tested on in the paper). Interestingly, however, ACE performance deteriorates significantly when epsilon exceeds a certain threshold. This sounds logical since conventional adversarial training only trains on epsilons large enough to cross the decision boundary, probably similar to those epsilons exceeding that threshold.

*"It would be nice if the paper could provide a more clear comparison of ACE with the standard adversarial attack: for example, how the attack objective of prediction confidence different from the standard cross-entropy? What if one applies ACE with the cross-entropy loss?"*

It's a good question. Application of ACE with cross-entropy cannot in principle guarantee a confidence increase of misclassified instances. 
Let's take an example of a 3-class classifier: for cats, dogs and fish, and suppose an instance of a cat was misclassified by the classifier as a dog in such a way that its softmax vector is [dog: 0.4, cat: 0.3, fish: 0.3]. The ACE objective is to make the classifier confident as possible on the wrong 'dog' prediction. Consider the following cases:
Case 1: ACE is applied with cross-entropy (CE). In this case, CE forces the algorithm to minimize the cat probability as much as possible.
Therefore, the following vector is a *possible* output after such an attack [dog: 0.37, cat: 0.27 , fish: 0.36]. If we are estimating the confidence with the softmax value for the max label, our confidence has decreased from 0.4 to 0.37, counter to our objective (it will not necessarily decrease, but neither will it necessarily increase).
Case 2: standard ACE. An attack objective targeting the wrongly predicted dog label will always make the best use of its resources to increase its confidence above its original confidence of 0.4.

*"What would happens if one performs adversarial training using the samples found by ACE? Will the resulting network gets better robustness/calibration/generalization?"*

This is a very interesting question. we defer it to future work.

 *1. "The algorithm could easily be modified for regression: while no regression experiments have been conducted in the paper to demonstrate the generability of the proposed attacking technique."*:

You are correct. We have conducted experiments with the basic idea we presented in Section 4, which provides preliminary support to this claim, but have not reported them. We will either add these results or remove the claim. 

*2. "The proposed attack technique severely harms some of the most popular techniques for estimating uncertainty: no experiments have been conducted to support this claim."*:

Sorry for the misunderstanding; to the best of our knowledge, the most popular uncertainty estimation technique is softmax response. The most powerful one to the best of our knowledge is using Deep Ensembles. Another popular choice for uncertainty estimation is MC-Dropout.
All these methods and also SelectiveNet are considered in our paper and the supporting experiments showing the severe consequences of ACE are presented in Section 5.

 Thanks for your comment.
We should indeed elaborate on the practicality and advantage of our proposed attack.
Let's consider a scenario of a debt landing company using Machine Learning to decide whether to approve or disapprove loans to its clients. An interesting example is:
https://lendbuzz.com/
Such a company operates by granting loans to highly probable non-defaulting applicants achieving the highest confidence according to their predictive model. The choice of rejection threshold must depend on uncertainty estimation (in accordance with the company's risk tolerance).
A malicious attacker might attempt to influence this company to grant loans to the maximal amount of defaulting applicants to increase its odds to go bankrupt. Aware of this risk, the company tries to defend itself in various ways. Two obvious options are:
1. Monitor their model's accuracy. if it drops significantly it might point to an adversarial attack. Since normal adversarial attacks harms accuracy significantly, it will alarm such monitors.
2. Subscribe to AI firewall services like those given by professional parties such as https://www.robustintelligence.com/ .
Since standard adversarial attacks generate instances that cross the decision boundary, they require a relatively large epsilon that is easier to detect relative to those required by ACE.

Even if the standard adversarial attack bypassed detection, the crossing adversarial instance they generated has a very high uncertainty estimate due to its proximity to the decision boundary. 

ACE, on the other hand, is more harmful in this context because (1) it typically uses a considerably smaller epsilon and is harder to detect; (2) accuracy monitoring is no longer relevant (since ACE does not change accuracy); and (3) ACE will give higher confidence to riskier applicants.

This paper presents a novel uncertainty estimation attack technique. Unlike most existing adversarial attacks, which cause incorrect prediction results, this novel attack technique only cripples the network’s capacity for uncertainty estimation. The result is that, after the attack, the DNN is more confident of its incorrect predictions than its correct ones without having its accuracy reduced. Also, as this proposed attack can be accomplished without harming the model’s accuracy, and thus avoid raising suspicion about a possible attack.  Pros:

1. The topic is quite interesting and well motivated. Though lots of attack and defend techniques have been explored, they mainly focus on altering the predictions of the models. This paper explores the attack techniques which only change the uncertainty estimation without causing incorrect prediction results. This topic is worth exploring as the uncertainly is also important for deep neural networks besides the predictions.

2. The experimental part is impressing. The paper conducts attacking on different models, settings and uses several metrics to quantify the consequences of the proposed attacking technique.

Cons:

Some of the claims in the paper are not supported. Such as:

1. The algorithm could easily be modified for regression: while no regression experiments have been conducted in the paper to demonstrate the generability of the proposed attacking technique.

2. The proposed attack technique severely harms some of the most popular techniques for estimating uncertainty: no experiments have been conducted to support this claim.
 Yes

The paper shows that one can manipulate the uncertainty estimation of a neural network without harming its accuracy by perturbing the inputs. To this end, the paper proposes a greedy heuristic to decrease the perturbation size of FGSM until the adversarial example does not flip the original prediction. Experimental results show that the proposed method indeed can increase several uncertainty estimation metrics (i.e., AURC, NLL, and Brier Score) of several types of neural networks regarding the uncertainty, e.g., softmax-based or MC dropout, without reducing their accuracy.   Overall, the paper tackles a new problem setup of attacking the calibration of neural networks without changing the accuracy, and I agree that there might be a practical scenario   that this threat model is more applicable than only considering adversarial examples which can also reduce the accuracy. Nevertheless, I feel the technical contributions of the paper are not significant enough to meet the NeurIPS bar upon the literature of adversarial attacks, as one might argue that the proposed method is simply a rescaled version of FGSM. While the observation here may re-confirm the high brittleness of neural networks around its decision boundary, but further insight beyond this seems to require more non-trivial works as far as I can see: the current experimental results only demonstrate that the proposed attack works, which is not that surprising, but lack on further analysis on it. It would be also nice if the paper could provide more concrete cases when the current threat model is more important than the standard adversarial examples.

- Table 1-4: the paper could also report the results when the attack is done by standard FGSM or PGD with the comparable size of $\epsilon$ or with allowing a larger value, to show that the standard method cannot achieve what ACE (the proposed method) can achieve.
- I also suggest the paper to discuss [Ovadia et al., 2019] in the text, as it also address the limited ability of neural networks on uncertainty estimation under distribution shifts.
- Is the proposed attack method still applicable to adversarially robust networks?
- It would be nice if the paper could provide a more clear comparison of ACE with the standard adversarial attack: for example, how the attack objective of prediction confidence different from the standard cross-entropy? What if one applies ACE with the cross-entropy loss?
- What would happens if one performs adversarial training using the samples found by ACE? Will the resulting network gets better robustness/calibration/generalization?

[Ovadia et al., 2019] Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift, NeurIPS 2019. I could not find some clear statements that discuss their limitations or negative societal impacts in the text, so I suggest the authors to add the respective paragraphs in the revision.

Meta-review: The paper proposes a simple heuristic for manipulating the uncertainty estimation of a neural network without harming its accuracy by perturbing the inputs. Two reviewers (omcp and UycQ) suggest reject, while one reviewer (PoYC) does acceptance. In the discussion period, the authors provide additional experimental results and the positive reviewer (PoYC) further raised the score. Although Reviewer UycQ also appreciated that point, but still concerned that the paper needs more convincing arguments (or detailed experimental supports) on it, and Reviewer omcp also supported the concern. AC (and all reviewers) also agrees that the contribution of this work would be better conveyed if more extensive evaluations with simple modification of various existing solutions are provided. For example, one may consider DeepFool [Moosavi-Dezfooli et al., 2016] as an alternative baseline of ACE, as it is for estimating the margin of a given clean sample so that it can be also used to craft a perturbation that does not flip the classifier prediction. Although AC thinks that the paper is still an original work, AC tends to recommend reject given the current status.

Decision: accept