Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks
Authors: Sizhe Chen
Zhehao Huang
Qinghua Tao
Yingwen Wu
Cihang Xie
Xiaolin Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: adversarial defense
black-box attack
model calibration
score-based query attack

Abstract: The score-based query attacks (SQAs) pose practical threats to deep neural networks by crafting adversarial perturbations within dozens of queries, only using the model's output scores. Nonetheless, we note that if the loss trend of the outputs is slightly perturbed, SQAs could be easily misled and thereby become much less effective. Following this idea, we propose a novel defense, namely Adversarial Attack on Attackers (AAA), to confound SQAs towards incorrect attack directions by slightly modifying the output logits. In this way, (1) SQAs are prevented regardless of the model's worst-case robustness; (2) the original model predictions are hardly changed, i.e., no degradation on clean accuracy; (3) the calibration of confidence scores can be improved simultaneously. Extensive experiments are provided to verify the above advantages. For example, by setting $\ell_\infty=8/255$ on CIFAR-10, our proposed AAA helps WideResNet-28 secure 80.59% accuracy under Square attack (2500 queries), while the best prior defense (i.e., adversarial training) only attains 67.44%. Since AAA attacks SQA's general greedy strategy, such advantages of AAA over 8 defenses can be consistently observed on 8 CIFAR-10/ImageNet models under 6 SQAs, using different attack targets, bounds, norms, losses, and strategies. Moreover, AAA calibrates better without hurting the accuracy. Our code is available at https://github.com/Sizhe-Chen/AAA.

Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attacks
Sizhe Chen', Zhehao Huang', Qinghua Tao', Yingwen Wu', Cihang Xie', Xiaolin Huang
*Dpartment of Automation, Shanghai Jiao Tong University
?ESAT-STADIUS, KU Leuven
CComuter Science and Engineering, University of California, Santa Cruz
Abstract
The score-based query attacks (SQAs) pose practical threats to deep neural ner- works by crafting adversarial perturbations within dozens of queries, only using the model's output scores. Nonetheless, we note that if the loss trend of the outputs is slightly perturbed, SQAs could be casily misled and thereby become much less effective. Following this idea, we propose a novel defense, namely Adversarial At- tack on Attackers (AAA), confound SQAs towards incorrect attack directions by slightly modifying the output logits. In this way, (1) SQAs are prevented regardless of the model's worst-case robustness; (2) the original model predictions are hardly changed, i.e., no degradation on clean accuracy; (3) the calibration of confidence provided
scores be improved simultaneously. Extensive experiments verify the above advantages. For example, by setting &o2 8/ 8/255 on CIFAR- 10, our proposed AAA helps WideResNNe-288 secure 80.59% accuracy under Square attack (2500 queries), while the best prior defense (i.e., adversarial training) only attains 67. 67.44%. Since AAA attacks SQA's general greedy strategy, such advantages of AAA over 8 defenses can be consistently observed on 8 CIFAR- PRR-10//mageee models under SQAs, using different attack targets, bounds, norms, losses, and strategies. Moreover, AAA calibrates better without hurting the accuracy.
Introduction
Deep Neural Networks (DNNs) are vulnerable to adversarial examples (AEs), where human- imperceptible perturbations added to clean samples can fool DNNs to give wrong predictions 2 Recently, such a threat made practically feasible by the black-box score-based query attacks (SQAs)
3.45. they only require the same information judgments, need the model's prediction confidence indicated by DNNs' output scores, which the only knowledge needed by SQAs perform attacks. In contrast, white-box attacks 66
to craft efficient AEs. Users, for better
7 g
transfer-based attacks 9 10) require the gradients or training data of DNNs. Moreover, has been shown that SQAs achieve non-trivial attack success rate by reasonable number of queries, e.g., dozens, compared to thousands queries for decision-based query attacks [2 [3. Thus, such feasibility and effectiveness of SQAs are attracting increasing attention from defenders [4]
Defending against SQAs a different goal compared to improving the worst-cuse robustness as commonly studied 15 16 [7]. Because in real-world scenarios which SQAs are designed for, DNNs are treated as black boxes, which the only interaction between models and users/attackers is the model's output scores. Thus, altering the scores is all defenders could do here, either in direct or indirect ways. Most existing defenses indirectly change outputs by optimizing the model 8 18. 19 pre-processing the inputs [14 20), which, however, severely affect models' normal performance for different reasons. Training a robust model, e.g., by adversarial training [ 21 22).
diverts the model's attention to learning AEs, yielding the so-called accuracy- robusstesssss trade-off
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

pre-pro
in-model
post-pro
adversarial training
margin loss
randomizing
special architecture
y
None
denoising
training randomness
ours
AAA-linear
quantifying
dynamic inference
AAA-sine
training- time defense
testing time defanse
adversarial direction
Figure 1: Compared to existing defenses inputs models, AAA post-processes to avoid SQAs as in (a). Our main idea is to show attackers the incorrect attack direction as in (b). Specifically, if we perturb the original undefended blue loss curve to the orange or green one, then attackers trying to decrease the loss would be mostly cheated away from their destiny, i.e. the adversarial direction.
23 Randomizing 14
26 27 and blurring 20 28
30 inputs reduce the signal-noise
ratio, inevitably hurdling accurate decision. Dynamic inference T9 is timecconumiig due to the test-time optimization on model. Thus, it is imperative to develop user-friende sser-friendly defense against SQAs. In this paper, we hereby consider a post-processing defense as demonstrated in Fig. (a). which naturally enjoys the benefits as follows: (1) model's decision is hardly affected since good predictions have already been obtained; (2) model calibration can be simultaneously improved via post- proseescessing 32] 33 to output accurate prediction confidence; (3) it can be flexibly used as plug-in module for pre-trained models with negligible test-time computation overhead. Despite these merits, it remains unexplored, to the best of our knowledge, that by post-processing,
How to serve users while avoiding SQA attackers when they access the same output information?
Since SQAs are black-box attacks that find the adversarial direction to update AEs by observing the loss change indicated by DNN's output scores, can perturb such scores directly to fool attackers into incorrect attack tracks, Following this idea, we propose the adversarial attack on attackers (AAA), which manipulates the loss trend so that attackers. trying to greedily update AES following the original trend, are led to an incorrect path. Specifically, AAA directly optimizes DNN's logits to control the output loss to approximate certain designed curves so that attackers, seeing our confounding outputs, would be attacked to lose their direction. In Fig. (b), we give two representative designs of the loss curve to achieve the above goal. One increase loss mostly and decrease dramatically and shortly along the correct attack direction, and another is to oscillate the curve around the original blue line. Now that a post-proesssing module is adopted here, we could simultaneously use to calibrate the model as commonly proposed [32 34 Thus, a simple plug-in post-module is all you need to both fool attackers and offer users accurate confidence.
By post-processing, AAA not only lowers the calibration error without hurting accuracy in all cases, but also efficiently prevents SQAs, e.g., helping a WideRes (35] on CIFAR-IO (36 secure 80.59% accuracy under Square attack [5] (2500 queries), while the best prior defense (i.e., adversarial training (37) only attains 67. .4%. Even if attackers try guess defender's 's strategy (which requires heavy queries) and develop adaptive strategies, AAA could also confound both non-adaptive and adaptive attacks by the sine design. Because AAA attacks the general greedy update of SQAs, such advantages of AAA over 8 baseline defenses can be consistently observed on 8 CIFAR- 10/mmgeNet models under SQAs, using different attack targets, bounds, norms, losses, and strategies, verifying AAA a user-friendly, effective, and generalizable defense. Our contributions are three-fold.
We analyze current defenses from the view of SQA defenders and point out that a post- processing module forms not only effective but also user-friendly and plug-in defenses.
We design a novel adversarial attack on attackers (AAA) defense that fools SQA attackers to incorrect attack directions by slightly perturbing the DNN output scores.
We conduct comprehensive experiments showing that AAA outperforms the other 8 defenses in the accuracy, calibration, and protection from all tested 6 SQAs under various settings.
2
Related work
Query attacks are black-box attacks that only require the model's output information. Query attacks can be divided into score-based query attacks (SQAs) B
4
38
33
41 and decision-
based query attacks (DQAs) [ 12 13 42). SQAs greedily update AEs from original samples by

---

observing the loss change indicated by DNN's output scores, i.e., logits or probabilities. Early SQAs try to estimate DNN's 's gradients by additional queries around the sample
Recently, is It has also been proposed which, however, demands unfeasible access [O]. Thus, we do not focus on defending such SQAs as also in [14], Besides SQAs, DQAs rely only on DNN's decisions, egg., the top-1 predictions, to generate AEs. Since DQAs could not perform the greedy update, they start crafting the AE from a different sample and keep DNN's s prediction wrong during the attack. Currently, DQAs need thousands of queries to reach non-trivial attack success rate [45] compared to dozens of times for SQAs [3] as compared in Appendix A limiting their threats. Additionally, SQAs are mostly applicable and attackers do not have to resort to DQAs because, in the real world, DNNs must not only be accurate but also indicate when they tend to be incorrect [32 by outputting confidence scores.
39
validated query-efficient to perform fast pure random-search SQAs
4 S
to use pre-trained surrogate models SQÅs
40 44
to DNN's training sample as in transfer-based attacks p
Adversarial defense mainly has two different goals. The first goal is to improve DNN's worst-case robustness 17). which demands DNNs to have no AE around clean samples bounded by a norm ball. Such robustness is originally evaluated by gradient- based white- box attacks 7 but later, gradient obfuscation phenomenon is discovered [15). motivating evaluations to incorporate random noise and black-box attacks [16] in adaptive methods [T6] 46 47.. In this assessment, adversarial training (AT)
8 21 is validated as the most effective defense as also against poisoning attacks Other defenses, e.g., using more or augmented data 50 51. designing special architecture
48
49
18
52
53
and inducing randomness in training [54] 55 all need collaborations with AT to achieve
good performance. Another defense goal is to mitigate real-case adversarial threats, and the defense performance is evaluated by the feasible and query-efficient SQAs. A more robust model in the worst cases certainly protects itself in real cases. Besides, is also possible to defend by dynamic inference [19] (1), and randomizing [1 24 25 26.. denoising 120) 28 or quantifying (29 30)
inputs. However, all aforementioned defenses on inputs or models exert a non non-negligible impact on accuracy, calibration, or inference speed due to the focus on learning AEs in AT, the reduction of signal-noise ratio in procppocsss,ng or the costly test-time optimization in dynamic inference as compared meticulously in Section 3
Model calibration performance is a higher demand for a good model besides high accuracy. Besides correct predictions, model calibration additionally requires DNNs to produce accurate confidence in their predictions [56). For instance, exactly 80% of the samples predicted with 80% confidence should be correctly predicted. In this point of view, the expected calibration error (ECE) B3 is widely used to quantify the error between accuracy and confidence. Various methods have been proposed improve DNN's calibration in training 34 57
or by a post-processing module after training 332 33 60 Among them, temperature scaling [33] is simple but effective method
58
59
[32. which divides all output logits by a single scalar tuned by validation set so that the ECE in testing samples would be significantly reduced. Since division is simple post-processing operation, calibration could be simultaneously achieved by AAA, the first post-processing defense. There have been methods to avoid attacks by calibration [59] or calibrate by attack [34], but the simultaneous improvement of defense and calibration has not been reported to the best of our knowledge.
3 Preliminaries and motivation
Before presenting the proposed method, we first introduce the key ideas of SQAs and analyze existing defenses. For sample æ labelled by y. an SQA on DNN f generates an AE x' by minimizing the
margin between logits 4 5 41 61
as
[(f(x).)) = fz(z) max fx)),
(1)
kyy
namely the margin loss 77. For defenders without the label y, it is possible to calculate the unsuper- vised margin loss based only on the logits vector z = f(x) as
Ca(z) A (((()),))
by assuming the model prediction á to be correct, since handling originally misclassified samples falls beyond the scope of defense [17. For attackers with label y, it can be known that the attack succeeds if (((()) y) < 0. To quickly realize this, SQAs only update AEs if query Dg has a lower
3

---

margin loss compared to the current best query 2k. i.e.
Lq.
((f(xa)))) < (() ((((xa)))) 2 (((()))))).
I++1
Existing SQAs 45 41 are quite effective, capable of halving the accuracy within 100 queries on CIFAR-10 when &c 8/ 3/255, posing significant and practical threats. Aware of this, the first defense specifically against SQAs has been recently proposed [1] to pre-procest inputs by random noise. also useful to avoid SQÅs, among which the most representative ones are adversarial training (AT) 8 21, dynamic inference [99 62], and pre-procsssing [14] 22)..
Besides, other existing defenses
These defenses work in different mechanisms illustrated in Fig. (a) and Section Here in Table
1 we present a comparison of their key characteristics related to the defense performance.
Table 1: Expectation and effects of current defenses (unpreferable effects are marked in red)
expectation AT 21
pre-pro 14 29
dyn-inf
19 31
AAA (ours)
accuracy calibration
testing cost training cost ace under SQA
M1
11
Real-case defenders are expected to serve users and meanwhile avoid SQA attackers. The former demands good accuracy, calibration, and inference speed. While the latter requires good protection, and preferably, no additional training (denoted as "="). As a training- exerts significant impact accuracy (23), e.g. ~10% accuracy drop ImageNet [52), with several- fold training computation and an indefinite impact (denoted as "/"" on calibration. Pre-processing is test-time defense on input, which avoids SQAs by reducing the input's s signal-noise ratio but inevitably hurts normal performance. Recently, the dynamic inference is proposed as a test-time defense by optimizing the model, which shows accuracy drop but dramatically increases the computation, egg. DENT [19 consumes >700% test-time calculation. Given the above discussion, it imperative to develop user-friendly and efficient method to effectively defend against SQAs.
defense on model, AT
4 Adversarial attack on attackers
Defending by post-processing naturally carries the advantages of good accuracy and calibration [32] no additional training and negligible test-time computation. Although does not improve the commonly-studied worst-case robustness [16), its potential to defeat SQAs in real cases has not been explored yet to the best of our knowledge. By investigating current defenses from the perspective of SQA defenders in Sec. it is interesting to find that in the real world, no matter whether defenders alter inputs, models, or outputs, what users/attackers get are just the changes in outputs, iee, the black-box setting. Thus, it is reasonable and preferable for SQA defenders to directly manipulate DNN's output scores, which has already become a standard practice in model calibration [2]
But how? Now that attackers conduct the adversarial attack on the model, why cannot we defenders actively perform an adversarial attack on attackers? If attackers search for the adversarial direction by scores, we can manipulate such scores to cheat them into an incorrect attack path. Following this idea, we develop the adversarial attack on attackers (AAA) to control what the SQA attackers base their actions on, i.e., the loss ( indicated by the DNN's output scores. Under such direct misleading, SQA attackers, following their original policy, would be guided to wherever we want them to go due to their greediness, seeing 33. But such change outputs should be slight to ensure it is just attackers (observing loss trend) rather than users (observing outputs) that are misled. Therefore, the loss trend can not be reversed completely, but only be reversed locally in periodically-set intervals.
In each local interval, there are lots of designs to manipulate the loss trend. For example, we can let the loss along the adversarial direction (almost) always increase and only dramatically decrease between intervals, seeing the orange line Fig. (b). is also possible to smooth the cross-interval loss change by making it oscillate like the green line in Fig. (b), so that attackers could not easily

---

guess defender's strategy by observing the loss drop. Either design cheats attackers by slight output modifications so that users get accurate scores. This because we manually divide loss values into small intervals and handle them separately, which is realized by designing the misleading loss curve based on periodic loss attractors as
latr = (floor (lorg/T) 1/2)) TT,
(4)
where lorg = Cu (org) the original loss for the unmodified logits Zorg. T the period of attractor intervals, and "floor" denotes rounding down decimals to integers. Eq. 4 sets loss values from Butr - 7/22, lutr + 7/2] as an interval. For example, if r = 6, then according to Eq. (4) the closest loss attractor of logits with lorg = 0 - 6 is latr = 3. Thus, the 1/2 term is necessary to avoid setting latr = 0, i.e., the decision boundary, which may flip model's decision.
Periodically-set loss attractors divide logits with different long into intervals, so that we could manipulate the loss trend in each interval form our designed loss curve that cheats attackers. The target loss values lug in the two misleading curves lug Iner lurg.stu in Fig. [bb) can be expressed as
lurg.lr latr a x (lorg latr) lirg.snn lorg x ssin(x(11 - 2(barg latr)/T)).
(5)
In the former linear design, when lorg decreases from late + 7/2 to latr - 7/2 as the sample approximates the decision boundary, we output loss that increases from late- -ax=aa to latr ta T/2 to fool attackers that this is an incorrect attack direction. In contrast, AAA-sine outputs an increasing loss when lorg decreases from around latr + 7/4 lat - 7/4. Although we do not always reverse the loss trend here, is sufficient to resist SQAS. Plus, AAA-sine enjoys an additional advantage to fool attackers that try to guess our defense strategy due to the smooth loss transition across intervals.
Although the periodic design has already altered output confidence very slightly, we could also step further to improve the precision of confidence because post-processing logits is a standard practice in model calibration [33 60]. Thus, simultaneous defense and calibration is obtainable in a single AAA module by controlling the loss while encouraging the output confidence a (() max((sftmaxx()) i.e., the maximum probability after softmax, to approach the calibrated one Parg as
min || Cz) leng Il + 3 (z)) - Porg Il
(6)
which is a straightforward design to fulfill two purposes. The first term encourages the perturbed logits z to have a loss (=) close to the target value liz.. forming the misleading loss curve for SQAs. The second item motivates the output confidence (z) close to the calibrated one ptrg so that users get accurate confidence scores. ,3 balances between the above two goals. Despite its simplicity,
Eq. (6) has to be solved by optimization because the exponential operation in softmax makes Eq. (6) a transcendental equation without closed-form solutions. Luckily, optimizing low-dimensional logits is not costly [33 66). By optimizing Eq. (6) for к iterations, DNN's output is both accurate and misleading, achieving two seemingly contradictory goals simultaneously.
The calibrated confidence Pirg is obtainable by various model calibration methods 33 60
Among them, temperature scaling [33] is simple but effective (.2 which divides all logits by a scalar T as Purg- = o (Forgg /T). The temperature T is tuned by validation set to minimize the calibration error, and here, such tuning is conducted when the optimization (6) also performed for x iterations so that we can find the temperature that suits AAA best. After that, T is fixed for inference in AAA.
We summarize our algorithm in Alg. D AAA first calculates the original loss lorg. according to which sets the target loss 4ig that forms the mis- leading loss curve. Then AAA optimizes the logits to reach lerg. and also the target confidence Perg, which is obtained by dividing the original logits using a pre-tuned temperature T. The overall pro-
Algorithm 1 Adversarial Attack on Attackers
Input: the logits Zorg, T, Qt B, K. Output: post-processed logits
I: get original loss lorg = Lu (zogg)) by 2
set target loss lerg by (5)
3: set target confidence Pirg = a (z TT)
4: initialize z
Zorg and optimize it for
cedure is in test-time with only optimization on
the logits, making AAA computation-effica in, and model-agnostic method with good accuracy, calibration, and defense against SQAs.
5: return z
5

---

5 Experiments
5.1 Setup
We evaluate AAA along with 8 defense baselines, including random noise defense (RND 34 )
adversarial training (AT [37 63 (2),, dynamic inference (DENT [9)), training randomness (PNI [54]), and ensemble (TRS (.4)). Results of AT with extra data (23) are in Appendix For AAA,
mostly experiment with the linear design, because the two designs of AAA share most good characteristics. AAA-sine could fool adaptive attacks, which is costly to design in real world as discussed in Sec 5.6 AAA-linear uses
100, and Adam optimizer [65] with learning rate 0.1, = 0.9, f2 = 0.999. AAA-sine uses a = 0.7 and inherits other hyper-parameters in study of adaptive attacks. To calibrate simultaneously, we perform temperature scaling using JK/5K samples for CIFAR-IO testing set set/ ImageNet validation set to and make them disjoint with the attack samples as much possible. Other hyper-parameters in Appendix
1, 6, 3 =
The defenses are assessed by 6 state-of-the- SQAs, which are random search methods including Square [5), SignHunter [41], and SimBA [4] and gradient estimation methods including NES [61]. and Bandits [3]. Square attack [5] selects a decreasing size of random square perturbations. SignHunter [1] estimates the sign of gradient and flips corresponding perturbations. SimBA 4 randomly samples perturbations given an orthonormal basis. NES [61] adopts the Natural Evolutionary Strategies for gradient estimation. Bandits 3 jointly uses the time and data-dependent gradient prior. Some SQAs use pre-trained models 40 44]. which, however, demands unfeasible access to DNN's training sample. To evaluate real-case defenses against training-based attacks, we alternatively consider a practical SQA QueryNet (66) based on model stealing. Results of DQAs and SQAs using other than margin loss are put in Appendix [A] We mostly perform untargeted for attacks, but we also test the targeted and Eg attacks under different bounds to observe AAA's generalization.
We use 8 DNNs, which mostly WideResNets [35 as in [7 The pre- pre-trained models of PNI (54) / TRS (64] are ResNet-20 (67], and we also test ResNeXt- 101 [68] in IetgeNet. Other studied DNNs come from RobustBench [17 and torchvision 669 as specified in Appendix B
AT models are tested using the same bound as in AT, if not otherwise stated. We use all 10K CIFAR- 10 testing samples. For ImageNet, we randomly-select IK validation samples from all JK classes respectively (1 image in I class) to eliminate the class bias as in [66]. All images are rescaled to [0, 1J. and the ImageNet ones are resized 224 224. Before feeding them DNNs, we quantify images 8-bit, imitating the real- case 8-bit image setting [66]. Experiments are performed on an NVIDIA Tesla A 100 GPU (a GPU with 4GB+ memory works). Our code available at hhtps: //tghbb.t com/ Siih/iizhhee Chen/ AAA
As defenders, we are concerned about DNN's remaining accuracy after it being attacked by SQAs for certain query times. Thus, report such SQA adversarial accuracy after 100 and 2500 queries, reflecting DNN's performance under mild and extreme SQAs. We include the average query times, another metric commonly used by attackers, in Appendix C To measure the calibration, the expected calibration error (ECE) [3 is commonly used. ECE divides all testing samples into M
bins, and each bin contains samples with confidence ranging from the mh quantile to the (m + +1) quantile [0, 1). Then ECE calculated by the difference between accuracy and confidence as M 1. EM= Eie B.. 1 y:) EieB p. where ú; is the predicted label of the 7th sample in
the thth bin Bm, and p. represents the probability confidence of this prediction.
margin loss
undefended defended (AAA)
10
decision boundary
100
200
300
400
500
600
query
Figure 2: The AAA(iinear-deefende and undefended margin loss value when attacking the unde- fended WideResNet-28 (35) in RobustBench [7 by Square attack [3] (Exc 8 8/255) using the 9953th CIFAR- 10 test sample (other samples have similar trends as in Appendix B). AAA fools attackers precisely as shown by the tiny symmetric oscillations of two lines.

---

Defenses' Influence on DNN Scores
Docisions
SQA Adversarial Accuracy of Defenses
1.0
100
Undefended
AAA
0.8
80-
RND
contidence (aceuracy
0.6
Confidence of Undefended model 3.F-3.522
Accuracy of Undefended model 94.-9%.78%
Confidence of AAA ECE-2.46%
0.4
Accuracy AAA model Acc 94.894
40-
Confidence of RND model ECE-6.32%
0.2
Accuracy RND model
Acc-91
Confidence of AT model ECE-11.00%
Accuracy AT model Acc-87.02%
20
40
60
80
100
0
500
1000
1500
2000
2500
sample peroentile Isorted by undefended model's ground-trutrelass probubility]
Qquery
Figure 3: The left figure illustrates the change of output scores for different defenses. We first sort 10K testing samples in ascending order according to their ground-truth--de the undefended model (blue line), and then divide them into 100 bins, so that the left bins in the figure stand for low-confidence or misclassified samples, and vice versa. Then for samples in each bin, we plot the confidence (solid lines) and accuracy (dotted line) for AT (37,, RND R4. and AAA-linear Compared to AT and RND, AAA alters scores most slightly without inftuencing the accuracy. The right figure shows DNN's accuracy under Square attack [5] when & 2.5, indicating that AAA outperforms alternatives in defending SQAs by large margin with also the highest clean accuracy.
probability predicted by
5.2 Visual illustration of AAA
We first visually illustrate the mechanism and effects of AAA. AAA, besides fooling attackers, also calibrates the model, forming the loss curve as in Fig. 2 Slightly different from the ideal defense in Fig. (b) is that the orange curve is lowered for calibration. However, calibration does not hurdle fooling attackers, i..,, the orange line is mostly going opposite to the blue line in precisely symmetric manner. Also, two lines cross the decision boundary at the same time, meaning that AAA does not change the decision (the negligible increase of AAA accuracy comes from randomness).
Besides the loss, is also necessary to observe the output probabilities of AAA, which is displayed in Fig. (left). AAA, indicated by the orange lines, changes the scores slightly without hurting accuracy (dotted lines) compared to RND 34] and AT [37] By such small modifications, AAA not only improves the calibration but also prevents SQAS by misleading them into incorrect directions. As displayed by the right part of Fig. 3 AAA is super effective in avoiding SQAs, even if it is a query-efficient attack working at large bound. Specifically, AAA preserves a standard trained DNN to have >00 accuracy after 2500 queries, doubling the performance of AT and tripling that for RND.
5.3 Numerical results of AAA
We report the main numerical results in Table where RND and AAA-lincar are directly implemented in the undefended model denoted as The AT method for the three models are PSSiLU (37]. vanilla AT [63] and feature denoising [52) respectively. According to Table 2 AAA-linear not only consistently reduces ECE by >12% but also does not hurt clean accuracy. In contrast, the AT models lose >7% accuracy and RND also endures drop in accuracy and calibration. Under SQAs, AAAllinear preserves a significantly higher accuracy than the undefended model, which is totally destroyed. The most threatening SQAS in two datasets more than halve the CIFAR-10 model's accuracy within 100 queries and degrade ImageNet ones to <15% by 2500 times. However, the AAA model remains >75% and >55% accuracy in extreme cases. AT and RND are useful in mitigating SQAs, but is AAA-linear that tops the defense performance in almost all cases.
Besides AT and RND, diverse defenses have also been proposed. DENT 19 optimizes the model in test-time. PNI (54) injects noise during training. TRS [64] ensembles three models with low attack transferability. They are developed for gradient-basse based attacks, but also provide protection against SQAs. However, seeing T they are not comparable to AAA-linear in real cases regarding the accuracy, calibration, and defense performance. Here, we also test a strong SQA QueryNet, which uses three aechitecture-alterabbe models to steal the DNN. Due to its utilization of large-scale testing samples, QueryNet greatly hurts DNNs, but AAA is still the defense that protects the model best.
7

---

Table 2: The defense performance under attacks (#query 100/2500)
Model
Metric / Attack
None
AT
37
63
52
RND [14]
AAA-linear
CIFAR-10
ECE (%)
3.52
11.00
6.32
2.46
loo 255
Acc (%)
94.78
87.02
91.05
94.84
Square 5
39.38
00.09
78.30 / 67.44
60.83 / 49.15
81.36 80.59
Wide- ResNet- 28 35
SignHunter
41
41.14 / 00.04
78.87 / 66.79
61.02 / 47.82
79,41 / 76.71
SimBA
4
53.04 03.95
84.21 75.85
76.39 64.34
88.86 83.36
NES 61
83.42 / 12.24
85.92 / 81.01
86.23 / 68.19
90.62 85,95
Bandit 3
69.86 / 41.03
83.62 / 76.25
70.44 41.65
80.86 78.36
ImageNet 6o0 255
ECE (%)
5.42
5.03
5.79
4.30
Acc (%)
77.11
66.30
75.32
77.17
Square 5
52.27
09.25
59.20 / 51.11
58.67 / 50.54
/ 62.51
Wide- ResNet- 50 [35
SignHunter 41
53.05 / 13.88
59.47 / 56.22
99.36/52.99
62.35
56.80
SimBA 4
71.79 / 20.90
65.64 / 47.60
66.36 / 63.27
74.16 2.16/612 67.14
NES 61
77.11 / 64.93
66.30 / 64.38
71.33 / 66.05
77.12 67.06
Bandit 3
71.33 / 65.77
65.30 / 63.98
65.15 / 61.38
72.15 / 70.53
ImageNet loo 255
ECE (%)
8.37
5.74
8.93
7.38
Acc (%)
78.21
63.94
76.75
78.21
Square 5
54.51 / 11.73
57.99 / 51.47
58.56 / 48.20
66.32 65.77
Res-
SignHunter [1
54.12 13.30
59.14 / 56.49
58.02 / 52.50
62.72
59.60
NeXt-
SimBA
4
70.86 / 24.64
59.40/57.55
68.31 / 66.16
74.06 / 67.26
101
NES 61
78.21 66.24
63.94 / 62.51
73.37 69.30
78.21 68.51
Bandit 3
72.19 / 67.73
63.18 / 62.26
67.69 / 64.62
72.91 / 71.48
Table 3: Various defenses under strong attacks (#query 100/2500, CIFAR- f. 8/255)
Metric/ / Attack
None
DENT [9
PNI [54)
TRS [64]
AAA-lincar
ECE (%)
3.52
5.20 94.80
3.09
3.64 88.64
2.46
Acc (%)
94.78
81.91
94.84
Square [5]
39.38 00.09
62.01 / 35.07
57.69 / 45,47
56.26 / 23.91
81.36 / 80.59
QueryNet
13.50 / 00.03
39.35 / 20.75
44.06 34.69
16.43 / 08.49
50.01 / 49.63
5.4 Generalization of AAA
Aside from the untargeted loo attacks, we also conduct targeted attacks, l attacks under different bounds to study the generalization of AAA. A targeted attack is successful only if AE is mispre- dicted as a pre-sct class, which, is randomly chosen from incorrect classes for each sample here. And lg attacks bound the perturbations by l2 norm, which is reported fool DNNs better 77). Here we additionally validate AAA's plug-in advantage by combining it with AT. Note that what differs from AAA and most existing defenses [52 53 (44 combined with AT is that AAA has already achieved excellent defense performance,
such combination is feasible but not necessary. We choose Square attack (5] to perform the above evaluations because is the most effective SQA as tested in Table 2 The results are presented in Table where all models are exactly the same in Table hyper-paranen parameters of defense to fairly evaluate the generalization of different methods. The results of le AT model and other &oo AT models are put in Appendix
2 (CIFAR--O)) without tuning hyper-
In the difficult targeted attack setting, the undefended model remains only 2.84% accuracy after 2500 queries, which is approximately just the accuracy drop of the AAA model. For &g attacks, AAA-linear still capable of mitigating threats without hurting users, and its superiority more outstanding as the attack bound becomes larger. AT models, although robust, suffer from attacks under large or different norm ball [59 Thus, its defense effects decrease as SQA alters the setting, seeing the bottom line. Defended by AAA, however, this drawback would be greatly avoided. An foc. AT model, even under f2 = 2.5 attack after 2500 queries, hardly influenced, iee, increasing query times after 100 queries hardly better the attack performance, discouraging SQA attackers.

---

90-
AdcMee
AdvAuc
Adh Assuriyy
00-
ECE (No Tongenuuu
ECR (S0) Tomp comeommrr
40- ECE (nd))
ECE (c4f)
CleanAee AdvAce
Tomp.
63-
2
DCE
70-
BCC
Clon Assimm
Temp.
88
60-
70
10' calibration loss weight
attractor interval
reversc step
Figure 4: The influence of AAA hyper-parameters (the attractor interval T in (4). the reverse step a in ( and the calibration loss weight / in (6) accuracy (left axis). ECE (right axis). SQA adversarial accuracy (left axis), and temperature (right axis). The dashed ECE (ref) means the undefended model.
Table 4: Generalization of AAA tested by Square attack [ (#query 100/2500, CIFAR-10)
Metric/ Attack
None
AAA-linear
AT 37
AT-AAA-line
ECE (%)
3.52
2.46
11.00
10.56
Acc (%)
94.78
94.84
87.02
87.02
untargeted 6o = 8/255
39.38 / 00.09
80.59 92.05
81.36
78.30 / 67.44
80.80 80.13
targeted 6oo 8/255
75.59 / 02.84
91.62
85.75 82.72
86.22 86.13
untargeted lg = 0.5
81.53/ / 18.75
92.66 92.63
84.26 78.97
85.12 84.31
untargeted 2.5
12.77 / 00.01
70.35 63.46
57.88 / 25.19
74.03 73.72
5.5 Hyper-parameters of AAA
The reported amazing results in various settings are all obtained using fixed parameters that are heuristically selected, and would be necessary to see how each them affects the results. The attractor interval in 44) decides the period of the margin loss attractors, and a larger divides the losses into fewer intervals so that attackers are harder to jump out of one of them. The reverse step in (5) controls the reverse degree, and if it increases, the modified loss curve would be steeper oppositely along the adversarial direction, emphasizing the defense. The calibration loss weight in (6) directly balances defense and calibration in optimization. Here we still consider the accuracy, ECE, and SQA adversarial accuracy (under 100 queries). Plus, we study an additional metric, the temperature T of logits rescaling tuned with AAA-linear.
The results are shown Fig. 4 The adversarial accuracy far exceeds the undefended model (39.38%) and mostly surpasses AT (67. (67.44%). The clean accuracy is hardly impacted and the ECE is mostly below the undefended model (the green dashed line). Thus, AAA's good performance is insensitive to hyper-parameters, even if tuned in logarithmic scale. Regarding the trend, intuitive conclusion that larger a, a larger T, or smaller 3 that highlights defense more v.s. calibration would thereby increase the adversarial accuracy and ECE. Interestingly, the temperature mostly decreases as the defense is emphasized (the logits are divided by a smaller value), indicating that the model tends to output lower confidence defend, consistent with the situation in AT [17].
We also study the influence of optimization times k. in AAA to see the balance between the amount of computation and defense performance. AAA's runtime is estimated by inferring 10K CIFAR-10 samples in Wed22esNNt28 model on an NVIDIA GeForce RTX 2080Ti GPU. The results are optimizing fow-dimensional logits is not costly, and good defense and
shown in Table 5 As calibration results are also obtainable by 60 to 80 iterations, which costs less time. Since optimizing logits is independent of model size, model owners could determine AAA runtime precisely.
Table 5: Influence of the optimization times in AAA (100-query Square attack on CIFAR- 10)
20
40
60
80
100
ECE
3.52
2.87
2.81
2.66
2.53
2.53
Adv-Acc
39.38
79.29
80.92
81.37
81.28
81.36
inference time per sample (ms)
1.016
1.034
1.088
1.099
1.143
1.163

---

5.6 Adaptive attacks of AAA
Designing adaptive attacks, though possible, is costly and easy to bypass in real-world scenarios. Because here, attackers and defenders are in a double-blind relationship, i.e., attackers do not know the model, including the defense strategy. Thus, the discovery process of defense strategies for developing adaptive attacks would require additional queries. After that, attackers also have to devote considerable manual efforts to creatively deduce what defenders actually do. In this regard, AAA-linear has already imposed great hurdle adaptive attackers.
But still, is interesting to see that after unveiling defender's strategy by great effort, how adaptive attackers could break the AAA model. There are two straightforward adaptive attacks against AAA: going bidirectional or opposite to the original attack direction, which are both based on the exceptional loss change that attackers observe in AAA-linear to decide the update direction. However, adaptive attackers are also easy to fool because the dramatic drop of loss across intervals could be smoothed, e.g. by using sine function to design the target loss as the second equation in (5). In this way, neither direction of search is likely to figure out the defense strategy, seeing results below.
The bidirectional opposite search weak- ens AAA-linear from 81.36% to 62.91% % / 57.31% %, but AAA-sine that ascends and descends the loss along the attack direction misleads both non-adaptive and adaptive SQAs. By a sine lug. the defended adver- sarial accuracy for the non-adaptive attack is kept at 78.34% while that under adaptive attacks is improved to 76.69% / 76.41 %. Thus, defenders could easily mitigate even adaptive SQAs following our idea to fool attackers.
Table 6: AAA under adaptive attacks (100 queries)
Defense
None
AAA-linear
AAA-sine
Square
39.38
81.36
78.34
bi-Square
57.09
62.91
76.69
op-Squar
94.78
57.31
76.41
6 Conclusion, impacts and limitations
We develop a novel defense against score-based query attacks (SQAs). Our main idea is to actively attack the attackers (AAA), misleading them into incorrect attack directions. AAA achieves that by post-processing DNN's logits while enforcing the new output confidence to be calibrated, making AAA a deterministic plug-in test-time defense with improvements in calibration and accuracy by costing negligible computation overhead. Compared to alternative defenses, AAA is effective in mitigating SQAs according to our study on 8 defenses, SQAs, and DNNs under various settings.
As a defense in real-world applications, AAA greatly mitigates the adversarial threat without without requiring huge computational burden. For example, in autonomous driving or supervision systems, the post- processing defense module could be directly implemented in pre-trained models. Despite the low cost, the benefits of adopting AAA are profound. In most cases, users would be provided with more accurate confidence score so that they know better when the model tends to fail. In adversarial cases, SQAs, the most threatening attack real cases, would be effectively prevented.
AAA is developed to especially prevent SQAs. Thus, defending other types of attacks is beyond our scope. For example, AAA does not improve the worst-case robustness evaluated in white-box settings where attackers have complete knowledge of the model (the AutoAttack [66 robust accuracy would be increased in undesirable manner by AAA) Also, AAA is not applicable to avoiding transfer-based attacks and decision-based query attacks, which are either unfeasible or inefficient in the real world, because AAA induces only negligible impact on the decision boundary.
16 47
Acknowledgments and Disclosure of Funding
This work was partially supported by National Natural Science Foundation of China (61977046), Research Program of Shanghai Municipal Science and Technology Committee (22511 (22511105600), 105600), and Shanghai Municipal Science and Technology Major Project S22DXX22D000022), Cihang Xie is supported by a gift from Open Philanthropy. The authors are grateful to the anonymous reviewers for their insightful comments.
10

---

References
[I] C. Szegedy, W. Zaremba, I. Sutskever, Bruna, D. Erhan, J. J. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," in International Conference on Learning Representa- tions (ICLR), 2014.
[2] I. J. Goodfellow, Shlens, and Szegedy, "Explaining and harnessing adversarial examples." in International Conference Learning Representations (ICLR), 2015.
[3] A. Ilyas, L. Engstrom, and A. Madry, "Prior convictions: Black-box adversarial attacks with bandits and priors." " in International Conference on Learning Representations (ICLR), 2019.
[4] C. Guo, J. Gardner, Y. You, A. G. Wilson, and K. Weinberger. "Simple Black-box adversarial attacks," in International Conference Machine Learning (ICML), 2019, pp.
[5] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, "Square attack: A query-efficient black-box adversarial attack via random search," in the European Conference on Computer Vision (ECCV), 2020, pp. 484-501.
[6] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, "Deepfool: A simple and accurate method to fool deep neural networks," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2574-2582.
[7] N. Carlini and D. Wagner, "Towards evaluating the robustness of neural networks," in the IEEE Symposium on Security and Privacy (SP), 2017, pp. 39-57.
[8] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks," in International Conference on Learning Representations (ICLR), 2018.
[9] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille, "Improving transferability of adversarial examples with input diversity," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2730-2739.
[10] S. Chen, Z. He, C. Sun, and X. Huang, "Universal adversarial attack on attention and the result- ing dataset DemageNet," in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022, pp. 2188-2197.
[11] W. Brendel, J. Rauber, and M. Bethge, "Decision-based adversarial attacks: Reliable attacks against black-box machine learning models," in International Conference on Learning Repre- sentations (ICLR), 2018.
[12] M. Cheng, T. Le, P.-Y. Chen, H. Zhang, J. Yi, and C.-J. Hsich, black- box attack: An optimization based approach," " in International Conference on Learning Representations (ICLR), 2018.
hard-label
[13] J. Chen, M. Jordan, and M. J. Wainwright, "HopSkiplumpAttack: A query-efficient decision- based attack," in the IEEE Symposium on Security and Privacy (SP), 2020, pp. 1277-1294.
[14] Z. Qin, Y. Fan, H. Zha, and B. Wu, "Random noise defense against query-based black-box attacks," " in Advances in Neural Information Processing Systems (NeurIPS). 2021, pp. 7650- 7663.
[15] A. Athalye, N. Carlini, and D. Wagner, "Obfuscated gradients give a false sense of security: Cireumventing defenses adversarial examples," in International Conference on Machine Learning (ICML), 2018, pp. 274-283.
[16] F. Croce and M. Hein, "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks," International Conference on Machine Learning (ICML), 2020, pp. 2206-2216
[17] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, "Robustbench: a standardized adversarial robustness benchmark." in Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[18] Y. Li, Z. Yang. Y. Wang, and C. Xu, "Neural architecture dilation for adversarial robustness," in Advances in Neural Information Processing Systems (NeurIPS), 2021, pp. 29 578-29 589.
[19] D. Wang, A. Ju, F. Shelhamer, D. Wagner, and T. Darrell, "Fighting gradients with gradients: Dynamic defenses against adversarial attacks," arXiv preprint arX1v:2105.08714, 2021.
11

---

[20] A. Prakash, N. Moran, S. Garber, A. DiLillo, and J. Storer, "Deflecting adversarial attacks with pixel deflection," " in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8571-8580.
[21] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Bonch, and P. D. McDaniel, "Ensemble adversarial training: Attacks and defenses," in 6th International Conference Learning Representations (ICLR), 2018.
[22] T. Li, Y. Wu, Chen, K. Fang, and X. Huang. "Subspace adversarial training," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 1340--13418.
[23] R. Rade and S.-M. Moosavi-Dezfooli, "Helper-based adversarial training: Reducing excessive margin to achieve better accuracy vs. robustness trade-off," in ICML 202/ Workshop on Adversarial Machine Learning. 2021.
[24] H. Salman, M. Sun, G. Yang. A. Kapoor, and J. Z. Kolter, "Denoised smoothing: A provable defense for pretrained classifiers," ! in Advances in Neural Information Processing Systems (NeurlPS), 2020, pp. 21 945-21 957.
[25] C. Xie, J. Wang. Z. Zhang, Z. Ren, and A. L. Yuille, "Mitigating adversarial effects through randomization," " in International Conference on Learning Representations (ICLR), 2018.
[26] X. Liu, M. Cheng, H. Zhang, and C.-J. Hsieh, "Towards robust neural networks via random self-ensemble," in the European Conference on Computer Vision (ECCV), 2018, pp. 369-385.
[27] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana, "Certified robustness adversarial examples with differential privacy," in the IEEE Symposium on Security and Privacy (SP), 2019,
pp. 656-672.
[28] Z. Liu, Q. Liu, T. Liu, N. Xu, X. Lin, Y. Wang, and W. Wen, "Feature distillation: Dnn-oriented JPEG compression against adversarial examples," " in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 860-868.
[29] J. Buckman, A. Roy, C. Raffel, and I. Goodfellow, "Thermometer encoding:
One hot way to resist adversarial examples," in International Conference on Learning Representations (ICLR), 2018.
[30] C. Guo, M. Rana, M. Cisse, and L. van der Maaten, "Countering adversarial images using input transformations." in International Conference on Learning Representations (ICLR), 2018.
[31] Y.-H. Wu, C.-H. Yuan, and S.-H. Wu, "Adversarial robustness via runtime masking and cleans- ing," in International Conference Machine Learning (ICML), 2020, pp. 10 10399-100400 409.
[32] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in International Conference on Machine Learning (ICML), 2017, pp. 1321-1330.
[33] M. P. Nacini, G. Cooper, and M. Hauskrecht, "Obtaining well calibrated probabilities using bayesian binning," " in the AAAI conference on Artificial Intelligence (AAAI), 2015, pp. 2901- 2907.
[34] Y. Qin, X. Wang, A. Beutel, and E. Chi, "Improving calibration through the relationship with adversarial robustness," in Advances in Neural Information Processing Systems (NeurIPS), 2021, pp. 14 358-14369.
[35] Zagoruyko and Komodakis, "Wide residual networks," " in British Machine Vision Confer- ence (BMVC), 2016, pp. 17.
[36] A. Krizhevsky et al., "Learning multiple layers of features from tiny images," 2009.
[37] S. Dai, S. Mahloujifar, and Mittal, "Parameterizing activation functions for adversarial robustness." arXiv preprint 2t. 2021.
[38] N. Papernot, McDaniel, I. Goodfellow, S. Jha, B. Celik, and A. Swami, "Practical black- box attacks against machine learning." in the ACM on Asia Conference Computer and Communications Security, 2017, pp. 506-519.
[39] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, "Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models," in the 10th ACM workshop on artificial intelligence and security, 2017, pp. 15-26.
[40] S. Cheng, Y. Dong, T. Pang, H. Su, and J. Zhu, "Improving black-box adversarial attacks with transfer-based prior," Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 10934-10944.
12

---

[41] A. Al-Dujaili and U.-M. O' O'Reilly, "Sign bits are all you need for black-box attacks," in International Conference on Learning Representations (ICLR), 2019.
[42] M. Cheng, S. Singh, P. H. Chen, P.-.. Chen, S. Liu, and C.-J. Hsieh, "Sign-opt: A query-efficient hard-label adversarial attack," in International Conference on Learning Representations (ICLR), 2019.
[43] A. N. Bhagoji, W. He, B. Li, and D. Song, "Practical black-box attacks on deep neural networks using efficient query mechanisms," in the European Conference on Computer Vision (ECCV), 2018, pp. 154-169.
[44] Y. Guo, Z. Yan. and C. Zhang, "Subspace attack: Exploiting promising subspaces for query- efficient black-box attacks," in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 3820-3829.
[45] D. C. R. Viet Quoe Vo, Ehsan Abbasnejad, "Ramboattack: A robust query efficient deep neural network decision exploit," " Network and Distributed Systems Security (NDSS) Symposium, 2022.
[46] C. Yao, P. Bielik, P. Tsankov, and M. Vechev, "Automated discovery of adaptive attacks on adversarial defenses," " in Advances in Neural Information Processing Systems (NeurIPS), 2021,
pp. 26 858-26687 870.
[47] F. Tramer, N. Carlini, W. Brendel, and A. Madry, "On adaptive attacks to adversarial example defenses," " in Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 1633- 1645.
[48] S. Chen, G. Yuan, X. Cheng, Y. Gong, M. Qin, Y. Wang, and X. Huang, "Self-ensemble protection: Training checkpoints are good data protectors." arXiv preprint arXiv:2211.12005, 2022.
[49] S. Wu, S. Chen, C. Xie, and X. Huang. "One-pixe pixel shortcut: on the learning preference of deep neural networks," arXiv preprint arXiv:2205. /2141, 2022.
[50] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann, "Improving robustness using generated data," in Advances in Neural Information Processing Systems (NeurIPS), 2021, pp. 4218-4233.
[51] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, and T. A. Mann, "Data augmenta- tion can improve robustness," " in Advances Neural Information Processing Systems (NeurIPS), 2021, pp. 29 995-9959999 948.
[52] C. Xie, Y. Wu, L. v. d. Maaten, A. L. Yuille, and K. He, "Feature denoising for improving adver- sarial robustness," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 501-509.
[53] Y. Fu, Q. Yu, Y. Zhang. S. Wu, X. Ouyang, D. Cox, and Y. Lin, "Drawing robust scratch tickets: Subnetworks with inborn robustness are found within randomly initialized networks," in Advances in Neural Information Processing Systems (NeurIPS), 2021, pp. 13 139--133772 072.
[54] Z. He, A. S. Rakin, and D. Fan, "Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 588-597.
[55] H. Salman, J. Li, I. Razenshteyn, P. Zhang, H. Zhang, S. Bubeck, and G. Yang, "Provably robust deep learning via adversarially trained smoothed classifiers," in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 11 292-11 11 303.
[56] A. Niculescu-Mizil and R. Caruana, "Predicting good probabilities with supervised learning." in International Conference on Machine learning (ICML), 2005, pp. 625-632.
[57] C. Szegedy, V. Vanhoucke, S. Ioffe, Shlens, and Z. Wojna, "Rethinking the inception archi- tecture for computer vision," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2818-2826.
[58] S. Thulasidasan, G. Chennupati. J. A. Bilmes, T. Bhattacharya, and S. Michalak, "On mixup training: Improved calibration and predictive uncertainty for deep neural networks." in Advances in Neural Information Processing Systems (NeurlPS), 2019, pp. 13 13911-13922 922.
[59] D. Stutz, M. Hein, and B. Schiele, "Confidence-calibne adversarial training: Generalizing to unseen attacks," International Conference on Machine Learning (ICMI.), 2020, pp. 9155- 9166.
13

---

[60] B. Zadrozny and C. Elkan, "Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers." " in the AAAI conference on Artificial Intelligence (AAAI), 2001, pp. 609-616.
[61] A. lyyas, L. Engstrom, A. Athalye, and J. Lin, "Black-box adversarial attacks with limited queries and information," in International Conference on Machine Learning (ICMI.), 2018, pp.
2142-2151.
[62] Y. Wu, S. Chen, K. Fang, and X. Huang, "Unifying gradients to improve real-world robustness for deep networks," arXiv preprint arXiv:2208.0622 2022.
[63] H. Salman, A. Ilyas, L.. Engstrom, A. Kapoor, and A. Madry, "Do adversarially robust imagenet models transfer better?" in Advances in Neural Information Processing Systems (NeurIPS), 2020, pp. 3533-3545.
[64] Z. Yang, L. Li, X. Xu, S. Zuo. Q. Chen, P. Zhou, B. Rubinstein, C. Zhang, and B. Li, "TRS: Transferability reduced ensemble via promoting gradient diversity and model smoothness," in Advances in Neural Information Processing Systems (NeurIPS), 2021, pp. 17 7722-17655 655.
[65] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in International Conference Learning Representations (ICLR), 2015.
[66] S. Chen, Z. Huang, Q. Tao, and X. Huang, "Querynet: Attack by multi-identity surrogates,"
arXiv preprint arXiv:2105. 15010, 2021.
[67] K. He, X. Zhang. S. Ren, and Sun, "Deep residual learning for image recognition." " in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778.
[68] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, "Aggregated residual transformations for deep neural networks," in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1492-1500.
[69] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, Segg-perffrmance deep learning library," in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 8024 8035.
[70] S. Chen, F. He, X. Huang, and Zhang, "Relevance attack on detectors," Pattern Recognition, 108491, 2022.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the third paragraph in Sec.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the second paragraph in Sec. for potential impacts. Negative ones have not been identified since we aim at defense.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our contribu- tions are mostly empirical.
(b) Did you include complete proofs of all theoretical results? [N/A] Our contributions are mostly empirical.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] All the code would be released with detailed instructions and annotations. Core code of AAA is
shown in Appendix D The data are described in experimental setup in Sec. [5.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Details are presented Sec. 5.1 and Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Error bars and computation amount reported in Appendix[C]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Our devices are described Sec.
5.1 and the amount of computation is quantified in Appendix C
4. If you are using existing assets (e.g., e., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We state the source of DNNs in experimental setup in Sec. 5.1
(b) Did you mention the license of the assets? [Yes] Most used code assets under MIT License. Square attack is under BSD 3-Clause License. ImageNet under a custom research and non-commercial license.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] We would release our code as an asset.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Our empirical studies are based on public datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] Our used public datasets generally do not contain personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Our experiments involve no human subjects and crowdsourcing. (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Our experiments involve no human subjects or crowdsourcing.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Our experiments involve no human subjects or crowdsoureing.
15


Reviews:
[{"review_id": "yVnwY3jIY5v", "reviewer": "Reviewer_cULW", "summary": "The present work introduces a novel defense to confound the score-based query attacks (SQAs). Its main novelty includes three parts: (1) an effective and user-friendly post-processing module; (2) a novel adversarial attack on attackers (AAA) defense by slightly deviating the output scores; (3) AAA beats other prior defenses in terms of the effectiveness and robustness. Due to the impractical cost of prior defenses, AAA designs the post-processing method to improve efficiency and exerts a significant impact on accuracy.", "questions": "1. Interesting, the clean model performance with AAA seems a little higher than original clean model, for example, 94.84 vs 94.78 in CIFAR10. Can you provide more analysis about this? Similar results can also be found in baseline DENT.\n\n2. In Equation (4), does it missing abs for ceil(·)? when l_0=0, ceil(l_0/t=0)-1/2 = -1/2? and the result is -2?\n\n3. The idea of attacking on attacker is straightforward, what is the motivation in designing the specific operated shown in Equation (4) and (5). For example, why push l_t even lower than l_a by a margin of (l_0-l_a), and what is the detailed motivation in designing equation (4). I am just curious if there are any other operations following the same idea achieves better defense results.\n\n4. Have you studied the proposed method without optimization in Equation (6)?\n\n5. The adjusted loss is closely related with confidence calibration. Can you show some defense results, in comparison with some confidence calibration methods?", "limitations": "The discussion about optimizing Equation 6 seems missing. What is the number of optimization steps? The time consumed? etc. What are the results without optimization Equation (6).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The idea is sound and interesting. Instead of enhancing the robustness of deep learning models, this proposed method mislead and confuse the attackers to protect the DNN.\n+ Meanwhile, the method costs a little to improve efficiency, which makes defense more practical in real-world settings.\n+ the general design of the experiments is meaningful in helping to understand the role of AAA in outperforming the considered defense approaches.\n\nWeaknesses:\n- Some operations for example, the determination of specific operation, like Equation (4) is short of discussion. The detail is as follows.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H9oqtwVcw_", "reviewer": "Reviewer_uWUQ", "summary": "This paper proposes a post-processing defense against score-based black-box evasion attacks. This defense modifies the original logits so that the trend of the margin loss is reversed in each small interval of the attack, but the overall trend remains unchanged to preserve accurate prediction confidence. As a result, score-based attacks using the margin loss will optimize towards the opposite (hence non-adversarial) direction. Meanwhile, this defense does not hurt the model’s benign performance, and the calibration performance is preserved by solving a joint optimization problem. Experiments show that the proposed defense outperforms 8 previous defenses under 6 score-based attacks on CIFAR-10 and ImageNet.", "questions": "My current score is mainly based on the weaknesses outlined in the Quality section. I am willing to raise my score if the following concerns are adequately clarified or justified.\n* [Quality-Weakness-1] Please clarify if the knowledge of the attack’s period $t$ at L171 is practical. Specifically, please explain how the proposed defense can be implemented *independently of the attack’s code* with only the following inputs: (1) query image, (2) model, and (3) the defense’s hyper-parameters. If you need any other inputs, please clarify relevant assumptions and their practicability.\n* [Quality-Weakness-2] Please discuss if the defense would still work if the attackers revert their update logic in Eq. (3), without significantly modifying the current defense.", "limitations": "The primary limitation is summarized in the Significance section. While I appreciate the effort of motivating post-processing defenses against evasion attacks, I strongly recommend the authors carefully address these limitations in future versions of this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\n**Strengths**\n* This paper studies post-processing defenses, which are less explored in defending evasion attacks.\n* The proposed defense avoids hurting the model’s benign performance and computational overheads.\n\n**Weaknesses**\n* Not much in this perspective.\n\n### Quality\n\n**Strengths**\n* Good experiments. I appreciate the evaluation of 8 defenses, 6 attacks, and 2 datasets.\n* Good ablation study. Most hyper-parameters are supported by the ablation study.\n\n**Weaknesses**\n* **Impractical threat model.** While the attack’s threat model is well defined, the proposed defense implicitly assumes complete control of the attacker’s optimization process. **Specifically, the defense assumes unrealistic knowledge of the attack’s running iteration, i.e., the period $t$ defined at L171**. Note that all compared defenses in this paper did not have such an assumption. This knowledge is only realizable if given the following two assumptions. First, the attacker discloses the attack’s current number of iterations for each query. Second, the defender can extract the sequence of queries coming from the designated attacker (across all queries from all legitimate users and potentially other attackers). These two assumptions, however, are largely unrealistic or challenging. Since such knowledge is the base assumption of this defense at Eq. (4), I am not sure if the current defense would work if the assumption did not hold. **In particular, the defense’s code in Appendix D seems to work inside the attack, which is impossible in practice.**\n\n* **Lack of adaptive evaluation.** This paper does not discuss how the attacker could potentially modify their attacking procedure to evade the proposed defense, although some adaptive attack papers like [46] are cited. Currently, the proposed defense assumes a static (i.e., non-adaptive) attack that is unaware of the defense. Following the paper’s own motivation, is it possible that the attackers also revert their updating logic in Eq. (3) to evade the proposed defense? For example, now that the returned score exhibits a reverted trend, **would this defense still work if the attackers update their adversarial examples only when observing an increased margin loss?**\n\n### Clarity\n\n**Strengths**\n* Good motivation for post-processing defenses.\n* The proposed defense is easy to follow.\n\n**Weaknesses**\n* The notation in Eq. (1) is slightly confusing. At L117, the x is defined as a clean sample, but it is later used as a placeholder in Eq. (1) and (2).\n* The unsupervised margin loss in Eq. (2) can be simplified. Specifically, if the defender could use one query to obtain the ground-truth label (from the black-box model’s perspective), Eq. (2) can reduce to Eq. (1) and therefore simplify the notations.\n* At L123, it is suggested to add some (brief) background of how these attacks sample their queries $x_q$. This would be greatly helpful for readers to understand the defense. Currently, it seems that all attacks are compressed into one high-level idea, which might reduce the confidence in understanding how the defense would work if the attack changes adaptively.\n* At L171 and L193, the notation $z_0$ is not defined until algorithm 1.\n\n### Significance\n\nI appreciate the effort of motivating post-processing defenses against evasion attacks, but the proposed defense considers a somewhat unrealistic threat model, which is different from the defenses compared in this paper. Moreover, this paper assumes a static non-adaptive attacker, which might be easily broken by adaptive attacks based on my assessment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2TwoWG0-FK", "reviewer": "Reviewer_AAQp", "summary": "This paper proposes an adversarial defense against black-box score-based query attack. Specifically, the AAA defense proposed in this paper can mislead SQA methods to simulate the direction of gradients by post-processing the logits output by the target model. The method is validated on cifar-10 and Imagenet datasets", "questions": "There seems to be some problems in the penultimate row of Table 2. The performance of 69.3 of RND defense under NES attack should be bolded, not 68.51 of AAA.", "limitations": "From the perspective of experimental performance, compared with other defense methods, AAA is still not significantly improved in terms of affecting SQAs’ attack effect.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths and weaknesses of this paper are summarized as follows:\n\nThe main strength of this paper is that it presents a relatively new approach of adversarial defense. The defense against SQA is rarely covered by previous work, and the AAA method proposed in this paper may bring inspiration to the subsequent score-based attacks and related defenses\n\nThe weaknesses of this paper are twofold. The first aspect is the motivation of the method. The AAA method proposed in this paper is in fact specially targeted for SQA. While SQA belongs to a class of query-based attacks in black-box attack methods in adversarial attacks. That is to say, the effectiveness of AAA is based on at least three constraints: the attacker cannot obtain the details of the target model, the attacker does not use a substitute model but can only query the target model, and the attacker can obtain the confidence information of the target model. This situation implies one-way transparency from the defender to the attacker, not the other way around. This setting is very unrealistic, which makes the motivation somewaht weak. \n\nIn addition, even from the perspective of exploring the robustness of deep neural networks, the method proposed in this paper to specifically deal with one class of attacks cannot improve the robustness of the model itself, or as the authors put it in the paper, worst-case robustness. However, improving the robustness of the model without considering the attack method is the direction that should be advocated in the field of adversarial defense [1]. Otherwise, this is only an arms race between adversarial attacks and defenses.\n\n[1] Athalye, Anish, Nicholas Carlini, and David Wagner. \"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.\" International conference on machine learning. PMLR, 2018.\n\nSecondly, because this paper defends against SQA, there is a realistic problem that other query-based attacks like Boundary Attack require less information than SQA (no confidence information, just hard labels). Can the method proposed in this paper effectively defense against such attacks? If so, AAA must changes the hard label, will this affect the availability of the target model? If not, can the attacker directly use decision-based attacks to bypass this defense?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0Z1wxB5KGXr", "reviewer": "Reviewer_78Et", "summary": "This paper proposes a defense called Adversarial Attack on Attackers (AAA) that is designed specifically towards mitigating score-based query attacks (SQAs). AAA is a post-processing attack that attempts to modify the logits loss curve to locally point in the incorrect attack direction in a periodic fashion, which steers SQAs away from a true adversarial attack. As such, AAA takes advantage of typical SQA behavior of sampling nearby loss value changes to find a local direction to optimize in and steers them in the wrong way. AAA could then be added on to other models as a way to deter and prevent these attacks in more real-world scenarios where such black-box access is present. The authors evaluate AAA over several baseline defenses on CIFAR-10 and ImageNet on various SQAs, finding AAA to be effective at improving robustness.", "questions": "This paper provides an interesting new take on adversarial robustness. I like the direction of targeting realistic attacks such as SQAs at a higher level than simply robustifying the model to all possible attack inputs. Indeed, such a post-processing approach such as AAA to deter attacks that that level is appealing and I believe that further research with this philosophy would be interesting. The experiments presented provide a useful understanding of the characteristics of AAA and show strong performance on a variety of settings. \n\nI have two main concerns. The primary one is whether or not SQAs could possibly be adapted to attack AAA. In Section 5.5 “Hyper-parameters of AAA” and Figure 4, it is clear that the choice of the attractor interval t can greatly influence the adversarial accuracy as the attack success rate would seemingly come down to the attack’s ability to jump out of an interval. While this is an interesting study to see how the choice of t impacts the performance of a static SQA, could SQAs be modified to look at a wider interval? \n\nFor example, perhaps an SQA could be modified to maintain two search branches, one that goes in the default direction implied by the loss changes and another that continually goes in the opposite for a limited time, and then if the one that goes in the opposite direction eventually drops by jumping out of the interval, an attacker could determine that this was actually the correct direction to go. Since the underlying model is not inherently more robust, a process that finds a jump may still be able to eventually break the model, albeit with more queries.\n\nSecondarily, how much does it cost to run AAA? It would appear to be a fairly lightweight process, but do you know how much slower this would make the model?\n\nGiven this, I still like the general direction of the paper and if my concerns are addressed I will consider raising my score.\n\n-----------------------------------------------\n\nIn the rebuttal, the authors provided results on the proposed bidirectional adaptive attack as well as a new sine function to use, as well as runtime results, alleviating these concerns.", "limitations": "The authors have adequately addressed limitations and impact. The authors discuss AAA in the context of real-world systems such as autonomous driving where AAA could be added with pre-trained models to increase the reliability of such systems. The authors also acknowledge the limitation that AAA is designed to target SQAs specifically, which is a useful defense approach, but does mean that worst-case robustness under white-box settings is not improved.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Interesting new defense approach that targets a realistic deployment of ML models by attacking the query process\n- Seems like a lightweight defense with a low cost\n- Preserves natural accuracy\n\nWeaknesses\n- Unclear if SQAs could possibly be adapted to re-attack AAA\n- Unknown what the exact runtime cost is\n- Does not increase underlying model robustness", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept