Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
Authors: Dequan Wang
An Ju
Evan Shelhamer
David Wagner
Trevor Darrell
Venue: NeurIPS 2021 Submitted (2021)
Keywords: adversarial
robustness
defense
dynamic

Abstract: Adversarial attacks optimize against models to defeat defenses. Existing defenses are static, and stay the same once trained, even while attacks change. We argue that models should ﬁght back, and optimize their defenses against attacks at test time. We propose dynamic defenses, to adapt the model and input during testing, by defensive entropy minimization (dent). Dent alters testing, but not training, for compatibility with existing models and train-time defenses. Dent improves the robustness of adversarially-trained defenses and nominally-trained models against white-box, black-box, and adaptive attacks on CIFAR-10/100 and ImageNet. In particular, dent boosts state-of-the-art defenses by 20+ points absolute against AutoAttack on CIFAR-10 at $\epsilon_\infty = 8/255$.

Fighting Gradients with Gradients:
Dynamic Defenses against Adversarial Attacks
Anonymous Author(s)
Affiliation
Address email
Abstract
Adversarial attacks optimize against models to defeat defenses. Existing defenses are static, and stay the same once trained, even while attacks change. We argue that models should fight back, and optimize their defenses against attacks at test time. We propose dynamic defenses, to adapt the model and input during testing, by defensive entropy minimization (dent). Dent alters testing, but not training, for compatibility with existing models and train-tima defenses. Dent improves the robustness of adversarially-traine defenses and nominally-trained models against white-box. black-box, and adaptive attacks on CIFAR- 10/100 and ImageNet. In particular, dent boosts state-of-the-an defenses by 20+ points absolute against AutoAttack on CIFAR-10 at 6 8/255.
2
3
10
Introduction: Attack, Defend, and Then?
11
Deep networks are vulnerable to adversarial attacks: input perturbations that alter natural data to cause errors exploit predictions (54). As deep networks are deployed in real systems, these attacks 14 are real threats (63] and so defenses are needed. The challenge is that every new defense is followed
15 by a new attack, loop (56). The strongest attacks, armed with gradient optimization, update to
16 circumvent defenses that do not. Such iterative attacks form an even tighter loop to ensnare defenses.
17 In cat and mouse game, the mouse must keep moving survive.
Current defenses, deterministic or stochastic, stand still: once trained, they are static and do not adapt during testing. Adversarial training [18] 30 learns from attacks during training, but cannot learn from test data. Stochastic defenses alter the network [IT] or input (20] 7 but their randomness is independent of test data. Static defenses do not adapt, and so they may fail as attacks update.
19
20
21
22
Our dynamic defense fights adversarial updates with defensive updates by adapting during testing (Figure []. In fact, our defense updates every input, whether natural adversarial. Our defense objective is entropy minimization, to maximize model confidence, so we call our method dent for defensive entropy. Our updates rely on gradients and batch statistics, inspired by test-time adaptation
24
25
26
58) In pivoting from training to testing, dent able to keep changing, so the attacker never hits the same defense twice. Dent has the last move advantage, as its update always follows each attack.
approaches
53
43
28
29
27
28
Dent connects adversarial defense and domain adaptation, which share an interest in the sensitivity of 30 deep networks to input shifts. Just as models fail on adversarial attacks, they fail on natural shifts like corruptions. Adversarial data is a particularly hard shift, as evidenced by the need for more parameters and optimization for adversarial training (30]. and its negative side effect of reducing
31
32
Submitted 35th Conference Neural Information Processing Systems (NeurIPS 2021). Do distribute.

---

attacker
81-1
8
0 A
8 ++-
0+4'*
defender
(a) train
(b) static defense
(c) dynamic defense
Figure 1:
Attacks optimize the input ò against the model 0. Adversarial training optimizes 0 for defense (a), but attacks update during testing while & does not (b). Our dynamic defense improves during testing (c), so the attack cannot hit the same defense twice.
robustness by adapting
Faced with these difficulties, we turn to adaptation, and change our 34 focus to testing, rather than training more still.
accuracy on natural data 152
65
35
Experiments evaluate dent against white-box attacks (APGD, FAB), black-box attack (Square), and adaptive attacks that are aware of its updates. Dent boosts state-of- ttt-heeartt adversarial training defenses on CIFAR-10 by 20+ points against AutoAttack iteration, parameterization, and batch size. Our code is included in the supplement.
36
37
9 at c = 8/255. Ablations inspect the effects of
39
Our contributions
We highlight an opportunity for dynamic defense: the last move advantage.
We propose the first fully test-time dynamic defense: dent adapts both the model and input during testing without needing alter training.
43
Dent augments state-of-ther of-the-art adversarial training methods, improving robustness by 30% relative, and tops the AutoAttack leaderboard by 15++ points.
44
45
We devise two adaptive attacks against dent: denying updates and mixing batches.
46
2
Related Work
47
Adversarial Defense For adaptive adversaries, which change in response to defenses, is natural consider dynamic defenses, which adapt in turn. Evans al. [14 explain dynamic defenses promising in principle but caution they may not be effective in practice. Their analysis concerns randomized defenses, which do change, but their randomization does not adapt to the input. We argue for dynamic defenses that depend on the input to keep adapting along with the attacks. Goodfellow supports dynamic defenses for similar reasons, but does not develop a specific defense. We demonstrate the first defense to optimize the model and input during testing for improved robustness.
50
51
52
53
Most defenses for deep learning focus first-order adversaries [8 30 that equipped with gradient optimization but constrained by f--norm bounds. Adversarial training and randomization 56 are the most effective defenses against such attacks, but are nevertheless limited, as they are fixed during testing. Adversarial training [88 30 trains on attacks, but a different or stronger adversary
57
58 (by norm or bound) can overcome the trained defense (6.55). Randomizing the input [37 32
59 or network [TT] requires the adversary optimize in expectation [] but can still fail with more
60 iterations. Furthermore, these defenses gain adversarial robustness by sacrificing accuracy on natural
61 data. Dent adapts during testing to defend against various attacks without more harm to natural accuracy.
63
Generative, self-supervised, and certified defenses try to align testing with training but are still
84 static. Generative defenses optimize the input w.r.t. autoregressive [50] GAN [42). or energy [23
65 models, but the models do not adapt, and may be attacked by approximating their gradients
86
Self-supervised defenses optimize the input w.r.t. auxiliary tasks (4), but again the models do not
adapt. Certified defenses [7] [66] guarantee robustness within their training scope, but are limited
to small perturbations by specific types of attacker during testing. Changing data distributions or

---

aduptive optimization
flg(z 2):6+ A)
entropy
input
model
I
g + + Σ)
H
(a) adversarial sample
(b) input transform
(c) input model adaptation
Dent adapts the model and input to minimize the entropy of the prediction (Ú)) The model f is adapted by a constrained update A to the parameters 0. The input is adapted by smoothing g with parameters Σ. Dent updates batch-by-batch during testing.
Figure 2:
69
adversaries requires re-training all of these defenses. Dent adapts during testing, without requiring (re- treaniraining, and is the only method to update the model itself against attack.
70
71
Domain Adaptation Domain adaptation mitigates input shifts between the source (train) and target (test) to maintain model accuracy [34 41 Adversarial attacks are such shift, and adversarial error is related to natural generalization error
72
73
How then can adaptation inform dynamic defense? Train-time adaptation is static, like adversarial training, with the same issues of capacity, optimization, and re-computation when the data/adversary changes. We instead turn test-time adaptation methods.
74
75
76
77
Test-time adaptation keeps updating the model as the data changes. Model parameters and statistics can be updated by self-supervision (53), normalization [43] and entropy minimization [58]. These methods improve robustness to natural corruptions [22]. but their effect on adversarial perturbations not known. We base our defense on entropy minimization as it enables optimization during testing without altering model architecture or training (as needed for self-supervision) For defense, we (I) extend the parameterization of adaptation with model and input transformations, (2) optimize for additional iterations, and (3) investigate usage data that is adversarial, natural, mixed. We the first to report test-time model adaptation improves robustness to adversarial perturbations.
78
79
80
81
82
84
85
Dynamic Inference A dynamic model conditionally changes inference for each input, while a static model unconditionally fixes inference for all inputs. There are various dynamic inference techniques, with equally varied goals, such as expressivity with more parameters or efficiency with less computation. All static models alike; each dynamic model is dynamic in its own way.
86
87
89
Selection techniques learn to choose subset of components I 57 Halting techniques learn to continue or end computation [19 (59] Mixing techniques learn to combine parameters 44 33 62 Implicit techniques learn to iteratively update 6 While these methods learn to adapt during training, our method keeps adapting by directly optimizing during testing.
90
91
92
Dynamic Defense by Test-Time Adaptation
93
Adversarial attacks optimize against defenses at test time, defenses should fight back, and counter- optimize against attacks. Defensive entropy minimization (dent) does exactly this for dynamic defense by test-time adaptation.
97 In contrast to many existing defenses, dent alters testing, but not training. Dent only needs differen- 98 tiable parameters for gradient optimization and probabilistic predictions for entropy measurement. 39 As such, it applies to both adversarially-traiee and nominally-trained models.
3.1
Preliminaries Attacks and Defenses
101
Let E IRd and {1, (1....,C) C} be an input sample and its corresponding ground truth. Given model (+:0):: Rd - RC parameterized by 0, the goal of the adversary is to craft a perturbation
102
5 E Rd such that the perturbed input ă = x + 5 causes a prediction error f(x + o; 0) + y-
103
104
targeted attack aims for specific prediction of 1/. while untargeted attack seeks any incorrect prediction. The perturbation 8 is constrained by choice of Ep norm and threshold {8 Rd ||,||, < <).. We consider the two most popular norms for adversarial attacks: foc and lz.

---

faat at- pt 2
8
far
8':0)
fla(2 + or :'0
(a) static
(b) dynamic
Figure 3: The adversary optimizes its attacks 51 against the model f. Static defenses (left) do not adapt, and are vulnerable to persistent, iterative attacks. Our dynamic defenses (right) do adapt, and update their parameters A, Σ each time the adversary updates its attack 6.
107
Adversarial training is a standard defense, formulated by Madry et al. [30 as saddle point problem,
argmin (2,))) max [( + 8;0),
which the model minimizes and the adversary maximizes with respect to the loss L(ū, y), such as cross-entropy for classification. The adversary iteratively optimizes o by projected gradient descent (PGD), standard algorithm for constrained optimization, for each step t via
110
8t 1pp(5'11 + (e- + 81: ;)),))))
for projection IIp onto the norm ball for lp step size hyperparameter and random initialization
112 80. The model optimizes 0 against å to minimize the loss of its predictions on perturbed inputs. This
113 is accomplished by augmenting the training set with adversarial inputs from PGD attack.
114
Adversarial training is state-of-the-art, but static. Dynamic defenses offer to augment its robustness.
115
3.2
Defensive Entropy Minimization
Defensive entropy minimization (dent) counters attack updates with defense updates. While adver- saries optimize to cross decision boundaries, entropy minimization optimizes to distance predictions from decision boundaries, interfering with attacks. As the adversary optimizes its perturbation 6, dent optimizes its adaptation A, Σ. Figure shows dent's model (A) and input (E) updates.
117
118
119
120
Dent is dynamic because both A, Σ depend on the testing data, whether natural adversarial 8. On the contrary, static defenses depend only training data through the model parameters A.
Figure 3 contrasts static and dynamic defenses across the steps of attack optimization.
123
Entropy Objective Test-time optimization requires an unsupervised objective. Following tent [58],
124
we adopt entropy minimization as our adaptation objective. Specifically, our defense objective is to minimize the Shannon entropy (ÿ) of the model prediction during testing ÿ = f(x; #)) for the
125
126
probability ÿ. of class
H()) = Σ
p(gc) log p(fc)
CEI....
127 Adaptation Parameters Dent adapts the model by A and input by E (Figure D). For the model, dent adapts affine scale y and shift parameters by gradient updates and adapts mean 4 and variance o² statistics by estimation. These However, they are effective for conditioning a model on changes in the For the input, dent updates Gaussian smoothing g by gradient updates of the parameter Σ. while adjusting the filter size for efficiency [48] This controls the degree of smoothing dynamically, unlike defense by static smoothing
128
129
small portion of the full model parameters in only the batch
130
normalization layers 25
task (33] data 43
58
132
139
134
In standard models the scale y and shift 3 parameters are shared across inputs, and so adaptation updates batch-wise. For further adaptation, dent can update sample-wise, with different affine parameters for each input. In this way, it adapts more than prior test-time adaptation methods with
135
137
batch-wise parameters [5 43
138
Our model and input parameters are differentiable, so end-to-end optimization coordinates them against attacks as layered defenses. This coordination inspired by CyCADA 24), for domain adaptation, but dent differs in its purpose and its unified loss. CyCADA also optimizes input and model transformations but does in parallel with separate losses. Our defensive optimization is joint and shares the same loss.
139
140

---

Table 1:
Dent boosts the robustness of adversarial training on CIFAR-I0 against AutoAttack. Adver- sarial training is static, but dent is dynamic, and adapts during testing. Dent adapts batch- wise, while dent+ adapts sample-wis wise, surpassing the state-of-tie oate--of-the-rt for static defense at coobustench..ithubeee
ACCURACY(%)
NATURAL
ADVERSARIAL DENT
STATIC
DENT+
EOc 8/255
CARMON ET AL.
5
89.6
59.5
74.7
82.3
SEHWAG ET AL..
44
84.4
54.4
61.2
75.2
WONG ET AL.
60
83.3
43.2
52.3
71.8
DING AL.
88.0
41.4
47.6
64.4
62 = 0.5
SEHWAG ET AL.
44
89.5
73.4
77.8
85.7
RICE ET AL. 38
88.7
67.7
69.7
81.3
RONY ET AL.
39
89.1
66.4
73.4
85.3
DING ET AL.
12
88.0
66.1
70.3
82.8
143
Update Algorithm In summary, when the adversary attacks with perturbation 8¹, our dynamic defense reacts with Σ A'. The parameters the model f and smoothing g are updated by argming,4 (f + 5; E); 0 + A)) through test-time optimization. At each step, dent estimates the normalization statistics μ4, and then updates the parameters 7, [3, E. by the gradient of entropy minimization. Figure 3 contrasts static defenses and dynamic defenses that update like dent.
144
145
148
Dent adapts on batches rather than samples. Batch-wise adaptation stabilizes optimization for entropy minimization. The defense parameters reset between batches.
149
150
Discussion The purpose of a dynamic defense is move when the adversary moves. When the adversary submits attack 8", the defense counters with A' In this way, the defense has the last move, and therefore an advantage.
153
Our dynamic defense changes the model, and therefore its gradients, but differs from gradient obfuscation 3 Our defense does not rely on (1) shattered gradients, as the update does not cause non-iifferentiabiiity or numerical instability; (2) stochastic gradients, as the update deterministic given the input, model, and prior updates; improves robustness with even single step (although more steps are empirically better).
154
155
(3) exploding/vanishing gradients, as the update
158
Dent forces the attack to rely on a stale gradient, as 6¹ follows 1-1, while the model adapts by A
4
Experiments
159
160
evaluate dent against white-box, black-box, and adaptive attacks with variety of static defenses and datasets. For attacks, we choose the AutoAttack (9] benchmark, which includes four attack types spanning white-boxx/grade
181
and black-box/query attacks. For static defenses, choose strong and recent adversarial training methods, and we also experiment with nominally trained models. For datasets, we evaluate dent CIFAR- 10/CIFAR-100 100 (27) as they are popular datasets for adversarial robustness, and ImageNet 40 as it is a large-scale dataset.
184
185
186
We ablate the choice of model/input adaptation, parameterization, and the number of updates.
187
4.1
Setup
188
Metrics We score natural accuracy on the regular test data v and adversarial accuracy on the perturbed
189
test data r + Each is measured as percentage accuracy (higher is better). We report the worst-case adversarial accuracy across attacks.
170
Test-time Optimization We optimize batch-wise A (dent) and sample-wise A (dent+). Dent updates by Adam 226 with learning rate 0.001. Dentt updates by AdaMod [3] with learning rate 0.006. Σ
5

---

173
updates use learning rate 0.25. All updates use batch size 128 and no weight decay. Dent+ regularizes updates by information maximization [16] 29 We tuned update hyperparameters against PGD) attacks. Please see the code for exact settings.
174
175
178
Architecture For comparison with existing defenses, we keep the architecture and training the same, and simply load the public reference models provided by RobustBench [TO]. For analysis and ablation experiments, we define residual net with 26 layers and width multiplier of 4 (ResNet-26-4) [4].6].
178
179
following prior work on adaptation 53 58
180
4.2
Attack Types & Threat Model
We evaluate standard white-box and black-box attacks with ddiversarially-triinee models (Section 4.3 and nominally-traines models (Section 4.4. as well as dent-specific adaptive attacks (Section 4.5
182
183
We primarily evaluate against AutoAttack ensemble of:
1. APGD-CE 30 91, untargeted white-box attack by cross-entropy,
2. APGD-DLR 9 targeted white-box attack with shift and scale invariant loss,
188
3. FAB 8 a targeted white-box attack for minimum-norm perturbation,
187
4. Square Attack [2], an untargeted black-box attack with square-shaped updates.
188
These attacks are cumulative, so a defense is only successful if it holds against each type. Following convention, we evaluate for attacks with 6xc = 8/255 and lg attacks with 62 = 0.5. This is the standard evaluation adopted by the popular RobustBench benchmark [TO].
189
190
We devise and experiment with two adaptive attacks against dent and its dynamic updates. The first interferes with adaptation by denying updates: it optimizes offline against 0 without A, E updates. The second interferes with adaptation by mixing data: it combines adversarial data and natural data in the same batch. Both are specific to dent to complement our general evaluation by AutoAttack.
193
194
195
These attacks fall under the usual white-box threat model. The adversary has full access to the classifier, including its architecture and parameters, and the defense, such as dent's adaptation parameters and statistics. With this access the adversary chooses attack for each input, but it cannot choose the inputs (the test set is fixed).
196
199
We include one additional requirement: dent assumes access to test batches rather than individual test samples. While independent, sample-wise defense is ideal for simplicity and latency, batch processing not impractical. For example, cloud deployments of deep learning batch inputs for throughput efficiency, and large-scale systems handle many inputs per unit time (3I.
200
201
202
203
The supplementary material covers more attacks, including AutoAttack Plus and Boundary, to confirm that AutoAttack is a sufficient measure of robustness.
204
205
4.3
Dynamic Defense of Adversarial Training
208
We extend static adversarial training defenses with dynamic updates by dent. Compared to nominal training, adversarial training achieves higher adversarial accuracy but lower natural accuracy. The purpose of dent is to improve adversarial accuracy without further harming natural accuracy.
207
208
Dent improves oftte-of-the-at defenses. Table]]shows state-of-the-art adversarial training defenses
209
5 44 39| 38 60 [2] with and without dynamic defense by dent. Note that dent does not specialize to the choice of norm or bound, unlike adversarial training, but instead adapts to each attack during testing. In each case, dent significantly improves adversarial accuracy and maintains natural accuracy.
212
213
Dent updates batch-wise for 30 steps. Dent+ more robust fewer steps by sample-wisg wise adaptation. With sample-wise (7, 9) parameters, dent+ needs only six steps to reach an adversarial accuracy within 90% of the natural accuracy. These experiments only include model adaptation of A, without input adaptation of E, as we found it unnecessary when combined with ad adversarial training.
214

---

Table 2: AutoAttack includes four attack types, and dent improves robustness to each on CIFAR-10 against fo attacks. We evaluate without dent (-) and with dent (+).
ACCURACY
APGD-CE
APGD-DLR
FAB
SQUARE
WONG ET
45.9
57.6
43.2
52.3
43.2
52.3
43.2
52.3
DING ET AL.
12
50.1
60.2
41.6
48.0
41.5
47.7
41.4
47.6
Table 3: Ablation of model adaptation (A), input adaptation (2), and steps the accuracy of nominally-traine model with dent.
ADVERSARIAL.
>
STEP
TIME
NATURAL
1.5
(2 0.2
255
NONE
1.0x
95.6
8.8
9.2
NONE
3.6x
95.6
15.0
13.5
STAT.
1.0x
86.2
25.8
23.6
STAT.
3.6x
86.3
27.5
24.4
STAT.
10
25.9x
86.3
37.6
30.9
DYNA.
10
26.1x
92.5
45.4
36.5
217
Dent helps across attack types. Table evaluates dent against each attack the AutoAttack ensemble. Dent improves robustness to each attack type. We report the worst case across these types the remainder of our experiments.
218
219
Dent helps across datasets and architectures. We experiment ImageNet to check scalability. We evaluate the defense of Wong et al. [60], one of few defenses that scales to this dataset, against strong L PPG attacks with 30 iterations, step size of 0.1, and five random starts. Dent improves the adversarial accuracy by 14 points against PGD at c = 4/255 and natural accuracy by 23 points.
222
223
Table ?? the supplement confirms improvement across more defenses, architectures, and datasets.
225
4.4
Dynamic Defense of Nominal Training
226
Dent improves the adversarial accuracy of off-the-shelf, nominally trained models. As dent does not assume adversarial training, it can apply to various models at test time.
227
For nominal training, exactly follow the CIFAR reference training in pycls 35 36 with ResNet-
26-4/ 26-4/ResNet-32-10 architectures. Briefly, train by stochastic gradient descent (SGD) for 200 epochs with batch size 128, learning rate 0.1 and decay 0.0005, momentum 0.9, and a half-period cosine schedule.
231
232
We evaluate against &o and & AutoAttack attacks on CIFAR- 10. As the nominally-tran ttained models have static defense, constrain the adversaries smaller perturbations.
234
Dent defends nominally-trained models. Table 3 inspects how each part of dent affects adversarial accuracy and natural accuracy. When applying dent to nominally-trained models, model adaptation through A further helped by input adaptation through Σ. In just a single step, the A update improves adversarial accuracy without affecting natural accuracy. from 8.8% to 15.0% against foo
235
236
237
238
attacks with just single step. With steps, and Σ adaptation, dent improves the model's adversarial accuracy to 45.4% against &o attacks and 36.5% against fa attacks. In total, dent boosts f.x and &2 adversarial accuracy by almost 40 and 30 points while only sacrificing 3 points of natural accuracy. Dent delivers this boost at test-time, without re- re-training.
242
Input adaptation helps preserve natural accuracy. Gaussian smoothing significantly improves adversarial accuracy. This agrees with prior work on denoising by optimization 20 or randomized smoothing (7). Tuned fixed hyperparameter, smoothing helps adversarial accuracy but hurts natural accuracy. Optimized end-to-end our dynamic smoothing reduces the natural accuracy gap. On natural data, the learned Σ for the blur decreases to approximate the identity transformation.
243

---

Table 5: Adaptive attack by mixing adversar- ial and natural data. We report the adversarial accuracy mixed batches, from low to high amounts of adversarial data. Dent improves on adversarial training (43.8%) across mixing proportions within 10 steps.
Table 4: Adaptive attack by denying updates. We transfer attacks from static models to dent and then evaluate nominal and adversarial training (30) against Lo and Eg AutoAttack. Attacks break the static models (static-static), but fail to transfer to our dynamic defense (static-dent).
STEP
10%
25%
50%
75%
90%
NOMINAL:
ADVERSARIAL.
c2=0.2
Ex
255
8
625
62.4
43.4
43.2
49.6
44.0
48.7
44.2
47.6
43.8
STATIC- -TAAIC
11.6
11.0
42.0
44.1
STATIC-DENT
82.5
81.6
50.0
50.2
V
10
54.9
47.6
41.7
47.7
41.4
49.7
43.2
44.1
50.6
50.9
44.7
4.5
Adaptive Attacks on Dent Updates
248
We adaptively attack dent through its use of adaptation by (1) denying updates and (2) mixing batches. To deny updates, we attack the static model offline by optimizing against 9 without A, E updates, then submit this attack to dent. This attempts short circuit adaptation by disrupting the first update with sufficiently strong perturbation. To mix batches, mix adversarial and natural data in the same batch. This attempts to prevent adaptation by aligning batch statistics with natural data.
249
250
252
253
Denying Updates The aim of this attack is to defeat adaptation on the first move, before dent can update to counter it. We optimize against the static model alone to prevent defensive optimization until adversarial optimization is complete. Under this attack, the input to dent is the final perturbation derived by adversarial attack against the static model.
254
255
257
We examine whether these offline perturbations can disrupt adaptation. Table 4 shows that dent can still defend against this attack. This suggests that updating, and having the last move, remains an advantage for our dynamic defense.
258
259
Mixing Batches
Dent adapts batch- wise, with the underlying assumption that one shared transfor- mation can defend the whole batch. We challenge this assumption by evaluating mixed batches of adversarial and natural data. In Table we vary the ratio of adversarial and natural data in each batch and measure accuracy on the adversarial portion.
262
263
264
At the extreme, we consider an adaptive attack with only one adversarial input per batch. Specifically, batch one adversarial input with 15 natural inputs randomly chosen from the test set. This adaptive attack aims to reduce adaptation by the dynamic defense, natural inputs do not need adaptation.
265
257
Dent is generally robust to batch mixing, and improves over adversarial training in 10 steps or less.
288
4.6
Ablations & Analysis
More updates deliver more defense. The number of steps balance defense and computation. Ta that more steps offer stronger defense for both dent and dent+. However, more steps do nevertheless require more computation: ten-step optimization takes 25.9x more operations than the static model (Table 3 As plus, dent+ is not only more robust, but also more efficient in needing fewer steps. Note that the computational difference between dent and dent+ is negligible, as the adaptation parameters are such a small fraction of the model.
271
272
273
274
Model adaptation updates depend on the attack type. Dent adapts by adjusting normalization statistics and affine transformation parameters. Dent can fix or update the normalization statisties
276
277
(n, σ) by using static training statistics (x) or dynamic testing statistics (v): Dent can fix or update
the affine parameters (7, 3) by not taking gradients (x) or applying gradient updates (v). Table 7
278
279
compares each combination: affine updates always help, but both updates together hurt &g robustness.
Batch size We analyze dent's sensitivity to batch size and focus small batch sizes. Some real-world tasks, such autonomous driving, naturally provide small batch of inputs (from consecutive video
8

---

Table 6: Dynamic defenses can trade computation and adaptation. More steps are more robust on CIFAR- with &or AutoAttack. Dent+ reaches higher adversarial accuracy fewer steps.
STEPS
DENT
20
40
CARMON ET AL. 5
59.5
68.3
74.7
76.
WONG
660
43.2
48.2
52.3
55.1
DING 12
41.4
45.4
47.6
48.7
DENT+
0
DING ET AL. 12
41.4
46.5
57.7
64.4
Table 7: Ablation of model adaptation with and without normalization statistics (μ, σ)) and affine parameters (y, B) updates.
Table 8: Sensitivity analysis of batch size and adver- sarial accuracy with dent. With static batch statis- tics ( (x), small batch sizes are better. With dynamic batch statistics (v), small batch sizes are worse.
ACCURACY(S))
ADVERSARIAL
7,8
255
255
6=50.
TYPE
4
8
16
32
64
8.8
9.2
43.8
47.3
x
NAT.
85.9
86.0
85.9
85.9
86.1
86.
86.2
ADV.
70.4
69.5
67.8
65.3
61.9
58.6
55.1
16.8
16.2
49.9
57.3
NAT.
11.1
68.1
76.3
80.9
83.4
84.9
85.8
15.2
53.0
ADV
55.3
54.4
282
frames or various cameras, for example), and so we confirm that dent can maintain robustness on such small batches. Table 8 varies batch sizes to check dent's natural and adversarial accuracy.
283
Discussion
284
285
In advocating for dynamic defenses, we hope that test-time updates can help level the field for attacks and defenses. Our proposed defensive entropy method takes first step by countering adversarial optimization with defensive optimization needed for the back-an mack-and-forld iteration of attacks and defenses, the cost of defense scales with the cost of attack, and some use cases may prefer slow and strong to fast and wrong.
286
the model and input. While more test-time computation
290
Limitations Dent depends on batches to adapt, especially for fully test-time defense without ad- versarial training. It also relies on a particular choice of model and input parameters. A different objective could possibly lessen its dependence on batch size and reliance constrained updates. More generally, dynamic defenses may present difficulties for certification or deployment, they could drift. Along with how to update, improved defenses could investigate when reset, how to batch inputs for joint optimization.
291
292
295
296
Benchmarking Standardized benchmarking, by AutoAttack and RobustBBncc example, drives progress by competition and empirical corroboration. Dent brings adversarial accuracy on their benchmark within 90% of natural accuracy for three of the most accurate methods tested [5] 61 [2]. This is encouraging, but more research is needed fully characterize dynamic defenses like dent. However, RobustBench is designed for static defenses, and disqualifies dent by its rule against test-time optimization. Continued progress could depend on a new benchmark to standardize rules for how attacks and defenses alike may adapt.
297
298
302
303
By fighting gradients with gradients, dent shows the potential for dynamic defenses to update and counter adversarial attacks. The next steps-by attacks and defenses- -iill tell.

---

References
305
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural module networks. In CVPR. 2016.
307
[2] Andriushchenko, M., Croce, F,, Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial attack via random search. In ECCV, 2020.
309
[3] Athalye, A., Carlini, N., and Wagner. D. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In ICML, 2018.
[4] Bai, S., Kolter, J. 7., and Koltun, V. Deep equilibrium models. In NewrIPS, 2020.
312
[5] Carmon, Y., Raghunathan, A., Schmidt, L.,, Duchi, J. C., and Liang, P. S. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
313
314
[6] Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. In NeurlPS, 2018.
316
[7] Cohen, J. Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing. In JCML, 2019.
317
318
[8] Croce, F. and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In ICML, 2020.
320
[9] Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In ICMI, 2020.
322 [10] Croce, F., Andriushchenko, M., Schwag, V., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: standardized adversarial robustness benehmark. arXiv preprint arXiv:2010.09670, 2020.
324
[1I] Dhillon, G. S.. Azizzadennehell, K., Bernstein, J. D., Kossaifi. J.. Khanna, A., Lipton, Z. C., and Anandku- mar, A. Stochastic activation pruning for robust adversarial defense. In ICLR, 2018.
326
[12] Ding, G. W., Sharma, Y., Lui, K. Y. C., and Huang, R. Mma training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
328
[13] Ding, J.. Ren, X., Luo, R., and Sun, X. An adaptive and momental bound method for stochastic learning. arXiv preprint /910. 12249, 2019.
330
[14] Evans, D., Beuyen-TTunay A., and Knight, J. Effectiveness of moving target defenses. In Moving target defense, pp. 29 48. Springer, 2011.
332 [15] Gilmer, J.. Ford, N., Carlini, N., and Cubuk, E. Adversarial examples are natural consequence of test error in noise. In ICML, 2019.
334 [16]
Gomes, R., Krause, A., and Perona, P. Discriminative clustering by regularized information maximization. In NeurlPS, 2010.
336
[17] Goodfellow, I. A research agenda: Dynamic models to defend against correlated attacks. arXiv preprint arXiv:1903.0 06293,62933.2 2019.
338
[18] Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arX6v:1412.6572, 2014.
[19] Graves, A. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv: 1603.08980 2016.
342
[20] Guo, C., Rana, M., Cisse, M., and van der Maaten, L. Countering adversarial images using input tranffomattios.. In ICLR, 2018.
[21] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
345
[22] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
347
[23] Hill, M., Mitchell, J. C., and Zhu, S.-C. Stochastic security: Adversarial defense using long-run dynamics of energy- based models. In ICLR, 2021. Utt/ http::///ppprrrreeeeeee//rooocca
349
[24] Hoffman, J., Tzeng, E., Park, T., Zhu, J.- Isola, P., Saenko, K., Efros, A., and Darrell, T. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
10

---

351
[25] loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
353
[26]
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In ICLRR, 2015.
354
[27] Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
355
356
[28] Liang, J. Hu, D., and Feng, Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
357
[29] Liang, J.. Hu. D.. Wang. Y., He, R., and Feng. J. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. arXiv preprint arXiv:2012.07297, 2020.
359
360
[30]
Madry, A., Makelov, A., Schmidt, I.., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
362
[31] Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F. Rajashekhar, V., Ramesh, S., and Soyke, J. Tensorflow-serving Flexible, high- perfforroe ml serving. In NeurlPS Workshop, 2017.
364
[32]
Pang*, T., Xu*, K., and Zhu, J. Mixup inference: Better exploiting mixup to defend adversarial attacks. In ICLR, 2020.
385
386
[33] Perez, E., Strub, F., De Vries, H., Dumoulin, V. and Courville, A. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
367
368
[34] Quionero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, D. Dataset shift machine learning. The MIT Press, 2009.
369
370
[35] Radosavovic, I., Johnson, J., Xie, S., Lo, W.-Y,, and Dollár, On network design spaces for visual recognition. In ICCV, 2019.
371
[36] Radosavovic. I., Kosaraju, R. P., Girshick, R., He, K., and Dollár, P. Designing network design spaces. In CVPR. 2020.
373
374
[37] Raff, E., Sylvester, J., Forsyth, S., and McLean, M. Barrage of random transforms for adversarially robust defense. In CVPR, 2019.
376
[38]
Rice, L., Wong, E., and Kolter, Z. Overfitting in adversarially robust deep learning. In ICML, 2020.
377
[39]
Rony, J., Hafemann, I. G., Oliveira, I.. S., Ayed, I. B., Sabourin, R., and Granger, F. Decoupling direction and norm for efficient gradient-based 12 adversarial attacks and defenses. In CVPR, 2019.
378
379
[40] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang. Z., Karpathy, A., Khosla, A.. Bernstein, M., et al. ImageNet large scale visual recognition challenge. IJV,, 2015.
380
[41] Saenko, K.. Kulis, B., Fritz, M., and Darrell, T. Adapting visual category models to domains. In ECCV, 2010.
382
[42] Samangouei, P., Kabkab, M., and Chellappa, R. Defense-GAN; Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018.
384
385
[43] Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS, 2020.
387
[44] Schwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., and Mittal, P. Improving adversarial robustness using proxy distributions. arXiv preprint 21Xiv22104.09425, 2021.
388
389
[45] Shannon, C. A mathematical theory of communication. Bell system technical journal. 27, 1948.
[46] Sharma, Y. and Chen, P.-Y. Attacking the madry defense model with 1_ f_1-based adversarial examples. arXiv preprint arXiv: /710. 10733,, 2017.
391
[47] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely gated mixture- -xeperts layer. In ICLR, 2017.
393
[48] Shelhamer, E., Wang, D., and Darrell, T. Blurring the line between structure and learning optimize and adapt receptive fields. arXiv preprint arXiv:1904.1 11487, 2019.
396 [49] Shi. C., Holtz, C., and Mishne, G. Online adversarial purification based on self-supervised learning. In ICLR, 2021. URL http:p/////ppeeeeeioeeee/tttttte htt.:/orrrr7..ddd11111111111.
11

---

398
[50] Song, Y. Kim, T., Nowozin, S., Ermon, S., and Kushman, N. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In ICLR, 2018.
400
[51]
Stutz, D., Hein, M., and Schiele, B. Disentangling adversarial robustness and generalization. In CVPR, June 2019.
402
[52]
Su, D., Zhang, H., Chen, H., Yi, J., Chen, P.-.,, and Gao, Y. Is robustness the cost of accuracy? comprehensive study on the robustness of 18 deep image classification models. In ECCV, 2018.
403
404
[53|
Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training for out-of-distribution generalization. In JCML, 2020.
405
408
[54] Szegedy, C., Zaremba, W. Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties neural networks. In ICLR, 2014,
407
[55]
Tramer, F. and Bonch, D. Adversarial training and robustness for multiple perturbations. In NeurlPS, 2019.
409
[56]
Tramer, F., Carlini, N., Brendel, W., and Madry, A. On adaptive attacks to advorsarial example defenses. In NeurIPS, 2020.
411
[57]
Veit, A. and Belongie, S. Convolutional networks with adaptive inference graphs. In ECCV, 2018.
412
[58] Wang, D., Shelhamer, Liu, S., Olshausen, B., and Darrell, T. Fully test tim-time adaptation by entropy minimization. In ICLR, 2021.
413
[59] Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., and Gonzalez, E, Skipnet: Learning dynamic routing in convolutional networks. In ECCV, 2018.
415
[60]
Wong, E., Rice, L.,, and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
417
[61]
Wu, D., Xia, S.-T,, and Wang, Y. Adversarial weight perturbation helps robust generalization. In NewrlPS, 2020.
419
[62] Yang, B., Bender, G., Le, Q. V., and Ngiam, J. Condconv: Conditionally parameterized convolutions for efficient inference. In NewrlPS, 2019.
420
421
633 Yuan, X., He, P., Zhu, Q. and Li, X. Adversarial examples: Attacks and defenses for deep learning. TNNLS, 2019.
422
[64]
Zagoruyko, S. and Komodakis, N. Wide residual networks. arXiv preprint arXiv:1605.077 2016.
424
[65] Zhang, H. Yu, Y., Jiao, J., Xing, E., El Ghaoui, L.,, and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
426
[66] Zhang, H., Chen, H., Xiao, C., Gowal, S., Stanforth, R., Li, B., Boning, D., and Hsich, C.-J. Towards stable and efficient training of verifiably robust neural networks. In ICLR, 2020.
Checklist
428
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] 1. The last move advantage is described in Section 3.2 under "Discussion". 2. Dent is unique in its adaptation of the model parameters during testing without altering training. Jection[]explan defenses optimize the input, but require auxiliary models, and their parameters are fixed. adversarial
432
439
that generative and self-supervised
434
435
Table is our main result showing that dent improves state-of-the training defenses. 4. Adaptive attacks are explained and reported in Section 44.)
436
437
(b) Did you describe the limitations of your work? [Yes] Section 4.2 describes the threat model for our work and its requirement of test batches. This is discussed more in Section
438
439
440
(c) Did you discuss any potential negative societal impacts of your work? [No] Our project is about defense, not attack, and so it has less potential for direct negative impact.
441
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
12

---

444
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
446
(b) Did you include complete proofs of all theoretical results? [N/A]
447
3. If you ran experiments...
448
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL.? [Yes] The experiment setup (Section D.T and code in the supplementary material specify experiment details and reproduce our main results for dent+.
449
450
451
452
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Our method alters testing, not training, but its test-time optimization details are given by the above.
453
454
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We evaluate with the standard adversarial benchmarks or AutoAttack and RobustBench, which do not report variability in this way.
457
458
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Our project does not require exotic resources, but standard deep learning hardware. In particular, use V100 and Titan Xp GPUs on local servers.
459
4. If you using existing assets (e.g., code, data, models) or curating/releusing assets...
463
(a) your work uses existing assets, did you cite the creators? [Yes] Libraries, datasets, and models are all cited in Experiments (Section 4
465
(b) Did you mention the license of the assets? [No] The licenses are mentioned at the given references and URLs.
466
(c) Did you include any new assets either in the supplemental material as URL? [Yes] Our code is included in the supplemental material.
468
469
(d) Did you discuss whether and how consent was obtained from people whose data you'le rou're using/curating? [N/A] We did not collect new data.
470
471
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We did not collect new data.
472
473
5. If you used crowdsourcing or conducted research with human subjects...
474
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
475
476
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
478
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
479
13


Reviews:
[{"review_id": "oB0W57Yr9KK", "reviewer": "Reviewer_jWma", "summary": "The paper proposes a novel test-time dynamic defense method, which adapts the model and input during testing by defensive entropy minimization (named dent). Given an input, the proposed method minimizes the entropy of the model prediction to adapt the input and updates parameters in batch normalization layers. Empirical results show that the proposed method improves the robustness of adversarially/nominally-trained models against various attacks including AutoAttack. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The idea is novel and very interesting.\n- The paper is easy to follow.\n- Experimental results show strong robustness against existing attacks including AutoAttack.\n- The paper considers adaptive attacks.\n\nCons:\n- There is a big concern about vulnerability against a kind of data poisoning attacks at test time.\n\n-------------\nOverall, this work is interesting and impressive. The idea is novel and the strong robustness against AutoAttack provides useful information for researchers in this field.\n\nHowever, there is a big concern about vulnerability against a class of adaptive attacks. The proposed method sequentially updates model parameters, and it is concerning that this type of defense methods could be vulnerable against a kind of data poisoning attacks at test time. Before attacks, between attack steps, and a part of each batch, attackers can feed malicious inputs (e.g., random noises) to make model parameters meaningless. The attackers can also craft inputs to broke model parameters.\n\nAlthough this type of attacks is not used in existing attacks, it is because the parameters of the target model is typically fixed, and these attacks are meaningless against the fixed target models. In white box settings, while the purpose of attackers is to find an adversarial example with small perturbation, attackers can feed any inputs to the model for their purpose. Is the proposed defense robust against these attacks? It is important to present discussions and experiments for this type of attacks to show the robustness of the proposed method in white-box settings. \n\n\nI would be happy to raise my score in case these points are properly addressed.\n\nOther comments:\n\n- The word “step” is sometimes confusing. There are two kinds of steps in the paper: attack steps and dent steps. For example, Figure 1 shows three attack steps for static/dynamic defenses, while “Step” in Table 3 is dent steps (i.e., steps to update $\\Delta$ and $\\Sigma$). However, it seems no sufficient explanation about dent steps in the paper. \n\n- The drawback of dent+ is not clear. In line 148, the authors wrote “Dent adapts on batches rather than samples. Batch-wise adaptation stabilizes optimization”, but Table 1 shows that sample-wise Dent+ achieves better performance. I feel the statement lacks consistency. Readers may think that dent is not necessary as dent+ is better than dent.\n\n- The difference in the optimizer for Dent and Dent+ is a bit weird. Why do the authors adopt AdaMod with large learning rate for Dent+, and how do you decide the optimizer? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed limitations of the proposed method in Section 5.\n\nThe authors have not described negative societal impacts since they propose a defense method.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "MKzTgQaZuuA", "reviewer": "Reviewer_XQWL", "summary": "This paper proposes to defend deep networks by dynamically modifying the model and the input images. The model and the input are modified by optimizing the Shannon entropy with respect to the model and the input. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Issues:\n1) First of all, modifying the input [1,2] and dynamically modifying the model [3] for defensive purpose has been explored in previous works. The authors should cite these works and be more careful and precise with the claim of :\n> Line 53 We demonstrate the **first** defense to optimize the model and input during testing for improved robustness.\n\n2) The key design of this work is the proposal of minimizing the Shannon entropy, which confuses me a lot. How come can optimizing the current prediction help find the correct prediction? For an adversarial example that is wrongly classified, wouldn't optimizing the wrong prediction worsen the prediction. To be more clear, from my experience, for an example with label A, if adversarial attacks can modify the example to make the model predict B, and the dent method optimizes the entropy for the adversarial example, it is likely to be in favor of B rather than A. After briefly reading the provided code of cifar10a.py, I think I find a possible reason. In the line 28 of cifar10a.py, the adversarial examples are generated on a dynamic model rather than a static one. The dynamic model may hamper the attack to find a proper adversarial example. Thus, I have two questions:\n* What is the accuracy of the attack in the line 28 and does the attack really succeed? This means what is the accuracy if adversarial examples are generated with dynamic model but predict by a static model? If the adversarial example generated on the dynamic model can not fool the static model, then this is an evidence that the attack in the code is not properly implemented.\n* What will happen if we generate adversarial examples on a static model then predict these adversarial examples on dynamic one? I suspect that, in this setting, the dent method will be broken. BTW, to show the effectiveness, [1] also tested on both static and dynamic/randomized model. The author should follow the same routine.\n\nWithout clarifying these facts, it is very likely that the huge improvement is simply an illusion of incorrect attacks implementation. \nAlso notice that, the author claims:\n> Line154 : Our defense does not rely on shattered gradients\n\nThis claim is questionable. It seems to me that the dynamic process is not differentiable and may hamper a proper gradient computation. The author should adopt the BPDA in [4] to show it does not suffer from the obfuscated gradient. Also, the author claims that their method is deterministic and thus does not suffer from stochastic gradient problem (Line155). This is somewhat confusing. If the method is **deterministic**, then this method should not be claimed as **dynamic**. Please explain.\n\n3) The evaluations in this work is questionable on several aspects. For robust models, the author omit two of the most important works: TRADES[5] and Adversarial Training[6]. Although the models tested in this paper are occasionally built up on these two methods, the author still should provide evaluations on these two important baselines for better comparison. Also, on Table 2, why only provide the detailed results on only two methods? Clearly the authors have the information for other models since they show it on Table1. The presented two methods Wong and Ding are either not the most representative nor have the best performance. \n\n4) As claimed by the author, the methods require batch predictions. What causes this deficit? To me, the modification of the model should not rely on batch information so much since it is stable after training. On Table 8, the author says:\n> With static batch statistics (×), small batch sizes are better. With dynamic √batch statistics ( ), small batch sizes are worse.\n\nWhat causes this phenomena?\n\nMinor Issues:\n1) For Line 29~32, I understand that adversarial robustness is related to domain adaptation. But how does this fact relate to the method in this work?\n2) For Line 36, what does \"adaptive attacks that are aware of its updates\" refer to\n3) For Line 135~137, please be more clear? Have dent solve the batch-wise deficiency?\n4) For Line 158, what does \"stale gradient\" mean?\n5) I recommend the author to write an algorithm to show their method more clear.\n6) For Line 224, Table what?\n\n\nThe second point of the major issues is my major concern about this work. I will change my score if the author can answer my questions and provide solid proofs for their effectiveness.\n\n\n[1] Mitigating adversarial effects through randomization.    ICLR 2018\n\n[2] Randomization matters How to defend against strong adversarial attacks    ICML 2020\n\n[3] Adversarial Robustness via Runtime Masking and Cleansing    ICML 2020\n\n[4] Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples    ICML 2018\n\n[5] Theoretically Principled Trade-off between Robustness and Accuracy    ICML 2019\n\n[6] Towards Deep Learning Models Resistant to Adversarial Attacks", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work does not have obvious negative societal impact. I have listed the limitations in the main review part.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "LN6IJ2K-wG", "reviewer": "Reviewer_B1uw", "summary": "This paper studies the problem of dynamic defenses agaisnt the potential adversarial attacks during the test time. The work introduces a new method derived from \"defensive entroy minimization\" approach that \"fights back\" and counter-optimize against attacks. In general, the proposed method updates the parameters of DNN slightly for every round of inferences, so as to balance the accuracy  and fuzziness of models. The basic idea was clear while experiment results demonstrated the potentials of proposed methods. During the experiments, authors aimed at establishing three major claims (1) Dynamic Defense of Adversarial Training, (2) Dynamic Defense of Nominal Training, and (3) Adaptive Attacks on Dent Updates, with sub-claims in details. Experiment results support these claims in part. Ablation studies also demonstrate the indispensibility of and contributed made made by every component.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Typo: 224 \"Table ?? in the supplement...\"\nThe work was comprehensive and intuitive with details in experiments and comparisons. \nSupplementary materials provided necessary information to follow this work.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My major concern on this paper is that how dent ensures the parameter update \\Delta would not hurt the performance of neural networks. As was learned in some previous (e.g., lottery ticket theory), the modification to the weights might not affect the way that DNN behave. Or we can say that updating weights might not be the most efficient way to achieve the goal. Just be curious, have you explored the potentials of feature maps perturbation, where the behaviors of the DNN could be slightly changed with accuracy of prediction preserved?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hrs"}, "extra_text": {}}, {"review_id": "KaROrCzLPJK", "reviewer": "Reviewer_K7sr", "summary": "This paper provided a new perspective on adversarial defense, i.e., optimizing adversarial defenses at inference. \n\nThe authors proposed dynamic defenses that adapt the model and input during testing to defend against adversarial attacks. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1 The new perspective on defense at inference time indeed deserves some explorations. This perspective is interesting. \n\n2 comprehensive experiments on CIFAR-10/100 and ImageNet are reported; the experimental results look good. \n\nCons:\n\n1 Auto-attack is meant for evaluating a single & static model, which may not be applicable to evaluating dynamic defense. Therefore, authors may consider the adaptive attacks for the robustness evaluations. \n\n2 Inference-time defense may hurdle the speed of the model prediction. \n\nSome other comments:\n\nThe authors implicitly assume adversarial data comes from a different distribution from that of natural data. Could authors elaborate more about it?\n\n* I may adjust my scores based on the authors' responses. \n\n\n#### Post rebuttal ####\nThe authors did not resolve my concerns. Please find my reasons as follows. \n\nAuto-attack is meant for evaluating a single & static model, which may not be applicable to evaluating dynamic defense. Therefore, using AA attacks for evaluating the dynamic defense is a factual error. \nAlthough there are some discussions of adaptive attacks on \"dent\", those adaptive attacks strategy still cannot convince me enough. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "1G6KoBWgmu_", "reviewer": "Reviewer_tNPw", "summary": "This paper proposed a new defense, namely, defensive entropy minimization (dent), to improve adversarially robustness. The proposed method alters testing but not the training phase, which is different from most existing defenses that are static and do not adapt during testing. The improvements of model robustness evaluated by different strong attacks and datasets have confirmed its effectiveness. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper is clearly organized and easy to follow.\n2. The proposed method (dent) is interesting and reasonable.\n3. Extensive experimental results on multiple datasets are provided and the effectiveness of the proposed method is confirmed.\n\nCons:\n1. The performance of the proposed method (dent) is not stable as the static defense (e.g., adversarial training). When the batch size of the test data becomes smaller, the defense effect will be weakened. Since the adaptation is based on the test data, dent may not be able to handle the test scenarios (e.g., using small test batch size or hard dataset like ImageNet) which are not friendly to the estimation for the sample-wise parameters. Could the authors give more discussion or analysis and make some improvements to the method?\n2. The relationship between this method and domain adaptation is not very clear. Could the authors give more explanation about it and how dent improves robust accuracy while maintaining natural accuracy?\n3. Lack of further analysis or discussion on the performance using different datasets. More experimental results or discussion about dent's performance on the large dataset (i.e., ImageNet) are encouraged.\n \nMinor Concerns/Questions:\n1. It seems that the correct reference is missing in line 224. (The \"Table ??\" in line 224)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As aforementioned, could the authors further lessen the method's dependence on the batch size of test data? (or give more discussion?)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed to dynamically modify the model and the input images during the test time (named "defensive entropy minimization"). While the idea is interesting, the true novelty needs to be carefully clarified given many missing related papers, because the test-time defense is in fact not new but one of the two main-stream approaches to adversarial robustness (the other is adversarial training, i.e., training-time defense by simulating the presumed attack without seeing the real attack). Besides the first 3 papers given by Reviewer XQWL, there is another paper closely related to test-time defense: Countering Adversarial Images using Input Transformations, ICLR 2018 (https://arxiv.org/abs/1711.00117). The paper cannot be accepted this time, and hope the authors can find the information helpful for their submission next time!
Decision: Reject