Evading Adversarial Example Detection Defenses with Orthogonal Projected Gradient Descent
Authors: Oliver Bryniarski
Nabeel Hingun
Pedro Pachuca
Vincent Wang
Nicholas Carlini
Venue: ICLR 2022 Poster (2022)
Keywords: Adversarial examples
adversarial attacks

Abstract: Evading adversarial example detection defenses requires finding adversarial examples that must simultaneously (a) be misclassified by the model and (b) be detected as non-adversarial. We find that existing attacks that attempt to satisfy multiple simultaneous constraints often over-optimize against one constraint at the cost of satisfying another. We introduce Selective Projected Gradient Descent and Orthogonal Projected Gradient Descent, improved attack techniques to generate adversarial examples that avoid this problem by orthogonalizing the gradients when running standard gradient-based attacks. We use our technique to evade four state-of-the-art detection defenses, reducing their accuracy to 0% while maintaining a 0% detection rate.

Published as a conference paper at ICLR 2022
EVADING ADVERSARIAL EXAMPLE DETECTION DEFENSES WITH ORTHOGONAL PROJECTED GRADIENT DESCENT
Oliver Bryniarski' UC Berkeley
Nabeel Hingun' UC Berkeley
Pedro Pachuca" UC Berkeley
Vincent Wang' UC Berkeley
Nicholas Carlini Google
ABSTRACT
Evading adversarial example detection defenses requires finding adversarial ex- amples that must simultaneously (a) be misclassified by the model and (b) be detected as non-adversarial. We find that existing attacks that attempt to satisfy multiple simultaneous constraints often over-optimize against one constraint at the cost of satisfying another. We introduce Selective Projected Gradient Descent and Orthogonal Projected Gradient Descent, improved attack techniques to generate adversarial examples that avoid this problem by orthogonalizing the gradients when running standard gradient-base attacks. We use our technique to evade four state- of-the-art detection defenses, reducing their accuracy 0% while maintaining a 0% detection rate.
INTRODUCTION
Generating adversarial examples (SZS 14, BCM+ inputs designed by an adversary to cause neural network behave incorrectly, straightforward. By performing input-space gradient descent (CW17b) MMS+ 17). it is possible to maximize the loss of arbitrary test examples. This process is both efficient and highly effective. But despite much effort by the community, attempts at designing defenses against adversarial examples have been largely unsuccessful and gradient-descent attacks continue to circumvent new defenses (ACW18) TCBM20).
As result, many defenses aim make generating adversarial examples more difficult by requiring additional constraints on imputs for them to be considered successful. Defenses that rely on detection, for example, will reject inputs if secondary detector model determines the input is adversarial (MGFBI7) XEQ17). Turning a benign input æ into an adversarial example x' thus now requires fooling both the original classifier, f, and the detector, g. simultaneously.
Traditionally, this is done by constructing a single loss function L that jointly penalizes the loss on f and the loss on g (CW177)) e.g.. by defining L(x') L(S) + XL(g) and then minimizing L(x') with gradient descent. Unfortunately. many evaluations using this strategy have had limited success-not only must À be tuned appropriately. but the gradients of f and g must also be well behaved.
Our contributions. We develop a new attack technique designed to construct adversarial examples that simultancously satisfy multiple constraints. Our attack approach is a modification of standard gradient descent (MMS* 17) and requires changing just few lines of code. Given two objective functions ff and g, instead of taking gradient descent steps that optimize the joint loss function f f+ Åg. we selectively take gradient descent steps on either or g. This makes our attack both simpler and easier to analyze than prior attack approaches.
We use our technique to evade four state-of-the-art and previously-unbroken defenses to adver- sarial examples: the Honeypot defense (CCS'20) 20), Dense Layer Analysis (IEEE Euro S&P'20) 19). Sensitivity Inconsistency Detector (AAAI21) (TZD21I, and the SPAM de- tector presented Detection by Steganalysis (CVPR' 19) (LZZ*19) In all cases, we successfully reduce the accuracy of the protected classifier to 0% while maintaining detection AUC of less than 0.5 meaming the detector performs worse than random guessing.
"Equal contributions. Authored alphabetically.

---

Published as a conference paper at ICLR 2022
2 BACKGROUND
2.1 NOTATION
We consider classification neural networks f 124 + IR" that receive a d-dimensional input vector (in this paper, images) R and output n-dimensional prediction vector f(x) E IR", We let g RR + R denote some other function which also must be considered, where ((x) I 0 when the constraint is satisfied and g(x) 0 if it is violated. Without loss of generality, in detection defense this function g is the detector and higher values corresponding to higher likelihood of the input being an adversarial example. To denote the true label of is given by we write c(x) = y. In abuse of notation, write y = (xx to denote the arg-max most likely label under the model f-
2.2 ADVERSARIAL. EXAMPLES
Adversarial examples (SZS* 14 BCM+13 have been demonstrated in nearly every domain in which neural networks used. (ASE+ + 18 CWI8 HPG+ Given input x corresponding to label c(x) and classifier f. an adversarial example is perturbation x' of the input such that d(x, x²) < and f(z') # c(x) for some metric d. Additionally, an adversarial example can be targeted if, given target label &f c(x) we have f(x') with d(xxx²)) < e. The metric d is most often that induced by a p-norm, typically either I|- ll2 || 11.2. With small enough perturbations under these metries, the adversarial example x' is not perceptibly different from the original input
Datasets. We attack each defense on the dataset that it performs best on. All of our defenses operate on images. For three of these defenses, this is the CIFAR-10 dataset (KH09), and for one, it is the ImageNet dataset (DDS+09). For each defense we attack, we constrain our adversarial examples the threat model originally considered perform fair re-evaluation, but also generate adversarial examples with standard norms used extensively in prior work in order to make cross- defense evaluations meaningful. We perform all evaluations on a single GPU. Our attacks on CIFAR- 10 require just few minutes, and for ImageNet few hours (primarily due to the defense having a throughput of one image per second).
2.3
DETECTION DEFENSES
We focus our study on detection defenses. Rather than directly improve the robustness of the model
(MSS 17 RSLI8; LAG+19 CRK19), detection defenses classify inputs as adversarial or benign MGFB XEQ17) so they can be rejected. While there have been several different strategies attempted detect adversarial examples in the past (GSS15) MGFB 17] FCSG17) X MC17 MLW 18 RKH19), many of these approaches were broken with adaptive attacks that designed new loss functions tailored to each defense (CW17al TCBM20).
2.4 GENERATING ADVERSARIAL EXAMPLES WITH PROJECTED GRADIENT DESCENT
Projected Gradient Descent (MMS+17 17) is powerful first-order method for finding such adversarial examples. Given a loss L(f, x, t) that takes a classifier, input, and desired target label, we optimize over the constraint set S. = d(x ((x < e) and solve
x' arg min L(f, ZES.
by taking steps T441 = IIs, (z; aV f(f t)).
Here IIs, denotes projection onto the set Se, a is the step size, and 2o is randomly initialized (MMS* 17). This paper adapts PGD in order solve optimization problems which involve minimizing multiple objective functions simultaneously. For notational simplicity, in the remainder of this paper we will omit the projection operator Ils.
Attacks using PGD. Recent work has shown that it is possible to attack models with adaptive attacks that target specific aspects of defenses. For detection defenses this process is often ad hoc. involving alterations specific to each given defense (TCBM20). An independent line of work develops automated attack techniques that are reliable indicators of robustness (CH20); however, in general, these attack approaches difficult apply to detection defenses.
2

---

Published as a conference paper at ICLR 2022
0
glo
Figure 1: A visualization showing how standard Lagrangian attack fails when succeeds over a non-convex loss landscape. Given two circular regions corresponding to when f(z) < O (above) and g(x) 0 (below), we would like find the central region where both satisfied. (left) With Lagrangian PGD, the attack falls in a local minimum and fails to satisfy both constraints simultaneously regardless of the value À selected. (middle) Our S-PGD attack first moves towards the upper region by minimizing fæz). Once this constraint is satisfied (and f(z) < 0), we begin to minimize gar); however this overshoots to point where f(x) > 0. A final step recovers a valid solution to both. (right) Our O-PGI) attack follows the same trajectory for the first two steps optimizing (x) However after this it takes steps orthogonal to fx) maintaining the constraint
fx) < 0 while simultancously minimizing g(r), giving valid solution more quickly.
RETHINKING ADVERSARIAL EXAMPLE DETECTION
Before we develop our improved attack technique to break adversarial example detectors, it will be useful to understand why evaluating adversarial example detectors is more difficult than evaluating standard classifiers.
Early work on adversarial examples often set up the problem slightly differently than we do above in Equation| The initial formulation of an adversarial example (SZS+ 14 CWI7b) asks for the smallest perturbation 6 such that f(r + 5) is misclassified. In the targeted case, this means solving
arg min |011| such that f(z + 5) 5)= =t
for given target that is not equal to the original correct label for Solving this problem as stated intractable. It requires searching over a nonlinear constraint set, which not feasible for standard gradient descent. As result, detection papers typically (SWW+20) [SCCB19) reformulate the search
with Lagrangian relaxation
arg min |0|12 + XC(f, 5,t)
This formulation is simpler, but still (a) requires tuning À to work well, and (b) is only guaranteed to be correct for convex functions C-that it works for non-convex models like deep neural networks is not theoretically justified. It additionally requires carefully constructing loss functions L (CW175).
Equation 1 simplifies the setup considerably by exchanging the constraint and objective. Whereas in Equation we search for the smallest perturbation that results in misclassification, Equation | iisteed finds an input x 8 with 8 that minimizes the classifier's loss. This is simpler formulation because now the constraint is convex, and so we can run standard gradient descent optimization.
Evading detection defenses is difficult because there are now two non-linear constraints. Not only must the input be constrained by distortion bound and be misclassified by the base classifier,
but we must also have that they are not detected, i.e., with g(x) < O. This new requirement nonlinear, and now it becomes impossible to side-step the problem by merely swapping the objective and the constraint as we did before: there will always be at least one constraint that is non- lonelinear function, and standard gradient descent techniques can not directly apply.
In order to resolve this difficulty, the existing literature applies the same Lagrangian relaxation as was previously applied to constructing minimum-distorte adversarial examples. That is, breaking detection scheme involves solving
arg min L(f, xg(x)) #ES.
where À is hyperparameter that controls the relative importance of fooling the classifier versus fool- ing the detector. However, this formulation again brings back all of the reasons why the community moved past
adversarial examples.
3

---

Published as a conference paper at ICLR 2022
3.1 A MOTIVATING EXAMPLE
E E R and ||l| = as visualized in Figure [ By setting exp((1))exp(-11ū + - - where that satisfies both f(ā) < 0 and g(z) 0 can be made arbitrarily close to the origin 7
Let [(z) - exp(ep) -1) - exp(- 1 = and g(
small constant, the only solution
However, no standard Lagrangian formulation will be able to find this solution. Consider the sum h(z; f()) + (z); then we can show that for all we will have arg min A) * O. To see this, observe that while it is possible for the gradient Vh(ō; A) = (one of the conditions for a value to be a local minima), the loss surface is always "concave down" at the origin. It will always be possible to move slightly closer to é or -č and decrease the loss. Therefore, minimizing (x) will never be able find valid stable solution to this extremely simple problem, it will always collapse to finding a solution of either é or -G, which only satisfies one the two equations.
4 OUR ATTACK APPROACHES
We now present our attack strategy designed to generate adversarial examples that satisfy two constraints. As we have been doing, each of our attack strategies defined below generates targeted adversarial example so that f(x') = t but g(x") < O. Constructing an untargeted attack nearly identical except for the substitution of maximization instead of minimization.
4.1 SELECTIVE GRADIENT DESCENT
Instead of minimizing the weighted sum of ʃ and our first attack never optimizes against a constraint once it becomes satisfied. That is, we write attack as
A(x,1) () = arg min D') ([((x) + 1) + g(x') 1[(())=
(4)
Liptas (10,t)
The idea here is that instead of minimizing a convex combination of the two loss functions, selectively optimize either f or depending on if fxx = t, ensuring that updates are always helping to improve either the loss f or the loss g.
Another benefit this style is that it decomposes the gradient step into two updates, which prevents imbalanced gradients, where the gradients for two loss functions are not of the same magnitude and result in unstable optimization (JMW+20), In fact, our loss function can be viewed directly this lens following the margin decomposition proposal (JMW+20) by observing that
/V(ff x, if ffx ø Vg(z)
V Lupdate (ar, 1)
fx)
That is, each iteration either take gradients on f or on g depending on whether f(x) / or not.
Recalling the motivating example from Figure I this selective optimization formulation would be able to find a valid solution. No matter where we initialized our adversarial example search, minimizing with respect to f(z) will eventually give a valid solution near e. Once this happens, we then switch to optimizing against g(x) (because f(x) is satisfied). From here we will eventually converge the solution R 0.
4.2 ORTHOGONAL GRADIENT DESCENT
The prior attack, while mathematically correct, might encounter numerical stability difficulties. Often, the gradients of f and g point in opposite directions, that is, ff gg. As result, every step spent optimizing of causes backwards progress on optimizing against g. This results the optimizer constantly "undoing" its progress after each step that is taken. To address this problem, would like to "remove" the portion of one gradient optimization step that "undoes" the progress of a previous optimization step.
To do this, we call on vector projections. Note that Vg(x) = VC(f.z,) projv VL x, f) is orthogonal to the gradient Vg(Ir), and similarly V VL(f,, +)+ is orthogonal to VL(f x,,, 1). We
4

---

Published as a conference paper at ICLR 2022
integrate this fact with Equation 5 to give slightly different update rule that again solves Equation however this time by optimizing:
-) Vf(frttt
# if if if f()) =
Lupdste (x. ()
(6)
Vg(z)) - provvcpraaa Vg((r)
The purpose of this update is to take gradient descent steps with respect to one of f or g in such a way that we do not significantly disturb the loss of the function not chosen. In this way, we prevent our attack from taking steps that undo work done in previous iterations of the attack.
It is also important to note that, in the high-dimensional space in which a typical neural network operates, the gradients of f and g are practically never exactly opposite. that is situation where Vf Vg. In this case, the projection of V f onto Vg and Vg onto ff would be and we could not make any meaningful optimizations towards satisfying either constraint with OPGD.
Again recalling Figure [ by taking steps that are orthogonal to f(x) we can ensure that once we reach the acceptable region for f, we never leave it, and much more quickly converge on an adversarial example that evades detection.
CASE STUDIES
We validate the efficacy of our attack by using it to circumvent four previously unbroken, state-of-the the art defenses accepted at top computer security or machine learning venues. Three of the case studies utilizes models and code obtained directly from their respective authors. In the final case the original authors provided us with matlab source code that was not easily used, which we re-implemented.
Attack Success Rate Definition. We evaluate the success of our attack by a standard metric called attack success rate at N (SR @ for short) (SKCB19). We use SR@ @ N to ensure comparability across
different case studies but more importantly between our results and our case studies' original results. SR @ N is defined the fraction of targeted attacks that succeed when the defense's 's false positive rate is set to N%. (To adjust a defense's false positive rate it suffices to adjust the detection threshold á so that inputs are rejected as adversarial when g(x) Φ.) For example, 94% S SR@5 could either be achieved through 94% of inputs being misclassified as the target class and 0% being detected as adversarial, or by 100% of inputs being misclassified the target class and 6% being detected as adversarial, or some combination thereof. We report $R@5 @5 and SR @ 50 for our main results D and for completeness also give the full ROC curve of the detection rate for more complete analysis.
A Attakk Hyperparameters. We use the same hyperparmaeter setting for all attacks shown below. We set the distortion bound e to 0.01 and .031: several of these papers exclusively make claims using the value of 0.01 SWW+ 20 SKC 20 TZLD2I), but the value 0.031 8/255 is more typical in the literature (CBM2)). We run our attack for N = 1000 iterations of gradient descent with a step size (that is, the step size changes as function of which follows standard advice (MMS+
5.1 HONEYPOT DEFENSE
The first paper we consider is the Honeypot Defense NWW++0) Instead of preventing attackers from directly constructing adversarial examples, the authors propose to lure attackers into producing specific perturbations that are easy to find and hard to ignore. These perturbations are called "honeypots" or trapdoors and can be easily identified by a detector. For their evaluation on the MNIST and 10 dataset, the authors use 5 sets of randomly selected 3 x 3 squares per label.
Formally, consider an input x to the classifier, S. During training. f is injected with a honeypot, A. The signature of a particular honeypot, SA. is the expectation of the neuron activations of f over multiple sample inputs containing A. During inference, the internal neuron activation pattern e(x) is compared to SA using cosine similarity. Specifically, for a predefined threshold o, if cos(e(x). Sa)= > d. then x flagged as adversarial. One additional modification the authors make is to use neuron randomization. When creating the signatures, a random sample of neuron activations
'The value 5% is used in many prior defenses in the literature (M.W 18 XEQI7). and 50% is an extreme upper bound and would reduce the model's 's accuracy by half.
5

---

255 Published as a conference paper at ICLR 2022
Attack
eps=0.01
eps=0.031
1.0
Orthogonal
SR@5 550
SR @5
@50
Selective
SWW+
0.02
True Positive
0.6
Reproduction
0.13*
0.4
Orthogonal 1.0
0.93
1.0
0.92
Selective 0.998
0.505
0.996
0.628
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) Attack success rate for our two proposed attacks. "When we reproduce the authors attack, we reach slightly a higher attack success rate of 13%.
False Positive Rate
(b) Defense ROC, 0.01, AUC of 0.30
Figure 2: Honeypot attack evaluation. Compared to the originally reported 2% success rate, our attack reaches 100% attack success rate under the same distortion bound. While the ROC curve does cross over the I = line, this only occurs after FPR of 70% which is completely unusable.
ignored and the same set of neurons ignored when calculating the cosine similarity during adversarial attack detection.
We test our method of generating adversarial images on CIFAR-10 with an Iso bound The original model has baseline accuracy of .87. To test the defense, we generate targeted attacks choosing the target from among the incorrect labels. As simple baseline attack, we confirm that FGSM achieves accuracy of .06 on the classifier but the ROC AUC of the detector is .994.
To evaluate the efficacy of our attack, we use the authors pre-trained code, and convert their models from TensorFlow to Py to an attack success rate above 99.9%, even at a 50% false positive rate, and an AUC of 0.30. In Table 2 we give the attack results for several configurations of attack. We shared results with the original defense authors who confirmed our adversarial examples successfully evaded their detection (an important step, given that we had converted the model to PyTorch).
attacks. We find that attack is extremely effective and has
5.2 DENSE LAYER ANALYSIS
Many recent defenses analyze the hidden activations of neural networks while processing benign and adversarial inputs (e.g., following (MGFB17)). These defenses aim capitalize differences in activation patterns among benign and adversarial inputs to train a separate classifier capable of detecting inputs as adversarial.
The most recent paper in this direction Sperl et al. extract dense layer activation patterns among benign and adversarial inputs and train secondary binary classifier that detects adversarial examples (SKCB19). The authors do this by first performing forward pass through a target neural network with both adversarial and benign inputs create feaked-eettue dataset activation-labe pairs. Then, using the mixed-feature dataset, they train secondary binary classifier capable of discerning between adversarial and benign inputs. When evaluating their models, the authors pass an input through the target model to obtain the activation feature vectors for particular input well as potential classification. They then pass this feature vector through the secondary classifier. the secondary classifier alerts that the input was adversarial, the classification is thrown away. Otherwise, classification proceeds as normal.
Sperl al. evaluate this defense with leading adversarial attacks the MNIST and CIFAR- 10 datasets using several models and report high accuracies for benign inputs and high detection rates for adversarial inputs. The authors report worst-case individual attack accuracy of 0.739.
In accordance with our framework, we assign the cross entropy loss the classifier to our primary function and binary cross entropy loss of the detector as secondary function.
We obtain source code and pre-trained defense models from the authors in order to ensure that our attack matches the defense closely possible. We now detail the results of our attack at .01

---

Published as a conference paper at ICLR 2022
Attack
eps=0.01
eps=0.031
SR@S
SR@50
SR $
SR 550
1.0
Orthogonal
Selective
SKCC
<0.13*
Reproduction 0.20
True Positive
0.6
Orthogonal 0.374
0.163
1.0
0.718
0.4
Selective
0.83
0.441
1.0
0.865
0.2
0.0
(a) Attack success rate for our two proposed attacks. "The original paper reported only at 20% FPR, we take this as an upper bound for what could be achieved at 5% FPR. *Whe we reproduce the authors attack, we reach slightly higher success rate of 20%.
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
(b) Defense ROC, 0.01, AUC of 0.38
Figure 3: DLA attack evaluation. Our attack succeeds with 83% probability compared to the original evaluation of 13% (with 0.01), and 100% of the time under the more typical 8/255 constraint.
and at = .03 at false positive rates of 5% and 50% in Figure 3 effective, resulting in accuracy of 0 detection rate of with false positive rate of 5% under = .03 bounds and an AUC of 0.38. Finally, to validate that our attack succeeded, we again shared the resulting adversarial examples with the authors who confirmed our attack results.
We find that our attack is extremely
5.3 SENSITIVITY INCONSISTENCY OF SPATIAL- TRANSFORM DOMAIN
We next evaluated our attack on the Sensitivity Inconsistency Detector (SID) proposed by Tian et al. (TZLD2I). This defense relies on the observations of Fawzi et al. (FMDFSI8 that adversarial examples movements, in the form of perturbations, of benign inputs in decision space along an adversarial direction. Tian et al. then conjecture that, because adversarial examples are likely to lie near highly-curve decision boundaries, and benign inputs lie away from such boundaries, fluctuations said boundaries will often result in a change in classification of adversarial examples but classification of benign inputs.
To measure sensitivity against decision boundary transformations, Tian et al. design a dual classifier which is the composition of weighted additive wavelet transform layer and DNN classifier with the same structure as the original classifier. When doing forward pass of the system, the authors run an input through both the primal and the dual model, then pass both results the detector that discriminates among adversarial and benign classes. With these models, the authors then define their so-called feature of sensitivity inconsistency S(xo).
S(zo) = (2)) (
where f((uu) and (x0) are the predictions of the primal and the dual respectively. Input 2o is classified as adversarial if S())) is greater than a threshold Φ. SID achieves improved adversarial example detection performance, especially in cases with small perturbations in inputs. The authors report a worst-case, individual attack detection AUC % 0.95.
Now, we want create adversarial examples that are misclassified by the original model and not flagged as adversarial by the Sensitivity Inconsistency Detector. We assign the loss of our target model our primary function and the loss of the Sensitivity Inconsistency Detector our secondary function. The initial target model had an accuracy of .94 and deemed .66 of all inputs adversarial.
We again obtain source code from the authors along with pre- pre-trained models to ensure evaluation correctness. We describe our attack's 's results at .01 and at = .03 at false positive rates of 5% and 50% in Figure Our attack works well in this case and induces an accuracy of a detection rate of 0 with false positive rate of 5% under .03 bounds with an AUC of 0.25.
5.4 DETECTION THROUGH STEGANALYSIS
Since adversarial perturbations alter the dependence between pixels in an image, Liu et al. (LZ 19 propose defense which uses steganalysia-inspiree approach detect "hidden features" within
7

---

Published as a conference paper at ICLR 2022
Attack
eps=0.01
eps=0.031
1.0
Orthogonal
Selective
SR@5 @5@50
SR@5
SR@50
(TZL221)
0.09*
True Positive
0.6
Orthogonal
0.931
0.766
1.0
0.984 0.886
0.4
Selective 0.911
0.491
1.0
0.2
0.0
(a) Attack success rate for our two proposed attacks. "The orig- inal paper only reports AUC values and does not report true positive/false positive rates. The value of 9% was obtained by running PGD on the author's defense implementation.
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
(b) Defense ROC, 0.01, AUC of 0.25
Figure 4: SID attack evaluation. Our attack succeeds with 93% probability compared to the original evaluation of 9% under 0.01-norm constraint, and 100% under 0.031.
image. These features then used train binary classifiers to detect the perturbations. Unlike the prior defenses, this paper evaluates on ImageNet, reasoning that small images such as those from CIFAR-10 and MNIST do not provide enough inter-pixe dependency samples construct efficient features for adversarial detection, so we attack this defense on ImageNet.
As baseline, the authors use two feature extraction methods: SPAM and Spatial Rich Model. For each pixel Xi, of an image X, SPAM takes the difference between adjacent pixels along 8 directions. For the rightward direction, difference matrix A computed so that A Xi, Xi.+1. transition probability matrix M between pairs of differences can then be computed with
M Pr(AT.+1 x|A =
where 2, y E feature set F. We use 7 3 in accordance with that used by the authors. The features themselves are calculated by concatenating the average of the non-diagonal matrices with the average of the diagonal matrices:
with T being a parameter used to control the dimensionality of the final
M M M M
M M
Fx+1
4
In order to use the same attack implementation across all defenses, we reimplemented this defense in PyTorch (the authors implementation was in matlab). Instead of re-implementin the full Fisher Linear Discriminant (FLD) ensemble (KFH12) used by the authors, we train a 3-layer fully connected neural network on SPAM features and use this as detector. This allows us to directly investigate the claim that SPAM features can be reliably used to detect adversarial examples, FLD is highly non-differentiable operation and is not fundamental component of the defense proposal.
The paper also proposes second feature extraction method named "Spatial Rich Model" (SRM) that we do not evaluate against. This scheme follows the same fundamental principle as SPAM in modeling inter-pixel dependencies thhre is only a marginal benefit from using these complex models, and so we analyze the simplest variant of the scheme.
Notice that SPAM requires the difference matrices A be discretized in order for the dimensionality of the transition probability matrices M to be finite. To make this discretization step differentiable and compatible with attacks, define count matrix X where, for example, X counts, for every pair i, the number of occurrences of y in A and r in A M y is then defined by:
X
MF P(A = x x|A = = Σn
To construct differentiable approximation, consider without loss of generality the rightward dif- ference matrix AT for an image. We construct shifted copy of it A₂" so that Azi, = -A A We then define a mask K so that
Ki = 1[x S Ab, < <x+1ng S AP <V+1
8

---

Published as a conference paper at ICLR 2022
1.0
Orthogonal
Attack
eps=0.01
eps=0.031
True Positive
0.6
SR@5 SR@50
SR@5
SR@57
50
0.4
197
0.03
.03
0.2
Orthogonal
0.988
0.54
1.0
0.62
0.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) Attack success rate for our proposed attack. For computational efficiency, we only run our Orthogonal attack as the detection model has throughput of one image per second.
False Positive Rate
(b) Defense ROC, 0.01, AUC of 0.44
Figure 5: Steganalysis attack evaluation. We find it difficult to decrease the detection score lower than the original score the non-adversarial input, thus the AUC is almost exactly 0.5.
Each element of the intermediate matrix X counts the number of pairs in AT and A² which would be rounded to and respectively after discretization:
Eig (K A
where o the Hadamard product. If we normalize X so that the sum of elements in each column equal to 1, we get the probability of difference values E Ag conditioned on column y E AT" Thus, for any pair indices j.
Mr P(AZ., =x =|A = y)
E
Using this differentiable formulation of SPAM feature extraction, train auxillary detector as described above and use its gradients to apply our attack on the original, non -on-differentiable detector.
The authors evaluate their defense on 4 adversarial attacks and report high accuracy for benign inputs and high detection rates for adversarial inputs. The best attack they develop still has a success rate less than 3%. In contrast, our attack on SPAM using the differentiable approximation has success rate of 98.8% 8% when considering a 5% false positive rate, with an AUC of 0.44, again less than the random guessing threshold of 0.5.
CONCLUSION
Generating adversarial examples that satisfy multiple constraints simultaneously (e.g.. requiring that an input both misclassified and deemed non-adversarial) requires more care than generating adversarial examples that satisfy only constraint (e.g., requiring only that input is misclassified). We find that prior attacks unnecessarily over-optimizes one constraint when another constraint has not yet been satisfied. Our new attack methogology is designed to avoid this weakness, and result can reduce the accuracy of four previously-unbrode detection methods to 0% accuracy while maintaining 0% detection rate 5% false positive rates.
We believe our attack approach is generally useful, but it is not a substitute for trying other attack techniques. We do not envision this attack as a complete replacement for standard Lagrangian- based attacks, but rather a complement; defenses must carefully consider their robustness to both prior attacks as well as this new one. Notice, for example, that for one the four defenses we study Selective PGD performs better than Orthogonal PGD inndiiinni these attacks are complementary to each other. Additionally, automated attack tools (CH20) would benefit from adding our optimization trick to their collection of known techniques that could (optionally) compose with other attacks. We discourage future work from blindly applying this attack without properly understanding its design criteria. While this attack is effective for the defenses may not the correct way to do so in future defense evaluations. Evaluating adversarial example defenses will necessarily require adapting any attack strategies to the defense's 's design.
consider, it is not the only way do and

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank the authors of the papers we use in the case studies, who helped us answer questions specific to their respective defenses and agreed to share their code with us. We are also grateful to Alex Kurakin and the anonymous reviewers for comments on drafts of this paper.
ETHICS STATEMENT
All work that improves adversarial attacks has potential negative societal impacts. Yet, we believe that it better for those vulnerabilities be known rather than relying on security through obscurity, We have attacked no deployed system, and so cause no direct harm; and by describing how our attack works, future defenses will be stronger. We have communicated the results of our attack the authors of the papers we break as form of responsible disclosure, and also to ensure the correctness of our results.
REPRODUCIBILITY STATEMENT
All of the code we used to generate our results will be made open source in GitHub repository. The datasets we use (MNIST, CIFARIO, ImageNet) are available online and widely studied. We obtained original copies of the code associated with 3 of the 4 case studies. We used code either directly from the authors or code released publicly alongside an academic paper. In the case of steganalysis, we reimplemented the paper to the best of our ability. We also provide Python class constructor that future work can test or improve our results. Again, we relayed results to the authors of each paper and received confirmation that our adversarial examples were indeed adversarial and not detected by the author's 's original implementations.
REFERENCES
[ACWI8] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, 2018
[ASE+18) 18] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivas- tava, and Kei Chang. Generating natural language adversarial examples. CoRR, abs/1804.07998, 2018.
[BCM+ 13] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šındić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pages 387-402. Springer, 2013.
[CH20] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Proceedings of the 37th International Conference on Machine Learning, pages 2206-2216. PMLR, 2020.
[CRK19] Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint urXiv: 2019.
[CW17a] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 3 14, 2017.
[CW17b] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy, pages 39-57. IEEE, 2017.
[CW18] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 20/8 IEEE Security and Privacy Workshops (SPW), pages 1-7, 2018.
10

---

Published as a conference paper at ICLR 2022
[DDS+09] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pages 248 148-555. Iece, 2009.
[FCSG17] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint (5ti:::7030041000 2017.
[FMDFS18] Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfone Pascal Frossard, and Stefano Soatto. Empirical study of the topology and geometry of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[GSS15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
[HPG 17] Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. CoRR, Abs/1702.02284 2017.
[J Linxi Jiang, Xingjun Ma, Zejia Weng, James Bailey, and Yu-Gang Jiang. Imbal- anced gradients: A new cause of overestimated adversarial robustness. arXiv preprint arXiv:2006. 137266 2020.
[KFH12] Jan Kodovsky, Jessica Fridrich, and Vojtéch Holub. Ensemble classifiers for steganalysis of digital media. In IEEE Transactions on Information Forensics and Security, pages 432-444, 2012.
[KH09] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[LAG + 19] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pages 656-672. IEFE, 2019.
[LZZ + 19] Jiayang Liu, Weiming Zhang, Yiwei Zhang, Dongdong Hou, Yujia Liu, Hongyue Zha, and Nenghai Yu. Detection based defense against adversarial examples from the steganalysis point of view. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 08344483,, 2019.
[MC17] Dongyu Meng and Hao Chen. Magnet: two-pronged defense against adversarial examples. Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 135-147, 2017.
[MGFB17] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint 1xXiv:/702.042000 2017.
[MLW+18] Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael Houle, and James Bailey. Characterizing adver- sarial subspaces using local intrinsic dimensionality. arXiv preprint arXiv: 1801.026, 2018.
[MMS+ 17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2017.
[RKH19] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: statistical test for detecting adversarial examples. In International Conference on Machine Learning, pages 5498-5507. PMLR, 2019.
[RSL18] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 2018.
[SKC+20] Philip Sperl, Ching- Yu Kao, Peng Chen, Xiao Lei, and Konstantin Böttinger. Dla: Dense-layer Dense-layer-nnallsiss for adversarial example detection. In 2020 IEEE European Symposium on Security and Privacy (FuroS&P)), pages 198-215. IEFF, 2020.
11

---

Published as a conference paper at ICLR 2022
[SKCB19] Philip Sperl, Ching- yu Kao, Peng Chen, and Konstantin Böttinger. DLA: dense-layer- analysis for adversarial example detection. CoRR, abs/1911.01921, 2019.
[SWW+200 Shawn Shan, Emily Wenger, Bolun Wang, Bo Li, Haitao Zheng, and Ben Y Zhao. Gotta catch' em all: Using honeypots catch adversarial attacks neural networks. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 83, 2020.
[S]S+14] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, and Rob Goodfellow, Ian an d Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[TCBM20] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. CoRR, abs/2002.083477 2020.
[TZLD21] Jinyu Tian, Jiantao Zhou, Yuanman Li, and Jia Duan. Detecting adversarial ex- amples from sensitivity inconsistency of Opatial-transfome domain. arXiv preprint arXiv:2103.04302. 2021.
[XEQ17] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155 2017.
12


Reviews:
[{"review_id": "oMY_MUEz1Wu", "reviewer": "Reviewer_Tn7g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-  The proposed approach is well motivated and novel. Empirically the contributions of the paper are good.\n-  Experiments are well designed. The ROC curves help in better understanding.\n-  The discussed defences have been reproduced well.\n\nWeaknesses:\n\n-  A more detailed discussion on the past works which broke adversarial-detection defences needs to be done in order to better highlight the novelty of the proposed approach. Could the authors clarify why these detection methods cannot be broken using existing adaptive attacks? [1,2] \n-  The clarity on the discussion related to the four defences that are broken could be improved. It is difficult to understand the defence approaches from Section 4 of the paper.\n-  Could the authors clarify why an epsilon bound of 0.01 was used to draw the ROC curves? Could the authors share the curves for an epsilon bound of 0.031 (8/255)?\n\n[1] Carlini et al., Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods\n[2] Tramer et al., On Adaptive Attacks to Adversarial Example Defenses\n\n##### Update post rebuttal #####\n\nThe authors' rebuttal sufficiently addresses my concerns and I am happy to update my recommendation to \"Accept\". \n\n\n", "summary_of_the_paper": "The authors propose an attack that could break 4 adversarial detection methods published recently. Traditionally, attacks against detection methods have attempted to maximize the loss for both classification and detection simultaneously. However, using a toy example the authors show that this is suboptimal, as it may not find the worst-case adversaries. The authors propose to minimize the loss (targeted attack setting) iteratively by optimizing either only for the classification pipeline or the detection pipeline at a time.  The attack first considers the classification loss and further tries to fool the detection pipeline until the classification prediction remains incorrect. The authors also propose a variant of the attack by considering gradient steps for the classification pipeline to be orthogonal to the gradients of the detection pipeline and vice versa. Finally, the paper shows that these two proposed attacks completely circumvent four recent adversarial detection methods.", "summary_of_the_review": "The proposed attack seems interesting. I think the main problem with the paper is with respect to the clarity in writing, especially in Section-4. Further, the past methods which broke detection methods need to be discussed in detail to highlight the significance of the proposed method. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jaQXAe3swaA", "reviewer": "Reviewer_K9MM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main proposed approaches are a fairly simple update to existing attack methods, though appear to be quite effective.\n\nThe presentation of the paper was generally clear, but with some areas of issue. For example\n-\tSection 2.4 was a bit confusing. It was not clear (to me) from the preceding discussion that the focus was to be on targeted attacks. For example, in equation (1), the minimisation of the loss suggests that this is targeted (to class t). Yet in section 2.2, t was used as the valid class of the data point x.\n-\tIn Section 3, however, it seems that t is the valid class, as the first equation is targeting attacks which result in distinct classifications from t. Also reference in the text just above  the bold heading refers to equation2 as maximising the loss (equation 2 is currently a minimum) – which it would do if t were the valid class. \n-\tIn Section 3.1, the function f as defined is –epsilon at x=e and positive (for small epsilon) at x=0. I think the 1 should probably be something like exp(-1) so that it evaluates to –epsilon at the origin. As it stands this section does not make sense.\n-\tIt does not seem clear in section 3.1, even if it were corrected, why a Lagrangian formulation would not work. (Note that that is an independent statement of whether gradient descent would work). Also it is said that where the gradient of h vanishes wrt x the loss function has a non-zero slope and so the loss can be further minimised. But if one respects the constraint g(x)<0 this may not be true. In fact, surely the solution to minimising f st g<0 is the point on the y-axis at the apex of g’s circle. At this point, the gradient of f will point upwards, but one cannot follow that without violating the constraint. I am not clear what the authors are arguing exactly.\n-\tFor the methods proposed in 4.1 and 4.2, it is not made made clear exactly how the proposed gradients are to be used. Obviously, presumably in a PGD style attack. (But even the PGD attack in section 2.4 is imprecisely specified as it is not made clear exactly what x_0 is chosen to be nor how may steps are to be performed, etc.) \n-\tIt is stated that the proposed attack is “easier to analyse” than prior approaches. Yet no analysis really has been done.\n\nThe experimental results seem to be quite convincing (see comments below also), though it would be good to have seen ROC curves for other (standard) attacks included in the figures for comparison (the only data from other attacks is a simple reproduction of the SN@5 value from the original paper (for usually only one epsilon value). Also, given the defence in 5.4 was reimplemented, it would have been good to see a statement that the original results of the original authors were reproduced, and that those authors had also confirmed the attacks, as was said in at least two of the other cases.\n\nThe experiments conducted seem to be fair and thorough, in the sense that the attacks are evaluated against the extant defences with both the threat model originally considered for that defence as well as with common parameter choices. In most cases, the original code of the defence’s authors is used, and also in most cases it seems the authors confirmed their attacks as successful with the defence’s authors.\n\nA side query: I accept that the goal is to not have the gradient of one target conflict with the other (i.e. f and g). But to make the gradient used purely orthogonal to the other is a bit extreme and throws away advantageous cases where the change is in the correct direction for both at the same time. Would it not be possible to instead simply use the orthogonal gradient in cases where its projection is in the negative direction on the other gradient, and use it unchanged otherwise, to get the best of both worlds.\n\nThough the results are good, the issues noted above with regard to clarity and clear motivation mean that my score (for now) is lowered.\n\nSome minor comments:\n-\tShould “defences” in the third paragraph of the Introduction be “attacks”?\n-\tIn section 3.1, h(x,lamda)=0 uses vector notation on the 0. Isn’t h() a scalar?\n-\tI found the notation confusing for projections (it may just be me!). But the paper seems to use proj_A B to denote the projection of the vector A along the direction of the vector B. This would seem counterintuitive i.e. A and B wrong way round? Not sure what is standard here, so I could be wrong.\n-\tIn section 4.2, “... exactly opposite, that is a situation where grad f = grad g”. Surely you mean “- grad g”?\n-\tSection 5: “Three of the case study” should be “Three of the case studies”\n-\tThe text “The equivalence ...” after equation (5) is trivial and need not be said.\n-\tFLD not defined??\n-\tIn section 5.4, the specification of the two ranges of features is the same. Should some be diagonal?\n-\t“non-diiferentiable”\n-\t“any every”\n-\tWhy is only one of the proposed approaches used in 5.4?\n-\tIn the conclusion it is claimed that 0% detection was achieved with 0% accuracy at 5% FPR. This was mentioned in one case, but not clearly achieved in the others? Or at least it wasn’t noted explicitly for the other cases. \n", "summary_of_the_paper": "This paper considers the problem of finding adversarial examples that simultaneously defeat a detector of adversarial examples. An argument is made that existing attacks often achieve one goal at the expense of another. This argument motivates the proposal of two attack techniques. These are evaluated against four existing detection-based defence methods, with successful results.", "summary_of_the_review": "Overall, the two proposed techniques, though a relatively simple update to existing approaches, seem to perform well and provide an effective attack against state of the art adversarial attack detection techniques. However, the presentation of the paper is a bit confused in places and the motivation for the approach a bit unclearly argued. It is hard to argue against such convincing numerical results, so with some rework to address the issues raised I think my recommendation for the paper could increase.\n\nEdit: Score increased after author response and revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OLHCvCdEXZ", "reviewer": "Reviewer_ndnu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find this paper pleasant to read in general: SPGD and OPGD are well motivated and clearly explained, while the experimental verification is sufficient and convincing. I only have several minor comments:\n\n1. How effective are SPGD and OPGD against an ensemble of detectors instead of just one?\n2. For Honeypot Defence and Sensitivity Inconsistency Detector, what is the impact of the threshold $\\phi$ on the effectiveness of SPGD and OPGD?\n3. Why is OPGD much less effective than SPGD against Dense Layer Analysis when $\\epsilon$=0.01 (Figure 3)?\n4. Why isn’t SPGD evaluated against Steganalysis?", "summary_of_the_paper": "This paper proposes two techniques for generating adversarial examples: Selective Projected Gradient Descent (SPGD) and Orthogonal Projected Gradient Descent (OPGD). In order to fool both the victim model $f$ and a detector $g$, SPGD selectively optimise either $f$ or $g$ depending on whether the modified input is misclassified as the target class, while OPGD further orthogonalizes the gradients. Evaluation on four previously unbroken, state-of-the-art defence methods demonstrate the effectiveness of the proposed attacks.", "summary_of_the_review": "The proposed attacks for creating adversarial samples are simple but effective, and the evaluation is sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5UZqq3uXJqi", "reviewer": "Reviewer_AXWZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\nBreaking the defenses of adversarial example detection is an important yet underdeveloped field. The approach proposed in this paper is straightforward and effective in practice. The experimental evaluations seem reasonable.\n\n### Cons\nThe proposed optimization formulation is based on intuition. It would be better if the author can provide theoretical evidences that can explain its effectiveness. \n\nCurrently, the empirical evaluations mainly shows its performance against the detection defense. It can be improved by adding more analytical results.\n\n\n\n", "summary_of_the_paper": "This paper targets on attacking the defensive mechanism of adversarial examples detection. It proposes a new optimization algorithm to simultaneously meet two different requirements. It verifies its effectiveness on several state-of-the-art adversarial example detection methods.", "summary_of_the_review": "I would love to recommend an accept for this paper, considering its topic of research, effectiveness, and novelty.\n\nIn addition, I want to clarify that I served as a reviewer of this paper for its last submission. I am glad to see that the authors followed my suggestions and made several improvements on their writing. For the last submission, I give a weak accept score in the final stage. Considering that the paper has been improved on several aspects, I raise my score to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose two new variants of (projected) gradient descent for attacking a classifier and a detector simultaneously. Using these two new variants they are able to break four recent detection methods for adversarial samples.

Strength:
- All the reviewer acknowledge that breaking these four defenses is a valuable contribution.

Weakness:
- From a technical perspective the paper is rather simple and no theoretical support for the suggested variants is provided. The justification is rather handwavy. From an optimization perspective it is unclear why not a simple penalty-based approach would have given the same results or would even work better. The choices maded in this paper seem a bit arbitrary and are mainly justified by the fact that they work for the four detectors
- the honeypot defense was already broken in 
A Partial Break of the Honeypots Defense to Catch Adversarial Attacks, Nicholas Carlini, arXiv:2009.10975

Minor:
- the authors should update the references, several papers have appeared in the meantime

While I appreciate the contribution of the broken defenses, in terms of technical contribution and discussion of the methods this paper is borderline.
Decision: Accept (Poster)