AttackDist: Characterizing Zero-day Adversarial Samples by Counter Attack
Authors: Simin Chen
Zihe Song
Lei Ma
Cong Liu
Wei Yang
Venue:  (2021)

Abstract: Deep Neural Networks (DNNs) have been shown vulnerable to adversarial attacks, which could produce adversarial samples that easily fool the state-of-the-art DNNs. The harmfulness of adversarial attacks calls for the defense mechanisms under fire. However, the relationship between adversarial attacks and defenses is like spear and shield. Whenever a defense method is proposed, a new attack would be followed to bypass the defense immediately. Devising a definitive defense against new attacks~(zero-day attacks) is proven to be challenging. We tackle this challenge by characterizing the intrinsic properties of adversarial samples, via measuring the norm of the perturbation after a counterattack. Our method is based on the idea that, from an optimization perspective, adversarial samples would be closer to the decision boundary; thus the perturbation to counterattack adversarial samples would be significantly smaller than normal cases. Motivated by this, we propose AttackDist, an attack-agnostic property to characterize adversarial samples. We first theoretically clarify under which condition AttackDist can provide a certified detecting performance, then show that a potential application of AttackDist is distinguishing zero-day adversarial examples without knowing the mechanisms of new attacks. As a proof-of-concept, we evaluate AttackDist on two widely used benchmarks. The evaluation results show that AttackDist can outperform the state-of-the-art detection measures by large margins in detecting zero-day adversarial attacks.

Under review as a conference paper at ICLR 2021
AttackDist: Characterizing Zero-day Adversarial Samples by Counter Attack
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNss have been shown vulnerable adversarial at- tacks, which could produce adversarial samples that easily fool the state-of-the-art DNNs. harmfulness
adversarial attacks calls for the defense mechanisms under fire. However, the relationship between adversarial attacks and defenses is like spear and shield. Whenever a defense method is proposed, a new attack would be followed bypass the defense immediately Devissin defense against new attacks (zero-day attacks) is proven to be challenging. We tackle this chal- lenge by characterizing the intrinsic properties of adversarial samples, via suring the norm of the perturbation after a counterattack. Our method is based the idea that, from an optimization perspective, adversarial samples would be closer to the decision boundary; thus the perturbation to counterattack adversar- ial samples would be significantly smaller than normal cases. Motivated by this, property to characterize adversar-
we propose AttackDist. an ial samples. We first theoretically clarify under which condition At AttackDi i st can provide certified detecting performance, then show that a potential appli- cation of At attacik is distinguishing zero-day adversarial examples with- evaluate
out knowing the mechanisms of new attacks. As proof-of-concept, AttackDist on two widely used benchmarks. The evaluation results show that AttackDist can outperform the state-of- state-of-the-art the-art detection measures by large margins in detecting zero-day adversarial attacks.
INTRODUCTION
Deep Neural Networks (DNNs) have flourished in recent years, and achieve outstanding perfor- mance in a lot of extremely challenging tasks, such as computer vision (He et al. (2016)) machine translation (Singh et al. (2017)) automatic speech recognition (Tüske et al. (2014)) and bioinfor- matics (Choi et al. (2 In spite of excellent performance, recent research shows that DNNs are vulnerable to ad adversarial samples (Dvorsky (2019)) of which the difference is unnoticeable for humans, but easily leading the DNNs to wrong predictions. This vulnerability hinders DNNs from applying in many sensitive areas, such as autonomous driving, finance, and national security.
To eliminate the impact of adversarial samples, researchers have proposed number of techniques to help DNNs detect and prevent adversarial attacks. Existing adversarial defense techniques could be classified into two main categories: (1) adversarial robustness model retraining (Tramer et al. 2017);; Ganin et al. (2016); Shafahi et al. (2019)) and (2) statistical-based adversarial samples de- tection Grrsse al. 2017 Xu et al. (2017); Meng & Chen (017)). However, while adversarial model retraining improves detense abilities, it also leads to huge costs during retraining process, es- pecially when the number of the parameters in current models grows larger and larger again. As for statistical-based adversarial samples detection techniques, one severe shortcoming is that all these techniques require prior knowledge about the adversarial samples, which is not realistic most real- world cases. For example, LID (Ma al. (2018)) and Mahalanobis (Lee et al. (2018)) need to train logic regression detectors on validation datasets. To make matters worse, adversarial attacks and defenses just like the relationship between spear and shield. Defensive techniques that perform well against existing attacking methods will always be bypassed by new attack mechanisms, which makes defending zero-day attacks challenging but urgent task.
To address this challenge, we propose AttackDist, an attack- aggnssic adversarial sample de- tection technique via counterattack. Our method is based insight that, from the perspective of

---

Under review as a conference paper at ICLR 2021
optimization theory, the process of searching adversarial perturbations is non-convex optimization process. Then the adversarial perturbations generated by the attack algorithm should be close to the optimal solution 8 (See Definition 1). Due to the property that optimal solution 8 is close to the decision boundary (Lemma 1). Thus, if we apply the counter attack on adversarial samples, the perturbation would be significantly smaller the original samples. Figure shows example of intuition, if we attack an adversarial sample, then the adversarial perturbation d2 would be much smaller than the adversarial perturbation attacking normal point d1. Thus by measuring the size of adversarial perturbation, we could differentiate normal points and adversarial samples.
To demonstrate the effectiveness of AttackDist, we first analyze the norm of adversarial perturbation for normal points and adversarial points theoretically, and give the conditions under which AttackDist could provide guaranteed detecting per- formance (Theorem 3). In addition to theoretical analysis, we also im- plement AttackDiss st two fa- mous and widely-used benchmarks, MNIST (Dengl (2012)) and Cifar- 10 (Krizhevsky et al.). and com- pare with four state-ol-the-- -her--rrtt tech- niques, Vinalla (Hendrycks & Gim- pel (2016)) KD (Feinman al. (2017)) MC(Gal & Ghahramani (2016) and Mahalanobis (Lee al. (2018) The experimental results show that AttackDiss performs better than existing works
normal point x from attacking
from attacking
optimal solution of equation
r
dp ad edveririial perturbution
d1
d: adv ersarial perturbation for
attack radius
d2
decision boundury
Figure 1: An example of our intuition.
in detecting zero-day adversarial attacks without requiring the prior-knowledge about the attacks.
In brief, we summarize our contributions as follows:
We formally prove general instinct property of adversarial samples (i.e., adversarial sam- ples are close to the decision boundary), which could be leveraged for detecting future advanced (less noticeable) adversarial attacks. And with more unnoticeable attacks, this property would contribute more to adversarial sample detection.
We propose AttackDi st, an attack-agnostic technique detecting zero-day adversarial attacks. We thcoretically prove when the adversarial perturbation satisfies the given condi- tion, AttackDist could have guaranteed performance in detecting adversarial samples.
We implement AttackDist two widely used datasets, and compare with four state- of-the-art approaches, the experiment results show AttackDiss could achieve the state of the performance in most cases. Especially for detecting & adversarial attacks, At AttackDist could achieve 0.99, 0.98, 0.96 AUROC score and 0.99, 0.92, 0.90 Accu- racy for tree different adversarial attacks.
BACKGROUND
In this section, we first define the notations used through the paper, then give brief review to adversarial attack and adversarial defense. Finally, we introduce our assumptions about the attackers and the defenders.
2.1 NOTATIONS
Let f(.) X y denote continuous classifier, where X' is the input space consisting of d- dimensional vectors, and y is the output space with labels. The classifier provides prediction on point based arg max,=1,2.... 1,2... fx). We then follow O to define adversarial perturbations. Let A(-) denote specific attack algorithm (e.g., FGSM, CW). As shown in Equation [ given point x and target classifier f. the adversarial perturbation A (x. f) provided by A(-) is a minimal perturbation that is sufficient change the original prediction f(x) (for shorthand, use A(x) to
2

---

Under review as a conference paper at ICLR 2021
represent (a,, f) throughout the paper).
= ming ||1||
s.t. f(x + 6) + f(x)
(I)
Adversarial samples are the points that applying the adversarial perturbations on the original points (i.e., Tady x + (z)))
Definition 1. Attack Distance: We define attack distance (AttackDist) of a point x as lp norm of the adversary perturbation.
AttackDist(r)) = ||Fade =|1p = 1|ا()))ا
Definition 2. Optimal Adversarial Perturbation: Given and f. the optimal adversarial pertur-
bation 8 (z) the most optimal solution of Equation [ In other words, 8* (z) satisfy Equation
3
(x)(x S [|A(r))l s.t. f(x A(x)) + f(x) A f(x + (x)) + f(x)
(3)
Definition 3. Optimal Adversarial Sample: Given x and f.we define the optimal adversarial sample
x² = x (z), that is applying the optimal adversarial perturbation 8* (x) on normal point (Note
x² is not a constart point, a function of x)).
Definition 4. Decision Boundary: We define the decision boundary B of classifier f as the collection of points which have the same prediction on different labels. More specifically, satisfy Equation 4
B = {xr J (1 S i.j j s K) A (i [K=(12...,Kxx)) S f+(x)= = {イ(r)}
Then let (xx, f) = minseB ||> (shorthand as (x))) denote the minimal distance from point to the decision boundary. And we define all points on the decision boundary are adversarial samples. Because according to the definition of decision boundary B, any points belong to B would provide more than one prediction results, which means it contains at least one prediction is contradict with the ground truth.
Lemma 1. The optimal adversarial sample x² belongs to decision boundary B, in other words, the relationship between 8 (x) and (xx) is 11ō (x))l> D(x).
We prove lemma by contradiction, assume the optimal adversarial samples xt does not belong to B (e.g., x² & B), then we want to prove x² is not the most optimal adversarial samples generated from x (i.e., there exists x, satisfies Equation
||x, x < ||2" =xlpp
s.t. f(x)= + f(x.)
(5)
Proof. Let f(x) i, f(x'xx = and + j. we connect the point x and xx to get line, then there must be a point on the line satisties Equation 5 We prove it by constructing the function g(x) - f()x ff (z), obviously, g(x)
and g(x") < O. Due to the continuous of f. from x to x", need to show that point P is the point we want (i.e. P = x.). Obviously, P xllp < ||x- - zIlp because P is a middle point of straight line with and x² endpoints, then we only need to prove P would get different prediction with x. There are two conditions of the prediction on point P: (1)
there exists point (P) = 0 and P + x- AP # Then,
= i; (2) S(P) & i. For condition 1, due to g( g(P) 0 fP( and f(P) i, then P satisfies the definition of decision boundary B. so woull have a different prediction with æ (We define all points on the decision boundary are adversarial samples). For condition 2, f(P) + i f(x). obviously, P is the adversarial sample for rr
Next, we introduce the definition of r-attack, to measure the the optimization capabilities of an attack algorithm. Although the definition of adversary samples is to optimize the Equation [] but none attack algorithm could always obtain the most optimal solution 8 (z)). We define r-attack, to measure how close the adversarial samples generated by one specific attack algorithm to the optimal solution 8*.
Definition 5. r-attack: we define attack algorithm A,(-) as an r-attack algorithm if all perturbations produced are lying in a sphere centered on optimal adversarial perturbation with radius r.
A, is r-alacck
Vx E X
||((x) - 6t (x)]lp S
(6)
From the definition of r-attack, we could see more advanced attacks (more unnoticeable attack) are attacks with less r. The best attack could always produce 8* (z)), whose r = As the goal of the attackers is to create less noticeable samples to evade the Human-beinge Then they tend to develop the more advanced attack algorithms with smaller r. Later, we would show how AttackDist leverage this point (Theorem 3) to detect more unnoticeable attacks.
3

---

Under review as a conference paper at ICLR 2021
2.2 ADVERSARIAL ATTACKS & ADVERSARIAL DEFENSES
Many existing works have been proposed for crafting adversarial examples to fool the DNNs, we introduce selection of such work here. The Fast Gradient Method (Goodfellow al. (2014a) search the adversarial samples by a small amount along the direction of gradients. The CW (Carlini & Wagner (2017) attack, model the adversarial samples generation as optimization problem and iteratively search the optimal solution. And the Deep Fool (Moosavi-Dezloon et al. (2016) attack, which is designed estimate the distance of one sample to the decision boundary.
We then follow the definition of zero-day vulnerabilities (Ablon & Bogart (2017)) to define zero-day adversarial attacks. zero-day adversarial attack one attack algorithm that is unknown to those who should be interested in mitigating the attacks (e.g,, the adversary sample detectors).
Besides the adversarial attack techniques, a number of defense techniques also have been introduced to reduce the harms of adversarial samples. For example, KD (Feinman et al. (2017)) estimate the kernel-density the training dataset and use the estimated kernel-density to distinguish normal samples and adversarial samples. LID, which estimate the local intrinsic dimensionality of normal, noisy and adversarial samples, and train a logic regression detector to characterize the subspace of adversarial samples. However, LID needs the prior-knowledge of adversarial attacks train the detectors, thus can not be applied for detecting zero-day adversarial attacks.
2.3 THREAT MODEL
In this paper, we assume the attackers could complete access to the neural networks and could apply white-box attacks. For the detectors, they could know some attack algorithms, but when new attack proposed, the detectors don't 't know anything about the mechanism of the new proposed attacks.
APPROACH
Our aim to gain intrinsic properties of adversarial perturbations, and derive potential provide new directions for new advanced attacks. We begin by providing a theory analysis of the bounds of the boundary distance (AttackDist) of r-attack adversarial samples. After that, we show how AttackDist could be efficiently estimated through applying counter attack. Finally, show why AttackDist could differential normal samples and adversarial samples; and the condition, under which AttackDist could have certificated detection performance.
3.1 ATTACKDIST OF ADVERSARIAL SAMPLES
Let æ is normal input, we first apply algorithm Ar1 to attack x to generate adversary sample y, and apply a different algorithm A to attack y to generate adversary sample 2 (i.e., y x + A (xr), 2 = y + (z)) We first provide our motivation by analysing the attack distance of and y.
lly =|1p (.) > (x) D(x)
(x) < (z, 8" (x)]l, + (x, fllp Sn + D(x)
In the first line of Equation 7 we measure the lower bound of the adversarial perturbation. The (x)llpp is due to the definition of 5'(x:) (See Definition 2 and
first inequality (x)
the second equality (x) = DDx) is due to Lemma [ In the second line of Equation 7 we
measure the upper bound of the adversarial perturbation. The first inequality due the trangle
inequality (i... ||A + BlIp < ||.A|pp + (pp).. Then because of the definition of r-attack, we have
[[A/(()) -- (z)llp S rı. We then assume the random variables ((x) for normal points belongs to a Gaussian distribution.
((x) ~ ((,, Var X
After the analysis the bound of the adversarial perturbation for normal point x, we then analysis the bound of the adversarial perturbation for adversarial sample
Theorem 2. If y is a adversary sample generated by x through ri -atttck,, and z is the adversary sample generated by y through ro-ottack then D(y) S and ||= vllp S r2
4

---

Under review as a conference paper at ICLR 2021
D(v) min(y, B) < 1ly > S || uulpp ( S r2 + D(y)= S r + r2
(8)
Proof. As shown in the first line in Equation g the distance of y decision boundary B is the minimum distance of y to any points in B. And x" is the points belongs to the decision boundary
B (Lemma I.. then min(y (y, B) S Ily==* Ipp- And according the definition of r-attack, ||y-~= Ib holds. (v) & r2- + (1) in second line of Equation is because the second line of Equation we just replace with y, and r1 with r2.
Theorem 3. If we have known attack with Ti S 2 3o), where μ and are the parameters of the Gaussian Distribution for D(x). Then for any advanced attacks (less noticed attack) with r2 rı, we have 99.86% probability that using the attack distance could correctly distinguish normal samples and adversary samples.
Proof. Combining Equation and 8 The lower bound for |yy =11p = D(x) ~ (N, σ). and the upper bound for | - vllp r1 + r2. r S j/u 1 (())) and r2 S ri, then rı + +22 S 2r S 1 1--3-- 30. According to the cumulative distribution function (CDF) of Gaussian Distribution, the probability of D(x) S 1--3 is less than O. 0.14% In other words, the lower bound of [1y ~I11p have the 99.86 99.86% probability larger than the upper bound of ||z y/lp. which means it at least have the 99.86% detection accuracy.
3.2 USING ATTACKDIST TO CHARACTERIZE ADVERSARIAL EXAMPLES
We next describe how AttackDist can serve as property to distinguish adversarial examples without the prior-knowledge about the zero-day attacks. Our methodology only requires one known attack algorithm Aknown for implementing the counter-attack, There are two main steps to calculate At SttackDist.
1) Applying counter-attack: for the point x under detection, we first attack with the known attack algorithm Aknown to generate y = x + Aknowm (I)).
2) Atta ckDist Estimation: We estimate AttackDist of point by measuring the norm of the adversarial perturbation ||y 105-x1) zl|p.
4 EVALUATION
In this section, we demonstrate the effectiveness of our method in distinguishing adversary sam- ples generated by three attack algorithms two widely used datasets. Our code is available at http://gitubb.com/moommoozz /ttta//kkDst
4.1 EXPERIMENTAL. SETUP
(012)) and CIFAR- 10 (Krizhevsky al.) datasets. We use the standard DNN model for each dataset. For MNIST choose LeNet-3 (LeCun et (2015)) architecture which reaches 98.6% accuracy on the testing set. On CIFAR-IO, train a ConNet (Carlini & Wagner (2017)) with 87.8% accuracy. The details of the model and the training setup could be found in Appendix
Datasets and Models: We evaluate our method on MNIST (Deng
We generate adversarial examples with white-box attack methods. Specifically, We consider three different attack algorithms for both & and fox bounded adversarial examples. The selected attack algorithms include (see also references within):
Attack Algorithms:
E bounded adversarial attacks
BrendelBethgcAtttack (BB) (Brendel et al. 2019))
Carlini Aatte (CW) (Carlini & Wagner (0017)
DeepFoolAAtack (DF) (Moosavi-Deztooli et 2016)
lxx bounded adversarial attacks
5

---

Under review as a conference paper at ICLR 2021
ProjettelCradieeDDeceetttttttt (PGD)
(aadry al. (2017))
BasiclterativeAAtteel (BIM) (Kurakin et al. (2016))
FastGraddienSSinnAttackk (FGSM) (Goodfellow et al. (014b)
The reason we select completely different attack algorithms for E and foo bounded adversarial examples that these algorithms are designed for different t norm purpose.
Evaluation Metric: We first point out that comparing detectors just through accuracy is not enough. For adversary samples detection, we have two classes, and the detector outputs a score for both the positive and negative classes. If the positive class is far more likely than the negative one, a detec- tor would obtain high accuracy by always guessing the positive class, which can cause misleading results. To address this issue, besides accuracy. we also consider four different metrics. we consider the trade-off between false negatives (FN) and false positives (FP), the trade-off between preci- sion and recall and the trade-off between true negative rate (TNR) and true positive rate (TPR), and employ Area Under the Receiver Operating Characteristic curve (AUROC), Area Under the Frecision-Recall curve (AUPR), TNR at 90% true positive rate TPR (TNR@90) and TNR 99% true positive rate TPR (TNR @ 99) as our evaluation metrics.
TNR @90 Let TP, TN, FP, and FN denote true positive, true negative, false positive and false negative, respectively. We measure TNR = TN / (FP+TN), when TPR is 90%.
TNR @99 We also measure TNR, when TPR is 99%.
AUROC is threshold wresholddindependenn metric. The ROC curve shows the true positive rate against the false positive rate. A "perfect" detector corresponds 100% AUROC.
AUPR is also threshold-independeen metric. The PR curve plots the precision and recall against each other. A "perfect" detector has an AUPR of 100%.
Accuracy We enumerate all possible thresholds 7 on the test dataset and select the best accuracy for evaluation.
Comparison Baseline:
There are many existing works could defense the adversarial attacks. How- ever, as we discussed earlier, some of them need prior knowledge about the attacks to train the detector, as our goal to detect zero-day adversarial attacks without the prior-knowledge, then we only consider four different approaches that require no prior knowledge of the attacks as baselines. We briefly introduce each baseline, more details about the comparison baselines could be found in related works (Hendrycks & Gimpel (2016); Feinman et al. (2017): Gal & Ghahramani (2016); Lee al. 2018)).
Vanilla (Hendrycks & Gimpel (2016)): Vanilla is a baseline which defines confidence
score as maximum value of the posterior distribution. Existing works also find it could be used to detect adversary samples.
KD (Feinman et al. (2017)): Kernel Density (KD) estimation is proposed to identify ad- versarial subspaces. Existing works () demonstrated the usefulness of KD-basee adversary samples detection, taking advantage of the low probability density generally associated with adversarial subspaces.
MC(Gal & Ghahramani] (2016)): MC Drop represeent the model uncertainty for a specific input activate the dropout layer in the testing phase.
Mahalanobis (Lee et al. (018)): Mahalanobis using Mahalanobis distance on the fea- tures (the output in the hidden layer) learned by the target DNNs to distinguish adversarial samples, it is an approach based uncertainty measurement.
For the baseline KD, needs to tune the hyperparameters for computing. We follow (Ma et al. (2018)) to set the optimal bandwidths chosen for MNIST, CIFAR- 10 as 3.79 and 0.26, respectively. As for MC, we activate the dropout layer and run 300 times. For Mahalanobis, it need selects the features in the hidden layer to Gaussian model. For MNIST, we select features before the last fully connected layer, and for CIFAR-10, we select the last two layers.
Experiment Process:
Since our approach is based on counterattack, we need to use a known attack algorithm during the implementation, which is easy meet because there are variety of open-source attack algorithms.

---

Under review as a conference paper at ICLR 2021
Normal CW
Normal
5
CW DF
2.5
B6
AttackDist
2
Density
1.5
0.5
0
50
100
150
200
0
ID
AttackDist
Figure 2: The Distribution and Probability density function of At AttackDist of two hundreds ran- dom selected data
Table 1: The Experiment Results of le norm Attacks On CIFAR-10 Dataset
Artack
Roselines
DF
TS
DF
ATROC.
DC
AUPRR
Vinalla
0.01
0.50
0.50
MC
0.00
0.00
0.01
0.97
0.97
0.50
0.50
0.00
0.00
0.01
0.72
0.37
0.37
0.50
0.50
Ma
0.05
0.05
0.05
0.05
0.16
0.15
0.16
0.16
0.51
0.51
0.51
0.51
0.49
0.49
0.49
0.49
0.53
0.53
0.53
0.53
oum
0.64
0.76
0.80
0.72
100
1.0
1.00
0.99
0.99
0.99
0.99
1.00
0.99
0.98
0.99
0.99
Vinalla
0100
000
010
000
012
002
012
0.2
034
0134
0194
0354
0.32
0.32
0.32
0.32
0.50
0.30
0.50
0.30
MC
0.00
0.00
0.00
0.00
001
0.01
0.11
0.01
0.6
0.96
0.96
0.96
0.32
0.32
0.32
0.32
0.50
0.50
0.50
0.50
CW
KD
0.00
0.00
0.00
0.00
0.02
0.02
0.02
0.02
0.70
0.70
0.70
0.70
0.37
0.37
0.37
0.37
0.50
0.50
0.50
0.50
Ma
0.14
0.66
0.04
0.04
0.16
0.16
0.16
0.16
0.51
0.51
0.51
0.51
0.50
0.50
0.50
0.50
0.53
0.53
0.54
0.53
0185
0.63
0.75
0.74
0.70
0.5
0.94
0.54
0.94
0.98
0.98
0.58
0.98
0.98
0.58
0.98
0.59
0.92
0.92
0.92
0.22
Vinzila
0.00
0.00
0.22
0.2
0.3
0.55
0.32
0.32
0.30
0.30
MC
0.00
0.00
0.11
0.01
0.94
0.94
0.32
0.32
0.50
0.50
0.00
0.00
0.05
0.68
0.38
0.38
0.50
0.50
0.05
0.54
0.54
0.96
0.90
Vinalla
075
MIL
0.00
0.96
0.6
0.55
0.75
KD
0.00
0.00
0.00
0.00
0.2
0.02
0.70
0.70
0.6
0.63
0.75
0.75
0.05
0.04
0.16
0.6
0.16
0.51
0.51
0.51
0.75
0.75
0.75
0.75
0.76
0.76
0UES
0.54
0.51
0.58
0.55
0.9
0.98
0.98
0.98
0.97
0.98
0.58
0.98
0.99
0.99
0.99
0.99
0.93
0.9
0.94
0.93
In our experiments, we literally treat one attack algorithms as known attack algorithm, and the other two algorithms the zero-day attacks to generate adversary samples, then evaluate whether At tatakkisst st could detect the generated zero-day adversary samples. For example, we use PGDM the known attack and treat zero-day adversarial samples for evaluation. The detail implementation of our attack and attack success rates can be find in Appendix
approach's input, then use the rest attack algorithms generate
4.2 BOUNDARYDDSTT CHARACTERISTICS OF ADVERSARIAL SAMPLES
We provide empirical results showing the AttackDist characteristics of adversarial samples crafted by the mentioned attacks. We use CW attack algorithm to counter attack the adversarial sam- ples generated by CW, DF, BB and the normal samples, and measure the AttackDi i st to show how AttackDist could distinguish normal samples and adversarial samples. The left subfigure in Figure 2 shows the AttackDist of 200 randomly selected normal, and adversarial examples from the MNIST dataset. Left figure shows the & norm attack and right figure shows the fif norm attack. Red circle points represent the normal points, while different color square points represents the different adversarial samples. We observe that AttackDist scores of adversarial examples significantly smaller than those of normal examples, especially for the lin norm attacks. This supports our expectation that the perturbation to counterattack adversarial samples would be sig- nificantly smaller than normal cases. The right subfigure in Figure 2 shows the probability density function (PDF) of normal, and adversarial examples. Clearly, the distribution of normal samples and adversarial smaples are totally different. The different PDFs suggest that by selecting property threshold, At AttackDis st could correctly detect the adversarial samples.
4.3 EXPERIMENTAL RESULTS
Due to the limit of space, we only present the results for CIFAR-10, the results about MNIST could be found in Appendix C
7

---

Under review as a conference paper at ICLR 2021
4.3.1 Eg ATTACK
Table shows our experimental results to detect E2 norm adversarial attacks on CIFAR-I0 dataset. For almost all cases, our approach outperforms the baselines for great margins, especially for N9R@99, when the requirement of TNR is 0.99, the performances of baselines are almost zero, which means the existing works fail to detect the new adversarial attacks without the prior- knowledge. However, AttackDi could still have a a 0.54, 051, 0.58 NTR for mixed adversarial attacks. As for the metric NTR @90, which is a slightly loose requirement than NTR99. At this sce- nario, the performance of the baseline is no-longer zero, however, they still have a poor performance, while At tackDist almost have perfect performance with 0.93, 0.94, 0.94 for mixed attacks. Another interesting finding is that AttackDist almost keep the same performance with different attack algorithms choose implementing the counter-attack. This means At AttackDist is not sensitive with the adversarial attack algorithms for counter-atten -annee-.
4.3.2 Lo ATTACK
Table 2: The Experiment Results of &o norm Attacks On CIFAR-10 10 Dataset
Attack
Biselines
155
Cw
TNR699
111
A5c
155
CW
TNR
11
Axe
155
CW
AUROC
1/
Axe
155
CR
AUPR
Axe
155
CW
Acc
1X1
Axe
Vinalla
0.02
0.02
0.02
0.02
0.19
0.19
0.19
0.19
0.37
0:57
0.57
0.37
0.53
0.53
0.33
0.53
0.57
0.57
0.57
0.57
MC
002
0.02
0.02
0.02
0.18
0.18
0.18
0.18
0.5%
0.58
0.5%
0.5%
0.55
0.55
0.55
0.55
0.57
0.57
0.57
0.57
8B
K.D
0.00
0.00
0.00
0.00
0.08
0.08
0.08
0.08
0.54
0.33
0.54
0.53
0.47
0.47
0.47
0.47
0.30
0.50
0.50
0.30
Ma
002
0.01
0.01
0.01
0.10
0.10
0.09
0.10
0.51
0.50
0.50
0.50
0.51
0.51
0.50
0.51
0.53
0.53
0.52
0:52
ours
0.06
0.05
0.19
0.10
0.18
0.17
0.19
0.18
0.56
0.55
0.54
0.54
0.45
0.45
0.45
0.45
0.58
0.55
0.55
0.55
Vinaila
0016
0106
000
0106
0.2
0.9
0.29
0.9
0.61
0127
0007
0.63
0063
074
0104
UTA
01:4
MC
0.04
0.04
0.04
0.04
0.25
0.25
0.25
0.25
0.65
0.65
0.65
0.65
0.60
0.60
0.60
0.60
0.63
0.63
0.63
0.63
CW
KD
0.00
0.00
0.00
0.00
0.07
0.07
0.07
0.07
0.59
0.59
0.59
0.59
0.3
0.43
0.43
0.43
0.50
0.50
0.50
0.50
0.01
0.01
0.01
0.1
0.08
0.06
0.07
0.50
0.50
0.50
0.54
0.53
0.55
0.55
0.53
MC
0.00
0.68
0.30
0.50
0.50
0.46
0.50
0.01
0.02
0.11
0.13
0.2
0.51
0.8
0.47
0.48
0.22
ouss
0.06
0.10
0.35
0.70
0.70
0.64
0.64
0.65
Vinalla
0.01
0.01
0.01
0.01
0.10
0.10
0.10
0.10
0.2
0.2
0.32
0.32
0.76
0.76
0.76
0.76
0.75
0.25
0.75
0.75
MC
0.01
0.01
0.01
0.01
0.09
0.09
0.09
0.09
0.32
0.2
0.32
0.32
0.76
0.76
0.76
0.76
0.75
0.75
0.75
0.75
Mix
KD
0.00
0.00
0.00
0.00
007
0.77
0.77
0.07
0.56
0.56
0.56
0.56
0.11
0.7
0.71
0.71
0.75
0.75
0.75
0.75
Ma
0.01
0.01
0.01
0.01
0.10
0.00
0.10
0.10
0.50
0.50
0.50
0.50
0.76
0.75
0.76
0.75
0.75
0.75
0.75
0.55
018
0.06
0.04
0.19
0.10
0.16
0.17
0.20
0.18
0.51
0.51
0.52
0.51
0.7
0.75
0.75
0.75
0.76
0.76
0.73
0.75
Table 2 shows our experimental results to detect fo norm adversarial attacks on CIFAR-10 dataset. The performance of detecting foe norm adversarial attacks is much worse than &g attacks. However, At attackDist still achieve a competitive performance, one possible reason that AttackDis can not have good performance as &g attacks is the condition in Theorem 3 is no longer hold for fxc attacks. The existing works (Carlini & Wagner (017)) studied the size of adversarial perturbation for lz and for attacks. On CIFAR-I0 dataset, &oo 0.013 is enough achieve the average 100% success attack rate, while E needs to be larger than 0.33. However, consider the different maximum distance on f2 and foo norm (i.e., the maximum l2 norm for CIFAR-IO is 32 32 3 = 3072 while the maximum lo is 1). Then the relative r for &g norm attacks would be smaller, which means the lg could produce more unnoticeable adversarial samples.
DISCUSSIONS AND CONCLUSIONS
In this paper, we proposed At ackDist to address the challenge of detecting zero-day adversarial attacks. From the perspective of optimization theory, we try to understand the general intrinsic prop- erties of adversarial samples rather than statistically analysis the hidden feature of existing adver- sarial samples. Trough counter attack the normal samples and the adversarial samples, we analysis the norm of the adversarial perturbation of normal samples and adversarial samples theoretically, and give the condition under which AttackDist would have a guaranteed performance for any advanced attacks. In particular, At tttack performs better than the existing works in detecting zero-day adversarial samples.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Lillian Ablon and Andy Bogart. Zero days, thousands of nights: The life and times of zero-day vulnerabilities and their exploits. Rand Corporation, 2017.
Wieland Brendel, Jonas Rauber, Matthias Kümmerer, Ivan Ustyuzhaninov, and Matthias Bethge. Accurate, reliable and fast robustness evaluation. Advances in Neural Information Processing Systems, pp. 12861-12871, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pp. 39-57. IEEE, 2017.
Edward Choi, Andy Schuetz, Walter Stewart, and Jimeng Sun. Using recurrent neural network models for early detection of heart failure onset. In Journal of the American Medical Informatics Association, pp. 361-370, 2016.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-142 2012.
George Dvorsky. Hackers have already started to weaponize artificial intelligence. 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint (703.004 10, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. Domain-adven adversarial training of neural networks. The Journal of Machine Learning Research, ((1):2099-20300 2016.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint (7XXiv:/702.000 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and nut-of-distribution examples in neural networks. arXiv preprint arXiv:1610.021366 2016
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). URL http: es toronto..o...o..o www.:kriz/cifar.aa.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprini 1oXXx:6607002533 2016.
Yann LeCun et al. Lenet-5, convolutional neural networks. URL: http://yann. lecun. com/exdb/lenet, 20(5):14, 2015.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of duttof--istribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp. 7167-7177, 2018.
Xingjun Ma, Bo Li, Yisen Wang. Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. arXiv preprint 1 1801.0261, 2018.

---

Under review as a conference paper at ICLR 2021
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0688. 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 135-147, 2017.
Seyed-Mohsen Moosavi-Dezfoud Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference com- puter vision and paltern recognition, pp. 2574-2582, 2016.
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensor- flow, and jax. Journal of Open Source Software, 5(53):2607, 2020. doi: 10.21 10.21105/joss.02607. URL https:://oi.org: 10. 21105/ joss 02607
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3358-3369, 2019.
Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. Machine translation using deep learning: An overview. In 2017 International Conference Computer, Communications and Electronics (Comptelix), 2017.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Me- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXi::1705.0704. 2017.
Zoltan Tüske, Pavel Golik, Ralf Schlüter, and Hermann Ney. Acoustic modeling with deep neural networks using raw time signal for Ivesr. 2014 Annual Conference of the International Speech Communication Association, 2014.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.011 0555, 2017.
A MODEL ARCHITECTURE AND TRAINING SETUP
Table 3: The Model Architecture we used in our experiments
Layer Type
MNIST
CIFAR-10 10
Convolution + ReLU
5 x x
3 x 3x 64
Convolution + ReLU
3 x 64
Max Pooling
2
2 x 2
Convolution + ReLU
5 5
128
Convolution + ReLU
3 x 3 x 128
Max Pooling
2
2 x
Fully Connected ReLU
128
128
Fully Connected + ReLU
128
softmax
10
10
The model architecture used is listed in Table for MNIST and CIFAR-10 10. the training dataset 50,000 and we randomly select 1,000 from the testing dataset for evaluation. We set the learning rate 0.01, with the momentum is 0.9.
10

---

Under review as a conference paper at ICLR 2021
B ADVERSARIAL SAMPLES GENERATION
We implement the listed attack algorithms through the python library foolbox (Rauber et al.] (2020), which is popular library for evaluating the robustness of DNNs. For MNIST dataset, we set hyper-parameter
as 3 and 0.25 for l and lint: for CIFAR-10 dataset, we set e as 0.35 and 0.015 for &g and lmt. The attack success rates for each attack algorithms are listed in Table
Table Attack Success Rate for Each Attack Algorithms
fg
Lin BIM
Dataset
BB
CW
DF
PGD
FGM
MNIST
1.000
1.000
0.997
0.983
0.953
0.773
CIFARIO
0.978
0.981
0.908
0.943
0.967
0.811
C EXPERIMENTAL RESULTS FOR MNIST BENCHMARK
Table 5: Results for MNIST on L2
Attack
Biselines
BB5
CW
TNR
DF
Aue
CW
DF
Axe
B13
CW
AUROC
DE
Axe
BB
CW
AUPR
DD
Axe
BB5
CW
Ace
Ave
Vinalla
0.00
0.00
0.00
0.00
0100
0.00
0.30
0.00
1.00
1.00
1100
1.00
0.31
0.31
0.31
0.31
0.30)
0.30
0.30
0.30
MC
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.96
0.96
0.56
0.96
0.31
0.31
0.31
0.31
0.50
0.50
0.30
0.50
8B
K.D
0.30
0.00
0.00
0.30
0.00
0.00
0.00
0.00
0.98
0.98
0.98
0.59
0.31
0.31
0.31
0.31
0.50
0.50
0.50
0.50
Ma
0.18
0.18
0.18
0.18
0.43
0.45
0.43
0.43
0.69
0.69
0.69
0.69
0.61
0.61
0.61
0.61
0.67
0.67
0.67
0.67
Inal
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Vinalla
0.00
0.00
0.00
0.00
100
100
0.31
0.31
0.31
0.00
0.30
ML
0.00
0.00
0.00
0.00
0.56
0.96
0.31
0.31
0.50
0.30
0.00
0.00
0.50
0.50
0.70
0.62
0.62
0.67
1.00
1.00
Vinalla
0000
0100
1330
1300
030
030
MC
0.00
0.00
0.00
0.00
0.96
0.96
0.31
0.50
0.50
DP
KD
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.98
0.98
0.58
0.98
0.31
0.31
0.31
0.31
0.50
0.50
0.50
0.50
Ma
0.19
0.19
0.19
0.19
0.12
0.42
0.42
0.42
0.68
0.68
0.68
0.68
0.60
0.60
0.60
0.60
0.66
0.66
0.66
0.66
tool
0.9%
1,00
1.00
0.99
1.00
1.00
1.00
1.00
1.00
1.000
1.000
1.00
1.00
1.00
1.00
1.00
0.9
1.00
1.00
0.99
Vinalla
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
1:00
100
1:00
1000
0.54
0.54
0.54
0.54
0.75
0.75
0.75
0.75
MC
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.66
0.56
0.56
0.96
0.55
0.55
0.55
0.55
0.75
0.75
0.75
0.75
Mix
KD
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.%
0.9%
0.58
0.9%
0.54
0.54
0.54
0.54
0.75
0.75
0.75
0.75
Ma
0.19
0.19
0.19
0.19
0.43
0.43
0.43
0.43
0.69
0.69
0.69
0.69
0.2
0.22
0.82
0.82
0.00
0.00
0.80
0.80
[nol
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.99
1.00
1.00
1.00
Table 6: Results for MNIST on Linf
Attack
Reectines
BR
CW
TNR699
DF
Ave
BR
CW
DF
Ave
BR
CW
DF
Ave
BR
CW
DF
Ave
BR
CW
DF
Ave
Vinalla
0.02
0.02
0.02
0.02
0.14
0.14
0.14
0.14
0.58
0.88
0.58
0.38
0.57
0.77
0.57
0.37
0.38
0.38
0.58
0.38
MC
0.01
0.01
0.01
0.01
0.16
0.16
0.16
0.16
0.55
0.55
0.55
0.35
0.51
0.51
0.51
0.51
0.56
0.56
0.56
0.56
88
KD
0.00
0.00
0.00
0.00
0.07
0.07
0.07
0.07
0.64
0.64
0.64
0.64
0.401
0.40
0.40
0.40
0.50
0.50
0.50
0.50
Ma
0.03
0.03
0.00
0.02
0.19
0.19
0.07
0.15
0.57
0:57
0.57
0.57
0.56
0.56
0.44
0.52
0.56
0.56
0.50
0.54
tnal
001
0.05
0.33
010
0.67
0.63
0.3
0.51
0.8
081
0.1
0.78
0.73
0.3
0.64
0.70
0.80
0.79
0.71
0.77
Vinalla
003
0.0
003
003
016
016
0.16
0.16
0.59
0.39
0.39
0.59
0.58
0.38
0.58
0.55
0.38
0.55
0.58
0.35
MC
0.13
0.65
0.03
003
0.19
0.19
0.19
0.19
0.56
0.56
0.56
0.56
0.53
0.53
0.53
0.53
0.57
0.57
0.57
0.57
CW
KD
0.00
0.00
0.00
0.00
0.07
0.07
0.07
0.07
0.64
0.64
0.64
0.64
0.39
0.39
0.39
0.39
0.50
0.50
0.50
0.50
0.0
0.19
0.15
0.57
0.57
0.56
0.52
0.56
0.54
[oo]
0.0
0.10
0.36
0.38
0.77
0.75
0.70
0.68
0.77
0.75
Vinalla
0.00
0.5
0.55
0.50
0.00
0.00
0.9
0.32
0.30
0.11
0.02
0.25
0.11
0.31
0.46
0.38
Inol
0.11
0.5
0.10
0.7
0.7
0.55
091
391
092
0.2
038
037
Vinalla
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.77
0.57
0.37
0.77
0.4
0.4
0.74
0.74
075
0.75
0.75
0.5
MC
0.00
0.00
0.00
0.00
002
0.02
0.02
0002
0.57
0.7
0.57
0.57
0.72
0.22
0.72
0.72
0.75
0.75
0.75
075
Mix
KD
0.80
0.00
0.0
0000
001
001
001
0.1
0.7
073
0.33
0.7
062
066
0
0.6
0.75
0.75
0.75
0.75
Ma
0.02
0.02
0.00
002
0.22
0.22
011
018
0.54
0.54
0.54
0.54
0.76
0.76
0.11
0.74
0.75
0.55
0.75
0.75
too]
0.01
0.05
0.23
0.10
0.62
0.63
0.33
0.49
0.4
0.8
0.77
031
0.2
0.2
0.90
0.2
0.85
0.84
0.78
0.8


Reviews:
[{"review_id": "nv8uxq46l1A", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: this paper is about adversarial detection based on counter-attack. The main intuition is that the adversarial sample lies closer to the decision boundary and hence if we do counter-attack to the data, the perturbation is expected to be much smaller than the clean data.\n\nAlthough the idea sounds interesting, I have a few questions to be answered:\n1. in eq(1) you already defined $\\Delta$ as the minimum distortion, how come it is larger than $\\delta^*$ in eq(3)?\n2. you assumed the $\\| y-x \\|_p=D(x)\\sim N(\\mu, \\sigma^2)$, have you verified this empirically?\n3. since $D(x)$ is obtained with optimal perturbation, i.e. $\\|\\delta^*\\|$, how do you obtain the $\\mu$ and $\\sigma$?\n4. from theorem (3) it seems that this detection method works for advanced attack, which requires $r_2 < r_1$, what if the attacker is a bad algorithm causing adversarial samples far from the boundary?", "extra_scores": {}, "extra_text": {}}, {"review_id": "XVgLpE5Sb35", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to detect adversarial examples. The detection\nscheme is based on the observations that typical adversarial attacks generate\nadversarial examples on the decision boundary, so if we use a \"counter attack\"\non the adversarial example, it will be easy to change its label.\n\nA main weakness of this paper is that the proposed approach does not include an\nadaptive attack for evaluation.  If the proposed detection scheme is known to\nthe attacker, the attacker can still generate visually indistinguishable\nadversarial examples that the detector fails to detect. This can usually be\ndone by adding the detection objective to the loss function for attack.  Many\nheuristic adversarial example detections and defense methods have been broken\nby stronger and adaptive attacks [1,2], and the use of adaptive attacks is\ncrucial [3].\n\nAdditionally, although the paper claims to detect zero-day, or unknown attacks,\nin evaluation the selection of attacks are quite limited. For example, it only\nincludes gradient based attacks, but not decision based attacks or evolutional\nadversarial attacks.\n\nThe paper attempts to make several theoretical justifications, but these\ntheorems are too simple (e.g., based on direct application of triangle\ninequality) and do not significantly improve the contribution of this paper.\n\nAs a conclusion, I cannot support the acceptance of this paper because the\nnovelty of the proposed method is limited and evaluation is insufficient.\n\n\n[1] Athalye, Anish, Nicholas Carlini, and David Wagner. \"Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.\" arXiv preprint arXiv:1802.00420 (2018).\n\n[2] Carlini, Nicholas, and David Wagner. \"Adversarial examples are not easily detected: Bypassing ten detection methods.\" Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security. 2017.\n\n[3] Carlini, Nicholas, et al. \"On evaluating adversarial robustness.\" arXiv preprint arXiv:1902.06705 (2019).\n\n[4] Brendel, Wieland, Jonas Rauber, and Matthias Bethge. \"Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.\" arXiv preprint arXiv:1712.04248 (2017).\n\n[5] Cheng, Minhao, et al. \"Query-efficient hard-label black-box attack: An optimization-based approach.\" arXiv preprint arXiv:1807.04457 (2018).\n\n[6] Alzantot, Moustafa, et al. \"Genattack: Practical black-box attacks with gradient-free optimization.\" Proceedings of the Genetic and Evolutionary Computation Conference. 2019.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MIAbWkKNoi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a novel approach to detect adversarial examples by measuring the perturbation norm after a counterattack. They theoretically provide a certified detecting performance and empirically show that AttackDist can characterize zero-day adversarial samples.\nStrength:\n1. The paper is well-organized and easy to understand.\n2. Zero-day attacks are challenging but more practical than other adversarial samples detection problems. AttackDist can perform significantly better than the baselines to distinguish zero-day adv examples.\n3. The authors show experiment results across different attacks and adversarial distances; AttackDist is consistently better than baselines.\n \n\nQuestions:\nThe idea of this paper is interesting and i would like to raise my rating if the authors can clarify my questions.\n\n1. The proof of lemma 1 is the key to developing AttackDist. However, I am confused about 'Due to the continuous of f, …'. My question is how you make sure there always exists the point P between x and x*, such that g(P)=0; what if the deep neural network output is bounded.\n2. Why not evaluate the BlackBox attacking approaches like ZOO and NATTACK[1,2]? Especially, NATTACK learns adversarial examples' distributions, and AttackDist is based on adversarial samples' distribution.\n3. I am curious about whether AttackDist still works for large margin-based classifiers; what if the deep neural network is optimized with large margin loss function.\n\n\n\n[1] Chen, Pin-Yu, et al. \"Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.\" Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security. 2017.\n[2] Li, Yandong, et al. \"Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks.\" arXiv preprint arXiv:1905.00441 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "CvENlRiR5Ve", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a counterattack strategy to attack an input x, and calculate the distance between x and the crafted example x' as the detection metric. There are two main concerns about this paper:\n\n1. The authors claim that their detection method is attack-agnostic, but their motivation in Fig 1 highly depends on the assumption of the attacking mechanism. There is no guarantee on the effectiveness of AttackDist when the attacks do not follow assumed patterns.\n\n2. In experiments, there are only oblivious attacks, where the adversaries do not know the mechanism of AttackDist. For a defense method, it is necessary to carefully design a convinced adaptive attack and demonstrate the effectiveness of the defense under the adaptive attack.\n\nMinors:\nIn the introduction section, the authors claim that \"Existing adversarial defense techniques could be classified into two main categories: adversarial training and detection\". Actually, there are many other types of defenses including input processing, robust architecture, random smoothing, certified defenses, etc. Besides, existing adversarial training methods like FastAT can easily scale to ImageNet, running for several hours on a single GPU. The authors should be more updated on these related progresses.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers liked the concept of the zero-day attack and yet raised different concerns about the other parts of the paper. In general, Reviewers wanted to see more thorough experimental evaluations (e.g., against blackbox attack and adaptive attack) and improved clarity of the theoretical analyses. AC encourages authors to incorporate Reviewers' comments when preparing the paper for elsewhere.
Decision: Reject