Title: IMPROVED DETECTION OF ADVERSARIAL ATTACKS VIA PENETRATION DISTORTION MAXIMIZATION

Abstract: This paper is concerned with the defense of deep models against adversarial attacks. We develop an adversarial detection method, which is inspired by the certificate defense approach, and captures the idea of separating class clusters in the embedding space to increase the margin. The resulting defense is intuitive, effective, scalable, and can be integrated into any given neural classification model. Our method demonstrates state-of-the-art (detection) performance under all threat models.

1. INTRODUCTION:

Defending machine learning models from adversarial attacks has become an increasingly pressing issue as deep neural networks become associated with more critical aspects of society. Adversarial attacks can effectively fool deep models and force them to misclassify, using a slight but maliciouslydesigned distortion that is typically invisible to the human eye (Carlini & Wagner, 2017c; Athalye et al., 2018; Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016) . Despite numerous developments, defense mechanisms are still wanting.

Many interesting ideas have been proposed to construct defense mechanisms for adversarial examples. Among these are adversarial training (Zuo et al., 2020; Yan et al., 2018; Tramèr et al., 2017; Madry et al., 2017) , ensemble methods (Strauss et al., 2017) , and randomization (Dhillon et al., 2018) to name a few. These works consider both detection and resiliency. However, many of these defense ideas were found to be inadequate (Athalye et al., 2018; Carlini et al., 2019; Carlini & Wagner, 2017b; He et al., 2017) . For example, adversarial training critically depends on the specific choice of adversarial attacks used to generate the adversarial training instances. As a result, often this method cannot withstand attacks based on different strategies. (Engstrom et al., 2018) .

A more formal approach to adversarial defense is the certification approach (Hein & Andriushchenko, 2017) , which is designed to provide a lower bound for the penetration distortion required to fool a given network. Certified defense methods are referred to as being either "exact" or "conservative". In exact methods no distortion smaller than the certification bound can penetrate the deep neural network (DNN) (Hein & Andriushchenko, 2017; Wong & Kolter, 2017; Wong et al., 2018; Cohen et al., 2019) . In "conservative" methods, the bound is merely a relative metric for comparing DNN robustness to adversarial examples (Tsuzuku et al., 2018; Zhang et al., 2019; Ding et al., 2018) . Both exact and conservative methods have been criticized for being computationally expensive and unscalable (Tjeng et al., 2018; Cohen et al., 2019) .

It is interesting to view adversarial attacks through activation geometry in embedding layers. A trained deep classification model tends to organize instances into clusters in the embedding space, according to class labels. Classes with clusters in close proximity to one another, provide excellent opportunities for attackers to fool the model. This geometry explains the tendency of untargeted attacks to alter the label of a given image to that of an adjacent class in the embedding space as demonstrated in Figure 1a . Thus, if we can modify the model to increase the margin between clusters, while lowering (or not increasing) the activation sensitivity in the embedding space to input changes, we can make the network more immune to attacks. This embedding sensitivity can be quantified through a Lipschitz constant or directly via the Jacobian.

Inspired by certificate defense methods, we developed an adversarial detection method, that captures the above separation in embedding space intuition. Ideally, we would like to lower bound the distor-tion, , required by the adversary to force a DNN F to misclassify x + , where x is an input image. We propose an approximation to such a bound which, while not formal, motivates a useful strategy for creating defense methods. The bound, η/||J F (x)||, which is similar to other known bounds, is given in terms of η, where η quantifies the "embedding margin" of the network, and J F (x), the Jacobian of F with respect to x (see details in Section 2). The embedding margin, for a given intermediate layer, is the minimal distance (under any p-norm) between two instances belonging to two different classes. This approximate relation motivates a strategy of penetration distortion maximization (PDM) whereby, we implicitly or explicitly maximize this lower bound without attempting to calculate it.

The notion of increasing a DNN classification margin has been discussed in several contexts Liu et al. (2016) ; Hoffer & Ailon (2015) ; Wen et al. (2016) . To apply the PDM approach we propose two procedures to increase the embedding margin. These two methods are complementary in the sense that we can benefit by applying them together. In conjunction, we use the reverse crossentropy method of Pang et al. (2018) , which tends to smooth the Jacobian. Our adversarial detection mechanism is constructed by training a resilient classifier using the above three procedures; we then apply standard kernel density estimation (KDE) on the embedding layer (Feinman et al., 2017) . We present an extensive empirical study focusing on the detection of adversarial examples under all threat models, in which we consider the FGSM, BIM, C&W and JSMA attacks. Our experimental procedure strictly adheres to the comprehensive evaluation desiderata proposed by Carlini et al. (2019) . The results we obtain indicate that the proposed defense achieves state-of-the-art detection.

2. PENETRATION DISTORTION MAXIMIZATION:

In this section, we explain the PDM strategy. Let F be a neural classifier and let x ∈ R h×w be an image assumed to have class label c = c(x). Let ∈ R h×w be a vector representing an adversarial distortion for image x such that the (successful) adversarial instance is x adv = x + whose label is different from c; namely, c adv = F (x adv ) = c. The attacker's goal is to find the smallest perturbation such that F misclassifies x,

min || || s.t. F (x + ) = c(x) .

For a successful adversarial attack whose distortion is required to be small, in the spirit of (Ding et al., 2018; Tsuzuku et al., 2018; Hein & Andriushchenko, 2017; Zhang et al., 2019) , we approximate a prediction for x adv using the first-order Taylor approximation

EQUATION

for vector-valued functions with J F (x) being the Jacobian of F . The same approximation applies to the output of any intermediate layer . Denoting by F (x) the output of layer we thus have,

F (x adv ) ≈ F (x) + J (x) .

For layer , we define its embedding margin,

η = arg min x1,x2,c(x1) =c(x2) ||F (x 1 ) -F (x 2 )||. Thus, ||J (x) || ≈ ||F (x) -F (x adv )|| ≥ η (2)

The Frobenius norm used here is sub-multiplicative (proof can be found in Appendix A); namely,

EQUATION

Combining ( 2) and (3) (and ignoring the approximation error) we lower bound the norm of the distortion in terms of the embedding margin and the norm of the Jacobian, While the attacker's goal is to find a small distortion that "penetrates" another class, our goal as the defender is to create a resilient model that forces a larger distortion. The lower bound (4) motivates our penetration distortion maximization (PDM) method whereby the goal is to explicitly maximize the right side of ( 4) with respect to the embedding layer of the model F . To successfully apply this technique we must increase the embedding margin η l (while not increasing the norm of the Jacobian) and/or smooth the network to decrease the norm of the Jacobian ||J (x)||. We note that similar and stronger, formal bounds, in terms of the Lipschitz constant 

EQUATION

3. INCREASING RESILIENCY USING PDM:

In this section we show how we use PDM, which is applied to the final layer that captures the full embedding of the network (often referred to as the "pre-logits"). We note that technically we can also apply PDM to any other layer in the model but defer such explorations to future work. The proposed approach consists of three components, which are described in this section. Two novel components are used to increase the margin, and the third is a known technique that is responsible for reducing the norm of the Jacobian.

Our approach for increasing the embedding margin relies on the observation that at higher embedding layers of a trained model, the embedding vectors (tensors) of instances tend to be structured in clusters according to class labels. This can be seen, for example, in Figure 1a where we observe the t-SNE visualization (Maaten & Hinton, 2008) of the embedding layer of a network trained for CIFAR-10. Moreover, we observe that an adversarial example created by an untargeted attack often obtains a class label whose cluster is in close proximity with the cluster of the original class.

In Figure 1b we present a color matrix showing the adversarial label distribution obtained by the C&W attack (Carlini & Wagner, 2017a) . For example, the color for the Cat and Dog entry is bright red indicating a frequent label change from cats (original) to dogs (adversarial), whose clusters are the closest. By increasing the margin between these clusters without increasing the norm of this layer's Jacobian ,we make it harder for an adversary to alter the label using distortion of the same magnitude.

Adopting ideas from cluster analysis, the increase in the embedding margin can be achieved by either increasing the distance between clusters or reducing the variance of each cluster. Let µ c = 1 Nc Nc i=1 z c i be the mean of each cluster, where N c is the number of samples from class c and z c i is the embedding of sample i from class c , and let M be the number of classes. We thus have,

Cluster Variance = M c=1 1 N c Nc i=1 ||z c i -µ c || 2 Cluster Distance = 1 M M c=1 1 M -1 M i =c ||µ i -µ c || 2

To increase the margin, we would like to maximize the cluster distance and minimize the cluster variance, hence Margin Maximization Objective = Cluster Variance -Cluster Distance.

A straightforward maximization of the cluster distance is problematic because the distance is potentially unbounded. However, we can proxy the distance using the angular distance between clusters. To this end, we use the cosine similarity. We now introduce two methods to optimize these components. We use a Siamese training procedure to maximize the cluster distance. The cluster variance is minimized by including a variance in the loss function.

3.1. SIAMESE TRAINING:

To explicitly increase the embedding margin, we propose using Siamese training. We create a Siamese network (Bromley et al., 1994) , where each sub-network is our classifier. The Siamese network has two input images denoted by x c i , x c j and three outputs: two classification outputs and an auxiliary output for the cosine similarity between each sub-network's embedding. We introduce an additional loss term to force embeddings from different classes samples to have a cosine similarity of 0 or 1 otherwise. Formally,

SiameseLoss = z c i • z c j ||z c i ||||z c j || ! = 1 if c = c 0 else 3.2 REDUCE VARIANCE LOSS

Inspired by Szegedy et al. (2016) , we include an additional loss term that penalizes large variance for each class' cluster individually. We refer to this component as the "reduce variance loss" (RVL). Formally,

σ c = 1 N c Nc i=1 ||z c i -µ c || 2 , RVL = 1 N c N classes c=1 σ c (5)

The variance is estimated per class on each mini-batch, then averaged and minimized as part of the learning process.

3.3. REVERSE CROSS ENTROPY:

We use the reverse cross-entropy loss introduced by Pang et al. (2018) to minimize the norm of the Jacobian. By labeling a sample with a "reverse" one-hot vector, we obtain

R c i =    0 , if i = c 1 (Nclasses-1) , else,

and using a reverse cross entropy loss

L RCE = -R c log F (x).

Similar to label smoothing (Szegedy et al., 2016) , this method smooths the classifier's gradients and prevents the network from becoming over-confident (Müller et al., 2019) . Intuitively, the differentiation between two samples has a tighter upper bound, given the reverse labels R c , than a one-hot labeling. We tested the gradient L 2 norm value on different layers. Comparing to the baseline model, the gradients were five to ten times smaller when using the RCE training process.

3.4. PDM TRAINING:

A simultaneous application of the three components described above, which can robustify a classification model, is obtained by training the model using an appropriate loss function as well as a specialized mini-batch construction procedure. A pseudo-code of the training procedure including the loss function appears in Algorithm 1 under Appendix F. The code is self-explanatory for the most part. We note that an epoch begins by creating a Siamese counterpart for each image-label pair in a given batch. With probability Q, the Siamese sample is selected from the same class, and its cosine similarity label is set to 1. Otherwise (probability 1 -Q), the Siamese sample is selected from a different class, and its cosine similarity label is set to 0. Notice that the Siamese and RVL (Equation 5) components of the loss function are computed from the embedding vectors of each mini-batch. The RCE component is calculated using the logits.

3.5. PDM VISUALIZATION:

Using t-SNE to visualize the embedding space activation, Figure 2 illustrates the effect of each of the components of our defense method. Figure 2c demonstrates how well the RVL reduces the variance, while the Siamese training process made a more profound impact on the between-class distance as shown in Figure 2d . We adopted the detection method presented by Feinman et al. (2017) , using a univariate Gaussianbased kernel density estimation (KDE), where the density was estimated using 1000 training sample embeddings per class. An input image is deemed adversarial if the distance to the predicted class' manifold exceeds a predefined threshold. As the decision is threshold-dependent, we report our results as the area under the ROC curve (AUC). We use ResNet 56 (He et al., 2016) as our classifier, and compare our results to two baselines: standard training of ResNet-56 (i.e., without any defense mechanism) and ResNet-56 equipped with the RCE defense, the current state-of-the-art model. The hyper-parameters used are listed in Appendix B.

In our study we used several attacks, which are described in Appendix E. For the bounded adversarial attack algorithms, we used two versions of FGSM and BIM, one with a small perturbation = 0.05, and another with a large perturbation = 0.1. For the unbounded attacks, we used JSMA and two versions of C&W: a lean version with zero confidence, and an extensive version with a higher confidence value, denoted by C&W-hc. We used the Cleverhans implementation (Papernot et al., 2018) for the attacks and applied them in an untargeted manner. A detailed description of the parameters used in the adversarial attacks appears in Appendix C. A description of the threat models we consider in this paper appears in Appendix D.

4.1. PERFORMANCE ON NORMAL SAMPLES:

We began by evaluating the performance of our model on normal samples, as shown in Table 1 .

While the RCE method of Pang et al. (2018) lowered the classifier's accuracy on CIFAR-10, using the Siamese training scheme and applying the reduce variance loss increased the accuracy for both CIFAR-10 and MNIST. These results indicate that these margin-increasing procedures may be of independent value in training standard classifiers, regardless of the need for adversarial robustness. We follow a strict definition of a gray-box threat model as in (Pang et al., 2018) , where the attacker has full access to the trained model, but is unaware of the detection mechanism. See Appendix D for precise definitions of the threat-models we consider in this paper. We evaluated the performance under the gray-box threat model by creating equally-sized groups of adversarial and normal examples.

We scored each example using our (KDE-based) detection mechanism. The results are shown in Table 2. The PDM detection AUC results over MNIST are outstanding, showing that all the unbounded attacks (such as C&W) were detected perfectly. On the CIFAR-10 dataset PDM performance lagged behind in defending the two BIM attacks. However, it achieved excellent AUC results for the other attacks, including a perfect score for the strong C&W high confidence attack. Figure 3 presents the resiliency of PDM and the baselines when attacked by FGSM and BIM. Consider, for example, Figure 3 (a) depicting the resiliency achieved against FGSM over the CIFAR-10 dataset. The x-axis corresponds to the distortion step size ( ) used by the adversary, which alters each pixel by ± . The y-axis measures the resiliency, namely how many perturbed instances were predicted correctly by the model. While the resiliency monotonically decreases as a function of the step size, as expected, the resiliency exhibited by PDM (blue) is consistently higher than the baselines. Similar behavior was observed for the BIM attack on this dataset. Over the MNIST dataset, PDM was more resilient than most methods, but not all step sizes.

Further investigation of the mediocre results obtained for the BIM attack revealed that the norm of the embedding layer gradients increased significantly for embedding vectors located in-between clusters. Since BIM makes a sequence of small gradient steps starting inside clusters, it is able to move further away into the center of a different class where it can no longer be detected using KDE. This observation was made by measuring the mean gradient norm after each BIM step. After several such steps we observed that the mean norm increased by an order of magnitude. This phenomenon does not occur when using solely RCE-trained model. 

4.3. WHITE-BOX MODEL:

The white-box threat model (see Appendix D) is perhaps the most interesting from the defender's viewpoint because no limitations are made regarding the information known to the attacker (Carlini et al., 2019) . For the white-box threat model, we applied the C&W modified attack (hereafter referred to as C&W-wb) (Carlini & Wagner, 2017a) , which has been shown to penetrate density estimation-based detection. To the best of our knowledge, this is the only known attack with this property. Following the evaluation procedure used by Pang et al. (2018) , we set the parameters of the C&W-wb attack such that all attacks would succeed in fooling the model, and detection is fully breached (i.e., their AUC score ≤ 0.5). Then, we measured the average (minimal) required distortion that was able meet these criteria. Thus, a stronger defense should yield larger distortion. We note that the distortion is quantified using the L 2 norm, namely,

d = 1 M M i=1 ||xi-x adv i ||2 √ n

, where M is the number of adversarial instances, and n is the number of pixels per image.

Baseline RCE PDM (ours) MNIST 0.087 0.104 0.162 CIFAR10 0.008 0.019 0.026

Table 3 : Distortion under the white-box threat model, scaled to [0,1], Our defense method requires 30% higher distortion on CIFAR-10 and 60% higher on MNIST

The white-box results are presented in Table 3 . PDM, clearly outperforms the baselines by a wide margin by forcing a 30% higher distortion than RCE on CIFAR-10, and 60% on MNIST. To evaluate our model in the black-box setting, we follow (Papernot et al., 2017; Carlini et al., 2019) and create a proxy model, which is trained using input-output pairs probed from the defender's (target) model (i.e., the proxy model is trained via teacher-student distillation of the target model). 

5. CONCLUDING REMARKS:

We introduced a powerful approach for the defense of deep models against adversarial attacks by building on procedures for margin maximization within a penetration distortion maximization framework and the RCE loss technique. Our empirical evaluation demonstrated state-of-the-art results in defense against all threat models (with mixed results for the BIM attack). In addition, we provide some geometric intuition on attacks and defenses using t-SNE visualizations.

D THREAT MODELS:

Rigorous analyses of adversarial defense systems requires precise specification of what the adversary knows and can do. Conducting this type of specification is called threat modeling (Kurakin et al., 2016) . In this paper we consider the three core models, which are differentiated by their knowledge of the classifier (target network) and the defense mechanism being used.

• Black-box: In this weakest scenario, the adversary has no knowledge of our system. It does not know the target network architecture, cannot access its gradients and does not know the defense methods. The adversary can, however, sample the targeted network for input-output pairs and has access to the dataset used to train the target network. • Gray-box (a.k.a. oblivious): Here the adversary has full knowledge of the target network and can access its gradients and parameters, including the data being used in the training process. However it has no information on the defense mechanism. • White-box: In this most challenging scenario, we assume the adversary has full knowledge of the entire system to be attacked, which includes the target network, the defense method, their parameters and the data used for training.

To ensure a clear distinction between threat models, Carlini et al. (2019) recently proposed a comprehensive methodology for evaluating adversarial attacks and defenses, thus emphasizing the difference between gray-box and white-box attacks. We believe that following the Carlini et al. guideline is crucial for advancing this line of research and therefore we adhere to them strictly.

E ATTACK ALGORITHMS:

Adversarial attack algorithms aim to fool DNNs such that a given image is misclassified with minimal perturbations. In untargeted attacks, the adversary aims to minimize the true class activation so that another arbitrary class will be predicted. In this paper, we focus on targeted attacks where the adversary aims to fool the classification in a controlled manner such that a specific class will be predicted instead of the correct one. Thus, targeted attacks are considered far more dangerous. For example, fooling an autonomous driving system to interpret a stop sign as a speed limit sign is far more dangerous than interpreting it as a yield sign.

In this section, we discuss the various attack algorithms we use to evaluate our defense method. We denote x,x as the input and adversarial images, respectively, as the target label, F as the target model with loss function L F (x, ) and = ||x -x || as the pixel-wise perturbation between the adversarial and normal images. The general formulation, therefore, becomes, minimize Goodfellow et al. (2014) introduced the fast gradient sign method (FGSM), which optimizes the adversarial image by back-propagating the input through the attacked DNN, in accordance with the desired target. Formally, letting be a fixed parameter, the adversarial example is,

EQUATION

FGSM:

EQUATION

While not as effective as other attack algorithms, this method has the advantage of being one of the fastest ones. Kurakin et al. (2016) introduced the Basic Iterative Method (BIM), which performs the FGSM method iteratively, clipping the perturbation if needed. Formally,

BIM:

x N +1 = x N + sign(∇L F (x N , l)),

where is a fixed parameter.

C&W:

Carlini & Wagner (2017c) introduced the C&W method, which operates by modifying the L-BFGS method as follows, minimize

x ||x -x || 2 + cf (x ),

where c is a hyperparameter and the loss function, f , is chosen such that f (x ) <= 0 if x is classified as the target class; namely,

f (x ) = max(Z(x ) -Z(x ) t , κ)

where Z is the target DNN logits, t is the correct label and κ is a hyperparameter referring to the confidence of the attack. The higher the confidence, the higher the activation for the target class is, and therefore, the larger the perturbation.

Three different Euclidean norms are considered with this algorithm, L 0 , L 2 , L ∞ . Following Pang et al. (2018) , we conduct our evaluation using L 2 . This attack method is considered very effective and has had great success in overcoming various defense methods Carlini & Wagner (2017c) , Papernot et al. (2016b) . Papernot et al. (2016a) introduced the Jacobian-based Saliency Map Attack (JSMA) which alters a single pixel of x at each iteration to maximize the saliency map. This method is known to enforce large perturbations but on fewer pixels than other methods. q ∼ Bernoulli(Q) 7:

JSMA:



F PDM TRAINING ALGORITHM:

if q == 1 then 

:



Figures and Tables:

Figure 1: Histogram of origin and target classes from C&W untargeted adversarial attack compared to embedding layer t-SNE

, have been introduced by Tsuzuku et al. (2018), Zhang et al. (2019), Hein & Andriushchenko (2017) and Ding et al. (2018).

While t-SNE is useful for visualization purposes, the aggressive dimensionality reduction may lead to misleading conclusions. To obtain quantitative evidence, we calculated the Davies-Bouldin index (DBI)(Davies & Bouldin, 1979), which scores clustering quality according to the distance between cluster centroids divided by the Euclidean distance between points within a cluster (lower score means better clustering). The DBI of the combined method is indeed the lowest at 0.23. (See the other DBIs in the figure.)4 EXPERIMENTSFollowing(Pang et al., 2018;Meng & Chen, 2017;Madry et al., 2017;Song et al., 2017;Dhillon et al., 2018;Samangouei et al., 2018) we evaluated our defense technique on theMNIST (LeCun  et al., 1998)  andCIFAR-10 (Krizhevsky & Hinton, 2009)  datasets.

Figure2: CIFAR-10 t-SNE visualization of the two margin increasing components of PDM . Compared to the baseline, each method contributes to the increase of the margin; the combined method display the best clustering according to the Davies-Bouldin Index.

Figure 3: Resiliency under the FGSM and BIM adversarial attacks. Our method displays significantly higher resiliency

X siamese = [], Y siamese = [], S = [] 5: for b = 1, . . . , batch size do 6:

← random sample from class Y [b] 14: Append x, y, s to X siamese , Y siamese , S 15:z 1 , z 2 ← F (X), F X siamese )sample's embedding 16:p 1 , p 2 ← F (X), F (X siamese ) Y log (p 1 )-R Ysiamese log (p 2 ) + RVL + SL

Detection AUC under the gray-box threat model.

Table content: <table><tr><td/><td/><td>MNIST</td><td/><td/><td>CIFAR-10</td><td/></tr><tr><td/><td colspan="5">Baseline RCE PDM Baseline RCE</td><td>PDM</td></tr><tr><td>FGSM-0.05</td><td>0.981</td><td colspan="2">0.983 0.988</td><td>0.958</td><td colspan="2">0.898 0.967</td></tr><tr><td>FGSM-0.1</td><td>0.988</td><td colspan="2">0.99 0.995</td><td>0.971</td><td colspan="2">0.926 0.983</td></tr><tr><td>BIM-0.05</td><td>0.983</td><td colspan="2">0.967 0.987</td><td>1</td><td>0.99</td><td>0.95</td></tr><tr><td>BIM-0.1</td><td>0.945</td><td>0.92</td><td>0.99</td><td>1</td><td colspan="2">0.996 0.962</td></tr><tr><td>C&amp;W</td><td>0.994</td><td>1</td><td>1</td><td>0.874</td><td colspan="2">0.918 0.933</td></tr><tr><td>C&amp;W-hc</td><td>0.88</td><td>0.98</td><td>1</td><td>0.637</td><td>0.94</td><td>1</td></tr><tr><td>JSMA</td><td>0.995</td><td>1</td><td>1</td><td>0.952</td><td colspan="2">0.96 0.973</td></tr></table>

The proxy model is then used by the attacker to generate adversarial examples under the white-box threat model (this black-box model variant is the most difficult, and was referred to by Pang et al. as "A white, black-box attack"). In our case, where detection is based on KDE, we could only use the C&W-wb because it is the only available attack known to penetrate KDE. Following Pang et al. in this setting we used ResNet32 He et al. (2016) for the proxy model. Here again, we measured detection AUC rates as in the gray-box setting. The results appear in Table4and show that PDM is consistently and significantly better than the baselines. Detection AUC under the black-box threat model

Table content: <table><tr><td/><td colspan="3">Baseline RCE PDM (ours)</td></tr><tr><td>MNIST</td><td>0.88</td><td>0.94</td><td>0.99</td></tr><tr><td>Cifar10</td><td>0.93</td><td>0.933</td><td>0.952</td></tr></table>

Peer review (iclr2020):
Summary ======== This paper proposes a defense against adversarial examples that detects perturbed inputs using kernel density estimation. The paper uses a combination of known (and often known to be broken) techniques, and does not provide a fully convincing evaluation. I lean towards rejection of this paper. Detailed comments ================= The idea of increasing robustness by maximizing inter-class margins and minimizing intra-class variance is fairly natural, but the author*s discussion of their approach (mainly in sections 1 and 2) is very hand-wavy and relies on a lot of general intuitions and unproven claims about neural networks. For example, in the introduction, the authors claim: *A trained deep classification model tends to organize instances into clusters in the embedding space, according to class labels. Classes with clusters in close proximity to one another, provide excellent opportunities for attackers to fool the model. This geometry explains the tendency of untargeted attacks to alter the label of a given image to a class adjacent in the embedding space as demonstrated in Figure 1a.* First, a t-SNE representation is just a 2D projection of high-dimensional data that is useful for visualization purposes, and one should be careful when extrapolating insights about the actual data from it. For example, distances in the 2D projection do not necessarily correspond directly to distances in the embedding space. The claim that untargeted attacks lead to a *nearby* cluster are hard to verify given just Figure 1. First, the colors of the labels between 1a and 1b do not seem to match (e.g., Dog is bright green in 1b but this color does not appear in 1a). If the other colors match, then this would seem to suggest that trucks (purple) often get altered to ships (orange). Yet, the two clusters are quite far apart in 1a. It seems hard to say something qualitative here. An actual experiment comparing distances in the embedding space and the tendency of untargeted attacks to move from one class to another would be helpful. The color scheme in Figure 1b is also unclear. A color bar would help here at the very least. These observations are then used to justify increasing cluster distance while minimizing cluster variance, but it would be nice to see a more formal argument relating these concepts to the embedding distance. The technique proposed in Section 3.2. to reduce variance loss estimates each class* variance on each batch. Would this still work for a dataset with a large number of classes (e.g., ImageNet)? For such a dataset, each class will be present less than once in expectation in each batch, which seems problematic. The plots in Figure 2 don*t give much of a sense of how the combination of the different proposed techniques is better than any individual technique. The evaluation compares PDM to RCE, but from Figure 2 one could guess that variance reduction alone (2c) performs very similarly to PDM (2e). An ablation study showing the contribution of each of the individual techniques would be helpful. The evaluation section could be improved significantly. FGSM, JSMA, and to some extent BIM, are not recommended attacks for evaluating robustness. The gray-box and black-box threat model evaluations are also not the most interesting here. Instead, and following the recommendations of Carlini et al. (2019), the evaluation should: - Propose an adaptive attack objective, tailored for the proposed defense in a white-box setting. The authors do this to some extent, by re-using the attack objective from Carlini & Wagner 2017, which targets KDE. It would still be good to provide additional explanations about how the hyperparameters for this attack were set. - Optimize this objective using both gradient-based and gradient-free attacks - As the proposed defense is attack-agnostic, I also suggest trying it out on rotation-translation attacks, as the worst-case attack can always be found by brute-force search Other ===== - The citations for adversarial training in the 2nd paragraph of the intro are unusual. Standard references here are for sure the first two below, and maybe some of the other three as is relevant to your work - Szegedy et al. 2013: *intriguing properties of neural networks* - Goodfellow et al. 2014: *Explaining and harnessing adversarial examples* - Kurakin et al. 2016: *Adversarial Machine Learning at Scale* - Madry et al. 2017: *Towards deep learning models resistant to adversarial attacks* - Tramer et al. 2017: *Ensemble Adversarial Training* - The Taylor approximation in (1) does not seem to be well defined. The Jacobian of F is a matrix, so it isn*t clear what evaluating that matrix at a point x means. - The *greater yet similar* symbol (e.g., in equation (4)) should be defined formally.