Title: PRIOR NETWORKS FOR DETECTION OF ADVERSARIAL ATTACKS Anonymous

Abstract: Adversarial examples are considered a serious issue for safety critical applications of AI, such as finance, autonomous vehicle control and medicinal applications. Though significant work has resulted in increased robustness of systems to these attacks, systems are still vulnerable to well-crafted attacks. To address this problem several adversarial attack detection methods have been proposed. However, system can still be vulnerable to adversarial samples that are designed to specifically evade these detection methods. One recent detection scheme that has shown good performance is based on uncertainty estimates derived from Monte-Carlo dropout ensembles. Prior Networks, a new method of estimating predictive uncertainty, have been shown to outperform Monte-Carlo dropout on a range of tasks. One of the advantages of this approach is that the behaviour of a Prior Network can be explicitly tuned to, for example, predict high uncertainty in regions where there are no training data samples. In this work Prior Networks are applied to adversarial attack detection using measures of uncertainty in a similar fashion to Monte-Carlo Dropout. Detection based on measures of uncertainty derived from DNNs and Monte-Carlo dropout ensembles are used as a baseline. Prior Networks are shown to significantly out-perform these baseline approaches over a range of adversarial attacks in both detection of whitebox and blackbox configurations. Even when the adversarial attacks are constructed with full knowledge of the detection mechanism, it is shown to be highly challenging to successfully generate an adversarial sample.

1. INTRODUCTION:

Neural Networks (NNs) have become the dominant approach to addressing computer vision (CV) (Girshick, 2015; Simonyan & Zisserman, 2015; Villegas et al., 2017) , natural language processing (NLP) (Mikolov et al., 2013b; a; 2010) , speech recognition (ASR) (Hinton et al., 2012; Hannun et al., 2014) and bio-informatics (Caruana et al., 2015; Alipanahi et al., 2015) tasks. However, as observed by (Szegedy et al., 2013) , they are susceptible to adversarial attacks -small perturbations to the input which are almost imperceptible to humans, yet which drastically affect the predictions of the neural network. It was found that adversarial attacks have several properties which make them a serious security concern. Firstly, adversarial attacks are transferable -an adversarial attack computed on one network may be able to successfully attack a different network (Szegedy et al., 2013; Goodfellow et al., 2015) . Secondly, there exists a plethora of adversarial attacks which are quite easy to construct (Goodfellow et al., 2015; Kurakin et al., 2016; Dong et al., 2018; Carlini & Wagner, 2016; Papernot et al., 2016a; b; Liu et al., 2016) . Thirdly, it is possible to craft adversarial attacks within the physical world, such as putting stickers on paper (Kurakin et al., 2016) . To compound the issue, it was found that adversarial attacks are hard to defend against (Carlini & Wagner, 2017) . Specifically, it was found that while adversarial training (Szegedy et al., 2013) and adversarial distillation (Papernot et al., 2016c) are able to improve the robustness of a network, it is still possible to craft successful adversarial attacks against these networks. Indeed, currently, there are far more methods of successfully attacking networks than there are of defending networks (Carlini & Wagner, 2017; 2016) . Altogether, this raises serious concerns about how safe it is to deploy neural networks for high-stakes applications, such as autonomous vehicle control, financial and medical applications.

While much work focuses on constructing neural networks which are robust to adversarial attacks, (Carlini & Wagner, 2017) investigate detection of adversarial attack and shows that adversarial attacks can be detectable using a range of approaches. However, it turns out that attacks can then be crafted to fool the proposed detection schemes. However, (Carlini & Wagner, 2017) singles out detection of adversarial attacks using uncertainty measures derived from Monte-Carlo dropout as being successful. Detection of adversarial attack using Monte-Carlo dropout was further investigated in (Smith & Gal, 2018) . Work by (Smith & Gal, 2018) suggests that adversarially modified inputs can be interpreted as inputs which lie off the manifold of natural images. In other words, adversarial samples can be seen as a special type of out-of-domain or out-of-distribution input which can be detected by measures of uncertainty derived via Monte-Carlo Dropout.

Recently, (Malinin & Gales, 2018) proposed Prior Networks -a new approach to modelling uncertainty which has been shown to outperform Monte-Carlo dropout on a range of tasks. Prior Network parameterize a distribution over output distributions which allows them to separately model data uncertainty and distributional uncertainty. Furthermore, they are explicitly taught to learn decision boundaries between an in-domain and an out-of-domain region, unlike Monte-Carlo dropout, where it is not possible to specify the behaviour of measures of uncertainty in different regions of the input space explicitly. Finally, Prior Networks allow measures of uncertainty to be obtained analytically, without the need for expensive Monte-Carlo sampling. Thus, this work investigates the detection of adversarial attacks using measures of uncertainty derived from Prior Networks. The contributions of this work are as follows:

• Measures for assessing the success of adversarial attacks in the context of detection are proposed.

• It is shown that whitebox and blackbox adversarial attacks with no knowledge of the detection scheme can be successfully detected based on measures of uncertainty.

• It is shown that whitebox detection evading adversarial attacks are difficult to construct for Prior Networks, and detection evading attack fail entirely against Prior Networks.

2. UNCERTAINTY FOR DEEP LEARNING:

In (Smith & Gal, 2018) adversarially modified inputs are interpreted as inputs which lie off the manifold of natural images -the stronger the adversarial perturbation, the further is the input from the manifold. Thus, adversarial samples can be seen as out-of-distribution inputs. 

P(y|x * , D) = P(y|x * , θ)p(θ|D)dθ (1)

Unfortunately, both the integral in eq. 1 and calculation of the model posterior are intractable for neural networks. Typically the model posterior distribution is approximated using either an implicit or explicit variational approximation q(θ) and the integral is approximated via sampling (eq. 2), using approaches such as Monte-Carlo dropout (Gal & Ghahramani, 2016) :

EQUATION

Bayesian approaches model distributional uncertainty through model uncertainty. By selecting an appropriate approximate inference scheme and model prior p(θ) Bayesian approaches aim to craft a model posterior p(θ|D) such that the ensemble {P(ω c |x * , θ (i) )} M i=1 is consistent in-domain and becomes increasingly diverse the further away x * is from the region of training data. The entropy of the expected distribution P(ω c |x * , D) will indicate the total uncertainty in predictions. Measures of the diversity of the ensemble, such as Mutual Information, assess uncertainty in predictions due to model uncertainty, which yields distributional uncertainty.

MI[y, θ|x * , D] M odel U ncertainty = H[E p(θ|D) [P(y|x * , θ)]] T otal U ncertainty -E p(θ|D) [H[P(y|x * , θ)]] Expected Data U ncertainty (3)

In practice, however, for deep, distributed models with tens of million parameters, such as DNNs, it is difficult to select an appropriate approximate inference scheme to craft a model posterior which induces a distribution over distributions with the desired properties.

2.2. UNCERTAINTY ESTIMATION VIA PRIOR NETWORKS:

Unlike Bayesian approaches, which indirectly specify a conditional distribution over output distributions, a Prior Network p(π|x * ; θ), proposed by (Malinin & Gales, 2018) directly parametrizes a prior distribution over categorical output distributions. In this work the Dirichlet distribution (eqn 5) is chosen due to its tractable analytic properties.

EQUATION

A Dirichlet distribution is parameterized by its concentration parameters α, where α 0 , the sum of all α c , is called the precision of the Dirichlet distribution. Higher values of α 0 lead to sharper, more confident distributions:

Dir(π|α) = Γ(α 0 ) K c Γ(α c ) K c=1 π αc-1 c , α c > 0, α 0 = K c α c (5)

The predictive distribution is given by the expected categorical distribution under the Dirichlet prior: A Prior Network is trained to display these behaviors by minimizing the following multi-task loss functions, as described in (Malinin & Gales, 2018) . A flat Dirichlet is chosen as the out-of-distribution target distribution p out (π), as it is the maximum entropy Dirichlet distribution. The target in-domain distribution p in (π|x) is constructed by specifying the mean and precision of the Dirichlet, as described in (Malinin & Gales, 2018) . In order to train with this loss function, it is necessary to have out-of-distribution training data. One way of doing this is to chose, in addition to the target training data, an 'out-of-distribution' dataset. For example, if a model is trained on CIFAR-10 ( Krizhevsky, 2009) , it is possible to use CIFAR-100 as the out-of-distribution dataset, as they don't have overlapping classes.

EQUATION

EQUATION

Given a trained Prior Network it is possible to calculate the Mutual Information using an expression similar to equation 3:

MI[y, π|x * ; θ] Distributional U ncertainty = H[E p(π|x * ; θ) [P(y|π)]] T otal U ncertainty -E p(π|x * ; θ) [H[P(y|π)]] Expected Data U ncertainty (8)

This expression, as in the Bayesian case, allows total uncertainty, given by the entropy of the expected distribution, to be decomposed into data uncertainty and distributional uncertainty

3. MEASURES OF PERFORMANCE:

In order to investigate detection of adversarial attacks, it is necessary to discuss how to assess the effectiveness of an adversarial attack in a scenario where detection of the attack is possible. Previous work on detection of adversarial examples (Gong et al., 2017; Grosse et al., 2017; Metzen et al., 2017; Carlini & Wagner, 2017; Smith & Gal, 2018) assesses the performance of detection methods separately from whether an adversarial attack was successful, and use the standard measures of adversarial success and detection performance. However, in a real deployment scenario, an attack can only be considered successful if it both affects the predictions and evades detection. In this section we develop a measure of performance to assess this.

An adversarial input x adv will be defined as the output of an adversarial attack generation process A adv applied to a natural input x:

EQUATION

This process is typically an optimization problem which tries to minimize a loss L(•), which is a function of the input x, the model P(y|x; θ) and a target class t, with respect to the input while also minimizing some distance δ(•, •) between the original image and the generated adversarial perturbation. This loss is typically the negative log-likelihood of a particular target class, in case of targeted adversarial attacks, or the (positive) log-likelihood of the predicted class, in case of an untargeted attack:

L tgt P(y|x; θ), t = -ln P(ω t |x; θ)

EQUATION

The distance δ(•, •), typically the L 1 , L 2 or L ∞ norm, is minimized so that the adversarial attack is still perceived to be a natural input to a human observer. The best adversarial attack is one which minimizes the chosen loss and has the minimal deviation from the original image x. There are multiple ways in which this optimization problem can be solved (Szegedy et al., 2013; Goodfellow et al., 2015; Kurakin et al., 2016; Dong et al., 2018) .

While the process A adv will always generate some kind of perturbation, it will not always yield a successful attack, as it may fail to affect the prediction. For the purposes of this discussion the adversarial generation process A adv will be defined to either yield a successful adversarial attack x adv or an empty set ∅.

A adv (x, t) ∈ {x adv , ∅} (11) It is necessary to point out that here we take the perspective of the attacker -a successful attack, in the case of targeted attacks, is one which yields the target class. However, consider the perspective of the system designer who does not know whether the attacker attempts a targeted attack or not. From the point of view of the system designer, any attack which is able to affect the predictions of the network may be considered successful, regardless of whether the attack produced the attacker's desired prediction. In the case of non-targeted attacks, however, the attacker's and defender's definition of success are the same.

In a standard scenario, where there is no detection, the efficacy of an adversarial attack on a model1 can be summarized via the success rate S of the attack:

EQUATION

)

Typically S is plotted against the total maximum perturbation from the original image, measured as either the L 1 , L 2 or L ∞ distance from the original image.

Now consider using a threshold-based detection scheme where a sample is labelled 'positive' if some measure of uncertainty, such as entropy H(x), is less than a threshold T and 'negative' if it is higher than a threshold:

EQUATION

The performance of such a scheme can be evaluated at every threshold value using the true positive rate t p (T ) and the false positive rate f p (T ):

EQUATION

The whole range of such trade offs can be visualized using a Receiver-Operating-Characteristic (ROC) and the quality of the trade-off can be summarized using area under the ROC curve.

However, a standard ROC curve does not give credit to the system for being robust to adversarial attacks -it doesn't account for situations where the process A adv (•) fails to produce a successful attack. In fact, if an adversarial attack is made against a system which has a detection scheme, it can only be considered successful if it both affects the predictions and evades detection. This condition can be summarized in the following indicator function:

ÎT (x) =    1, T > H(x) 0, T ≤ H(x) 0, x = ∅ (15)

Given this indicator function, a new false positive rate fP (T ) can be defined as:

EQUATION

This false positive rate can now be seen as a new Joint Success Rate which measures how many attacks were both successfully generated and evaded detection, given the threshold of the detection scheme. The Joint Success Rate can be plotted against the standard true positive rate on an ROC curve to visualize the possible trade-offs. This ROC curve will now give credit to the system when the attacker fails to generate a succesful attack. One possible operating point is where the false positive rate is equal to the false negative rate, also known as the Equal Error-Rate point:

EQUATION

Throughout the remainder of this work the EER false positive rate will be quoted as the Joint Success Rate.

4. DETECTING ADVERSARIAL ATTACKS:

The previous sections discussed how to obtain estimates of uncertainty using several different approaches and how to assess the performance of adversarial attacks in the scenario where a detection scheme is in place. In this section detection of adversarial samples using measures of uncertainty is investigated. Four threat models are evaluated:

• Whitebox attack with no knowledge of the detection scheme;

• Blackbox attack with no knowledge of the detection scheme;

• Whitebox attack with complete information about the detection scheme;

• Blackbox attack with complete information about the detection scheme.

Experiments were conducted on the CIFAR-10 ( Krizhevsky, 2009) dataset. Four models were constructed: standard classification network DNN, Monte-Carlo dropout (MCDP) ensemble derived from the same DNN, a 'standard' prior network 'PN' and an 'adversarially trained' prior network 'PN-ADV'. DNN and MCDP were used as baselines in this work. The baseline prior network PN was trained on CIFAR-10 as in-domain data and CIFAR-100 as out-of-distribution data in exactly the same setup as (Malinin & Gales, 2018) . The adversarially trained Prior Network PN-ADV was additionally trained on un-targeted Fast Gradient Sign Method (Goodfellow et al., 2015) attacks which were dynamically generated during training. The perturbation values were randomly sampled from a normal distribution for each mini-batch. Standard VGG-16 (Simonyan & Zisserman, 2015) architecture was used for all networks. Entropy of the predictive distribution was used as the measure of uncertainty of a DNN and Mutual Information was used as the measure of uncertainty for MCDP (eqn 3) and Prior Networks (eqn 8). For each model type, 5 model-instantiations with different random seeds were trained.

Three types of non-targeted adversarial attacks are considered: Fast-Gradient Sign Method (FGSM) attacks (Szegedy et al., 2013) , Basic-Iterative Method (BIM) Attacks (Kurakin et al., 2016) and Momentum-Iterative Method (MIM) (Dong et al., 2018) attacks. These attacks are chosen because they were the ones evaluated in (Smith & Gal, 2018) and MIM attacks are considered to be quite strong. The L ∞ norm version of the FGSM, BIM and MIM attacks were investigated. Each of the iterative attacks was run for 10 iterations of gradient descent with their 'default' settings , as described in (Kurakin et al., 2016; Dong et al., 2018) . Further details on the training and construction of all models can be found in appendix A.

4.1. ATTACKS WITH NO KNOWLEDGE ABOUT THE DETECTION SCHEME:

Figure 2 displays plots of the Joint Success Rate (eqn 16) at the EER threshold against the perturbation for all attacks averaged across different seed values with ±σ bounds.

Figures 2a-c show that a deterministic DNN and MC Dropout achieve nearly identical performance across all attacks. Curiously, they are both able to detect FGSM attacks, but are almost completely ineffective against BIM and MIM attacks, which achieve joint success rates in excess of 90%2 for small values of epsilon. The standard prior network PN is robust FGSM attacks and is more robust against BIM and MIM attacks than the baselines -the highest success rate of BIM and MIM attack is now 80% at a higher epsilon value of 30. The adversarially trained prior network PN-ADV achieves the best performance by a large margin across all attacks. It is totally robust to FGSM attacks and significantly more robust to BIM and MIM attacks, which now achieve a peak joint success rate of 15% in a narrow range of perturbations.

Performance against black-box attacks is described in figures 2d-f. Whitebox attacks against each random seed are used as blackbox attacks against all the other seeds for a model type. All models, even the DNN and MCDN, are able to achieve very low joint success rates and are robust to all attacks considered. Curiously, a standard Prior Network PN has the worst performance. However, an adversarially trained prior network achieves by far the best performance against all considered black-box adversarial attacks.

These results indicate that a standard prior network learns a better decision boundary between the in-domain and out-of-domain regions than Monte-Carlo dropout and is able to yield better measures of uncertainty. However, as it was trained using only 'on-manifold' out-of-distribution data, it still is susceptible to adversarial attacks which are far from the manifold. Training a prior network on FGSM data allows it to learn a better decision boundary, allowing the prior network to generalize the out-of-domain region beyond the manifold of natural data. Interestingly, the results suggest that the 'holes' in a model's understanding of where the in-domain/out-of-domain region discovered using whitebox attacks fail to generalize as blackbox attacks.

4.2. ATTACKS WITH COMPLETE KNOWLEDGE ABOUT DETECTION SCHEME:

The results of section 4.1 show that prior networks are able to detect both whitebox and blackbox adversarial attacks successfully. However, as discussed in (Carlini & Wagner, 2017) , it is necessary to evaluate whether is is possible to construct adversarial attacks specifically to evade detection. This section investigate whether is is possible craft adversarial attacks which avoid detection by measures of uncertainty derived from DNNs, Monte-Carlo Dropout and Prior Networks.

Conceptually, the best approach to avoiding detection using measures of uncertainty is to generate an adversarial attack which changes a model's prediction while leaving the measures of uncertainty (entropy, mutual information) unchanged. In the case of a DNN or Monte-Carlo dropout, the simplest approach to do this is to simply permute the predicted distribution over classes so that the probability of the max class is assigned to the target class, and the probability of the target class is assigned to the max class. The loss function minimized by the adversarial generation process will be the KL divergence between the predicted distribution over class labels P(y|x; θ) and the target permuted distribution P t (y): A adv applied to a natural input x:

EQUATION

For prior networks the equivalent approach would be to permute the values of α and to minimize KL divergence to the permuted target Dirichlet distribution:

EQUATION

In the following experiments, MIM adversarial attacks are generated using the loss functions given by equations 18 and 19 for DNN/MCDP and Prior Network models, respectively. The iterative approaches are run for a range of iterations, ranging from 10 to 100. The second most likely class is selected as the target class, as this should be a more aggressive attack than switching to the least likely class. Figure 3 shows the Joint Success Rate, ROC AUC and success rate S (eqn. 12) vs the number of iterations for MIM attack generation at a perturbation of 40. The change of x-axis from perturbation to iteration is used to illustrate the added computational difficulty in generating detection evading attacks. The figures show that a detection evading whitebox attack against a DNN is completely successful.

Figure 3c shows that in almost 100% of cases the attack yields the target class. Monte-Carlo dropout performs better in terms of success rate and AUC. However, whitebox detection evading attacks against PN and PN-adv were far less successful, yielding the target class 45% and 40% of the time after far more iterations, respectively. Furthermore, AUC only fell by about 10% absolute for prior network models, and and only after many iterations. Interestingly, blackbox detection evading attacks failed altogether and were detectable using all considered models. This suggests that it is very difficult to generalize the weaknesses a detection evading attack discovers in a model's detection scheme.

The experiments in this section show that it is non-trivial to successfully construct whitebox adversarial attacks which are able to evade detection by measures of uncertainty for appropriately secured prior networks. This suggests that the optimization problem in equation 9 becomes more difficult to solve for Prior Networks, which, in turn, suggests that the manifold hypothesis of adversarial attacks is correct. Specification of the behaviour of the network in-domain and out-of-domain on-manifold as well as out-of-domain off-manifold greatly constrains the space of solutions to equation 9 where the attack both yields the target class and avoids changing properties of predicted distribution over distributions. Using measures of uncertainty in the prediction, uncertainty in the model's understanding of the data, allows the space of solutions to the adversarial optimization problem to be constrained in a way which methods proposed in (Metzen et al., 2017; Gong et al., 2017; Grosse et al., 2017) do not.

5. CONCLUSION:

This work shows that it is possible to detect FGSM, BIM and MIM adversarial attacks using measures of uncertainty derived from DNNs, MC Dropout and Prior Networks. Prior Network models are the most robust to FGSM, BIM and MIM L ∞ adversarial attacks and outperform all the other methods by a large margin in both detection of whitebox and blackbox adversarial attacks. In section 4.2 it is shown that it is possible to construct targeted adversarial attacks which also avoid detection by directly attacking the measure of uncertainty derived from DNNs and Monte-Carlo dropout ensembles. However, results in section 4.2 show that it difficult to construct detection avoiding attacks for adversarially trained prior networks. The results in this work are encouraging and show that is appropriate measures of uncertainty are used, then adversarial attacks are not a great security concern. However, further work in evaluating the robustness of uncertainty estimates derived from Prior Networks and Monte-Carlo dropout against stronger detection-evading adversarial attacks must be done. 

:

Given an evaluation dataset Dtest = {xi, yi} N i=1

Note -a joint success rate larger than 50% means that the AUC is less than 0.5 and can be improved by using a reversed detection scheme

P(ωc|x; θ) is a shorthand for P(y = ωc|x; θ)

Figures and Tables:

6) The desired behaviors of the Prior Network can be visualized on a simplex (fig 1), where figure 1:a describes confident behavior (low-entropy prior focused on low-entropy output distributions), figure 1:b describes uncertainty due severe class overlap and figure 1:c describes the behaviour for an out-of-distribution input.

Figure 1: Desired Behaviors of a Dirichlet distribution over categorical distributions.

Figure 2: Plots of joint success rate at EER threshold for whitebox (a-c) and blackbox (d-f) attacks. Note difference in y-axis scale of plots d-f.

Figure 3: Summaries of performance against whitebox and blackbox detection evading MIM attack. Note different y-axis scale of plot d.

Figure 4: Plots of Joint Success Rates, ROC AUC and Success rates against perturbation ( ) for whitebox FGSM, BIM and MIM attacks.

Figure 7: Plots of uncertainty distributions for real (in-domain) and adversarial (out-of-domain) data for MIM attack at iteration 100

Peer review (iclr2019):
 Summary: The authors propose a new method to detect adversarial attacks (examples). This approach relies on prior networks to estimate uncertainty in model predictions. Prior-networks are then trained to identify out-of-distribution inputs, and thereby used to detect adversarial examples. The authors evaluate their methodology on CIFAR-10 in different white-box and blackbox settings. This work addresses an important question - detecting adversarial examples. Since it may not always be possible to build models that are completely robust in their predictions, detecting adversarial examples and/or identifying points where the model is uncertain is important. However, I am not convinced by the specific methodology as well as the proposed evaluation. Detailed comments: - This work is largely based on the recent work by Malinin and Gales, 2018, where prior networks are developed as a scheme to identify out-of-distribution inputs. As a result, the authors rely fundamentally on the assumption that adversarial examples lie off-the data manifold. There has been no convincing evidence for this hypothesis in the literature thus far. Adversarial examples are also likely to be on the data manifold, but form a small enough set that it doesn’t affect standard generalization. But because of the high-dimensional input space, a member of this small set is still close to every “natural” data point. - I do not find the specific choice of attacks the authors consider convincing. (These being the attacks studied in Smith & Gal, 2018 does not seem to be a sufficient explanation). Specifically, the authors should evaluate on stronger Linf attacks such as PGD [Madry et al., 2017]. Further, it seems that the authors consider Linf eps around 80. These values seem extremely large given that eps=32 (possibly even > 16) causes perceptible changes in the images. Did the authors look at the adversarial examples created for these large eps values? - The authors should include evaluation on a robust DNN (for example PGD trained VGG network) in the comparison. I believe that the joint success rate for this robust model will already be comparable to the proposed approach. - I am not convinced by the attack that the authors provide for the setting where the detection scheme is known. This attack seems similar to the approach studied in Carlini and Wagner, 2017 (Perfect-Knowledge Attack Evaluation) which was insufficient to break the randomization defense. Why did the authors not try something along the lines of the attack in Carlini and Wagner, 2017 (Looking deeper) that actually broke the aforementioned defense? Specifically, trying to find adversarial examples that have low uncertainty as predicted by the prior networks. The uncertainty loss -- minimizing KL between p_in(.pi|x_adv) and p(.pi|x_adv, .theta) -- could be added to cross entropy loss. - In Section 4.2, how do the authors generate black box attacks? If they are white box attacks on models trained with a different seed (as in Section 4.1) the results in 4.2 are surprising. Carlini and Wagner, 2017 found white-box attacks for randomization schemes transferrable and as per my understanding, this should be reflected in Fig 3, at least for prior work. - I am confused by the authors comment - “Figure 3c shows that in almost 100% of cases the attack yields the target class.” The joint success rate being lower than the success rate should convey that adversarial examples couldn’t be found in many of these cases. What was the value of the epsilon that was used in these plots? Quality, Novelty and Significance: The paper is written well, but clarity about the evaluation procedures is lacking in the main manuscript. I am also not convinced by the rigor of the evaluation of their detection methodology. Specifically: (1) they do not consider state-of-the-art attack models such as PGD and (2) the scheme they propose for a perfect knowledge attack seems insufficient. While the paper asks an important question, I do not find the results sufficiently novel or convincing. More broadly, I find the idea of using a secondary network to detect adversarial examples somewhat tenuous as it should be fairly easy for an adversary to break this other network as well.