Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes an unsupervised method, called Parallel Checkpointing Learners (PCL), to detect and defend adversarial examples. The main idea is essentially learning the manifold of the data distribution and using Gaussian mixture models (GMMs) and dictionary learning to train a "reformer" (without seeing adversarial examples) to detect and correct adversarial examples. With PCL, one can use hypothesis testing framework to analyze the detection rate and false alarm of different neural networks against adversarial attacks. Although the motivation is well grounded, there are two major issues of this work: (i) limited  novelty - the idea of unsupervised manifold projection method has been proposed in the previous work; and (ii) insufficient attack evaluations - the defender performance is evaluated against weak attacks or attacks with improper parameters. The details are as follows.

1.  Limited novelty and performance comparison - the idea of unsupervised manifold projection method has been proposed and well-studied in "MagNet: a Two-Pronged Defense against Adversarial Examples", appeared in May 2017. Instead of GMMs and dictionary learning in PCL,  MagNet trains autoencoders for defense and provides sufficient experiments to claim its defense capability. On the other hand, the authors of this paper seem to be not aware of this pioneering work and claim "To the best of our knowledge, our proposed PCL methodology is the first unsupervised countermeasure that is able to detect DL adversarial samples generated by the existing state-of-the-art attacks", which is obviously not true. More importantly, MagNet is able to defend the adversarial examples very well (almost 100% success) no matter the adversarial examples are close to the information manifold or not. As a result, the resulting ROC and AUC score are expected be better than PCL. In addition, the authors of MagNet also compared their performance in white-box (attacker knowing the reformer), gray-box (having multiple independent reformers), and black-box (attacker not knowing the reformer) scenarios, whereas this paper only considers the last case.

2. Insufficient attack evaluations - the attacks used in this paper to evaluate the performance of PCL are either weak (no longer state-of-the-art) or incorrectly implemented. For FGSM, the iterative version proposed by (Kurakin, ICLR 2017) should be used. JSMA and deep fool are not considered strong attacks now (see Carlini's bypassing 10 detection methods paper). Carlini-Wagner attack is still strong, but the authors only use 40 iterations (should be at least 500) and setting the confidence=0, which is known to be producing non-transferable adversarial examples. In comparison, MagNet has shown to be effective against different confidence parameters. 

In summary, this paper has limited novelty, incremental contributions, and lacks convincing experimental results due to weak attack implementation.  
 



review_title: An unsupervised manifold projection method for defending adversarial examples based on Gaussian mixture models and dictionary learning. However, similar methodology has been proposed in previous works and the attack evaluations are too weak to claim the contributions.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper present a method for detecting adversarial examples in a deep learning classification setting.  The idea is to characterize the latent feature space (a function of inputs) as observed vs unobserved, and use a module to fit a 'cluster-aware' loss that aims to cluster similar classes tighter in the latent space. 

Questions/Comments:

- How is the checkpointing module represented?  Which parameters are fit using the fine-tuning loss described on page 3? 

- What is the rationale for setting the gamma (concentration?) parameters to .01?  Is that a general suggestion or a data-set specific recommendation?

- Are the checkpointing modules designed to only detect adversarial examples?  Or is it designed to still classify adversarial examples in a robust way?

Clarity: I had trouble understanding some of this paper.  It would be nice to have a succinct summary of how all of the pieces presented fit together, e.g. the original victim network, fine-tuning loss, per-class dictionary learning w/ OMP.  

Technical: It is hard to tell how some of the components of this approach are technically justified. 

Novel: I am not familiar enough with adversarial deep learning to assess novelty or impact. 
review_title: interesting method; difficult to follow

final_score: rating: 7: Good paper, accept
review_content: Summary:
 The paper presents an unsupervised method for detecting adversarial examples of neural networks. The method includes two independent components: an ‘input defender’ which tried to inspect the input, and a ‘latent defender’ trying to inspect a hidden representation. Both are based on the claim that adversarial examples lie outside a certain sub-space occupied by the natural image examples, and modeling this sub-space hence enables their detection. The input defender is based on sparse coding, and the latent defender on modeling the latent activity as a mixture of Gaussians. Experiments are presented on MInst, Cifar10, and ImageNet.
 
-	Introduction: The motivation for detecting adversarial examples is not stated clearly enough. How can such examples be used by a malicious agent to cause damage to a system? Sketching some such scenarios would help the reader understand why the issue is practically important. I was not convinced it is. 
Page 4: 
-	Step 3 of the algorithm is not clear:
o	How exactly does HDDA model the data (formally) and how does it estimate the parameters? In the current version, the paper does not explain the HDDA formalism and learning algorithm, which is a main building block in the proposed system (as it provides the density score used for adversarial examples detection). Hence the paper cannot be read as a standalone document. I went on to read the relevant HDDA paper, but it is also not clear which of the model variants presented there is used in this paper.
o	What is the relation between the model learned at stage 2 (the centers c^i) and the model learnt by HDDA? Are they completely different models? Or are the C^I used when learning the HDDA model (and how)? 
If these are separate models, how are they used in conjunction to give a final density score? If I understand correctly, only the HDDA model is used to get the final score, and the C^i are only used to make the \phy(x) representation more class-seperable. Is that right?
-	Figure 4, b and c: it is not clear what the (x,y,z) measurements plotted in these 3D drawings are (what are the axis).
Page 5:
-	Section 2: the risk analysis is done in a standard Bayesian way and leads to a ratio of PDFs in equation 5. However, this form is not appropriate for the case presented at this paper, since the method presented only models one of these PDFs (Specifically p(x | W1)  - there is not generative model of p(x|W2)).  
-	The authors claim in the last sentence of the section that p(x|W2) is equivalent to 1-p(x|W1), but this is not true: these are two continuous densities, they do not sum to 1, and a model of p(x|W2) is not available (as far as I understand the method)
Page 6:
-	How is equation 7) optimized?
-	Which patchs are extracted from images, for training and at inference time? Are these patchs a dense coverage of the image? Sparsely sampled? Densely sampled with overlaps?
-	Its not clear enough what exactly is the ‘PSNR’ value which is used for the adversarial example detection, and what exactly is ‘profile the PSNR of legitimate samples within each class’. A formal definition of PSNR and’profiling’ is missing (does profiling simply mean finding a threshold for filtering?)
Page 7:
-	Figure 7 is not very informative. Given the ROC curves in figure 8  and table 1 it is redundant. 

Page 8:
-	The results in general indicate that the method is much better than chance, but it is not clear if it is practical, because the false alarm rates for high detection are quite high. For example on ImageNet, 14.2% of the innocent images are mistakenly rejected as malicious to get 90% detection rate. I do not think this working point is useful for a real application
-	Given the high flares alarm rate, it is surprising that experiments with multiple checkpoints are not presented (specifically as this case of multiple checkpoints is discussed explicitly in previous sections of the paper).  Experiments with multiple checkpoints are clear required to complete the picture regarding the empirical performance of this method
-	The experiments show that essentially, the latent defenders are stronger than the input defender in most cases. However, an ablation study of the latent defender is missing: Specifcially, it is not clear a) how much does stage 2 (model refinement with clusters)  contribute to the accuracy (how does the model do without it? And 3) how important is the HDDA and the specific variant used (which is not clear) important: is it important to model the Gaussians using a sub-space? Of which dimension?

Overall:
Pros:
-	 A nice idea with some novelty,  based on a non-trivial observation
-	The experimental results how the idea holds some promise
Cons
-	The method is not presented clearly enough: the main component modeling the network activity is not explained (the HDDA module used)
-	The results presented show that the method is probably not suitable for a practical application yet (high false alarm rate for good detection rate)
-	Experimental results are partial: results are not presented for multiple defenders, no ablation experiments


After revision:
Some of my comments were addressed, and some were not.
Specifically, results were presented for multiple defenders and some ablation experiments were highlihgted
Things not addressed:
 - The risk analysis is still not relevant. The authors removed a clearly flawed sentence, but the analysis still assumes that two densities (of 'good' and 'bad' examples) are modeled, while in the work presented only one of them is. Hence this analysis does not add anything to the paper-  it states a general case which does not fit the current scenario and its relation to the work is not clear. It would have been better to omit it and use the space to describe HDDA and the specific variant used in this work, as this is the main tool doing the distinction.

I believe the paper should be accepted.

review_title: A nice novel idea, but the empirical evaluation is medium and some details are missing in the presentation

Meta-review: The paper proposes a method to detect and correct adversarial examples at the input stage (using a sparse coding based model) and/or at a hidden layer (using a GMM). These detector/corrector models are trained using only the natural examples. While the proposed method is interesting and has some novelty wrt to the specific models used for detection/correction (ie sparse coding and GMMs), there are crucial gaps in the empirical studies:

- It does not compare with a highly relevant prior work MagNet (Meng and Chen, 2017) which also detects and corrects adversarial examples by modeling the distribution of the natural examples

- The attacks used in the evaluations do not consider the setting where the existence (and architecture) of the defender models is known to the attacker

- It does not evaluate the method on a stronger PGD attack (also known as iterative FGSM)
Decision: Reject