Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Authors propose to use a modified autoencoder at the input of a network to ward off adversarial samples by purifying them. The encoder-decoder is trained with a discriminator which can identify whether the input is malicious or normal. A malicious input image is purified with the help of a search procedure in the latent space of the auto-encoder. The gradient based search obtains a latent code that corresponds to the reconstructed image that is closest to the input. The algorithm uses output of the decoder as a starting point in the search. Experiments show a reasonable performance against adversarial attacks on MNIST and F-MNIST images. The proposal seems to be an alternative to the Defence-GAN method of (Samangouei et. al. 2018) where search is computationally expensive - due to multiple starting points and many iterations. The ease of optimization in the proposed method comes at the cost of accuracy (Table 1). However, for a single starting point during search and fewer iterations, it outperforms Defense-GAN by a significant margin (Table 2). 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper aims to refine DefenseGAN (ICLR 18), where an autoencoder is used to initialize the search for projecting an adversarial examples to the manifold of real examples. 

The main contribution is to reduce the computational cost of DefenseGAN, the claim is "by an order of magnitude". 

Though the idea is good, I found the contribution to be too incremental for the paper to be accepted:
* The comparison should include the state of the art adversarial training defenses, PGD Adversarial Training (Madry et al; you might want to cite the ICLR 18 paper) and TRADES (Zhang et al., Interpreting adversarially trained convolutional NN, ICML 2019);
* I would consider the Szegedy baseline as obsolete;
* Fig. 4 is unclear: the percentage of adversarial examples detected by the detector, but quid of the false alarms; 

There are quite some typos (nosie; iterarion; tabel; unsafety) and missing words. The paper has been hastily written; and it includes one page more than allowed.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper combines an auto-encoder (AE) based approach to correct the adversarially perturbed samples with a GAN based approach for the same task. More specifically, the encoder of the AE is used to provide a better initialization for the latent space optimization employed in the Defense-GAN approach. 

The authors propose a two-stage inference process. First, the autoencoder is used to detect if an input sample is from the natural image distribution or adversarial distribution. In the latter case, Defense-GAN is employed that uses gradient descent in the latent space, with encoder output as the initialization, to find a natural or non-adversarial counterpart of the input sample. Results are reported for MNIST and F-MNIST that show that the proposed method is computationally cheaper than Defence-GAN.

Questions / Concerns:

 - The methods seem to heavily rely on the autoencoder's ability to detect adversarial samples. The detection performance can be still susceptible to white-box attacks. The methods also relies on the capacity of the auto-encoder to learn good representations/reconstructions. It will be useful to show the performance on more complicated datasets was learning a good AE model is more challenging.   
 - The autoencoder is also equipped with an adversarial loss such that the decoder produces realistic images. Is there any assurance that the latent distribution F(x) from the data and the prior p(z) will be similar. It might be useful to explicitly ensure this. The discriminator currently never sees images that re reconstructions of the true data samples. So the decoder behavior could be different for different regions of the input space. 
- It is not clear why the proposed method performs better on stringer white-box attacks compared to much weaker black-box attacks. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Summary: This paper proposes AE-GAN+sr, an auto-encoder based GAN for equipping neural networks with better defenses against adversarial attacks. The authors evaluate their method on black-box attacks, white-box attacks, and gray-box attacks on MNIST and Fashion-MNIST, and show decent empirical results when compared to baselines.

Decision: Reject. The writing was hard to follow and the experimental evaluations could have been stronger.

Supporting Arguments/Feedback: 
- There are a lot of phrases which should be reworded in the text (e.g. “it requires attention to” in paragraph 1, “detect if sample is from a normal distribution” at the end of Section 1, etc.) for better clarity. The writing was pretty hard to follow and the paper would really benefit overall after it has been improved.
- It was hard for me to tell from the text how exactly the models were implemented and trained (see Question #1). Making this point more clear, in either the main text or the Appendix, would be really helpful.
- The performance of AE-GAN+sr as evaluated on MNIST and FashionMNIST did not provide convincing evidence that it outperformed baselines such as Defense-GAN or adversarial training, though I did appreciate the authors’ additional analysis on the tradeoffs between computational cost and performance for the Defense-GAN. The results would also have been more compelling had the authors evaluated their method on more complex datasets such as CIFAR-10.

Questions:
- It wasn’t clear to me how the encoder-assisted search process (Section 3.4) works in practice. When you use gradient descent to find the best encoding for a corrupted input, how many steps do you need to take? Does this step happen while the autoencoder and GAN are being trained, or are those pre-trained and you just take additional gradient steps in the latent space?
- Given that the BiGAN also incorporates an encoder (and they were also benchmarked against in the experiments), where do the advantages of the AE-GAN+sr come from? I think making this point in the text would also be helpful as well.

review_title: Official Blind Review #2

Meta-review: The paper proposes a defense for adversarial attacks based on autoencoders that tries to find the closest point to the natural image in the output span of the decoder and "purify" the adversarial example. There were concerns about the work being too incremental over DefenseGAN and about empirical evaluation of the defense. It is crucial to test the defense methods against best available attacks to establish the effectiveness. Authors should also discuss and consider evaluating their method against the attack proposed in https://arxiv.org/pdf/1712.09196.pdf that claims to greatly reduce the defense accuracy of DefenseGAN. 
Decision: Reject