Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper developed a method for defending deep neural networks against adversarial attacks based on generative cleaning networks with quantized nonlinear transform. The network is claimed to recover the original image while cleaning up the residual attack noise. The authors developed a detector network, which serves as the dual network of the target classifier network to be defended, to detect if the image is clean or being attacked. This detector network and the generative cleaning network are jointly trained with adversarial learning so that the detector network cannot find any attack noise in the output image of generative cleaning network. The experimental results demonstrated that the proposed approach outperforms the state-of-art methods by large margins in both white-box and black-box attacks. 

A few comments: 

1. It does not provide theoretical reasons why the prosed method can defend against those attacks. 

2. The experiments are a bit messy and the attacks' setup need to improve. 

3. The proposed defense showed only empirical results against the target attack. It seems to provide no theoretical / provable guarantees. 



review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper proposes a new method to defend a neural network agains adversarial attacks (both white-box and black-box attacks). By jointly training a Generative Cleaning Network with quantized nonlinear transform, and a Detector Network, the proposed cleans the incoming attacked image and correctly classifies its true label. The authors use state-of-the-art attack methods on various models, and the proposed model consistently outperforms all baseline models, even dramatically outperforming them for some specific attack methods.

Comment:
Is there a reason the authors did not test the same set of attack methods for SVHN as they did for CIFAR-10?
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper proposes a method for adversarial defense based on generative cleaning.

The paper does not follow any of the best practices for evaluating adversarial robustness, e.g. in these two papers:
"On Evaluating Adversarial Robustness" https://arxiv.org/abs/1902.06705
"Obfuscated Gradients Give a False Sense of Security" https://arxiv.org/abs/1802.00420

For instance the paper does not use a large number of PGD iterations (10 is too small) and does not check that accuracies go to zero for large epsilon (an important sanity check to reveal gradient masking). In the one place where a larger number of attack iterations is used (100 for BPDA) the gap with adversarial training mostly vanishes.

In the absence of these best practices it is impossible to assess the validity of the results, so the paper should be rejected.
review_title: Official Blind Review #3

Meta-review: This paper presents a method to defend neural networks from adversarial attack. The proposed generative cleaning network has a trainable quantization module which is claimed to be able to eliminate adversarial noise and recover the original image. 
After the intensive interaction with authors and discussion, one expert reviewer (R3) admitted that the experimental procedure basically makes sense and increased the score to Weak Reject. Yet, R3 is still not satisfied with some details such as the number of BPDA iterations, and more importantly, concludes that the meaningful numbers reported in the paper show only small gains, making the claim of the paper less convincing. As authors seem to have less interest in providing theoretical analysis and support, this issue is critical for decision, and there was no objection from other reviewers. After carefully reading the paper myself, I decided to support the opinion and therefore would like to recommend rejection. 

Decision: Reject