Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors claimed a classic adversarial training method, FGSM with random start, can indeed train a model that is robust to strong PGD attacks. Moreover, when it is combined with some fast  training methods, such as cyclic learning rate scheduling and mixed precision, the adversarial training time can be significantly decreased. The experiment verifies the authors' claim convincingly.
Overall, the paper provides a novel finding that could significantly change the adversarial training strategy. The paper is clearly written and easy to follow. I recommend the acceptance.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The main claim of this paper is that a simple strategy of randomization plus fast gradient sign method (FGSM) adversarial training yields robust neural networks. This is somewhat surprising because previous works indicate that FGSM is not a powerful attack compared to iterative versions of it like projected gradient descent (PGD), and it has not been shown before that models trained on FGSM can defend against PGD attacks. Judging from the results in the paper alone, there are some issues with the experiment results that could be due to bugs or other unexplained experiment settings. 

The most alarming part of the results is the catastrophic failure with larger step sizes 16/255 for CIFAR10 in Table 1. This is very strange because the method works well when using epsilon=10/255 to defend against an adversary with epsilon=8/255. 
The authors explain this with overfitting, but this is not satisfactory. Suppose I want to use the method to defend against an adversary with power epsilon=14/255, then it is conceivable that I would use a slightly larger step size, say 16/255, as suggested by the authors.  The results in the table tells me that this method will fail completely, because it cannot defend against epsilon=8/255, let alone the target perturbation 14/255. The method is probably not failing completely, because it does have good accuracy on clean data and does learn something. So it cannot be due to the model not having enough capacity to learn against an epsilon=16/255 adversary. 

The authors should check some potential issues with the experiments: 
1. Is there any label leakage in the FGSM training? 
2. The pseudo-code does not contain any projection onto the feasible set; the authors should check it. 

Since the claim of this paper is somewhat unexpected given previous works on defending against adversaries, the experiment results have to be very solid. With these issues with the experiments I don't believe the current paper is ready for publication yet. 

After rebuttal: 
The authors' new experiments and response answers most of my concerns. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper revisits Random+FGSM method to train robust models against strong PGD evasion attacks. Coupled together with tricks for accelerating natural training, such as cyclic learning rate, mixed precision, the robust models can be trained faster than previous methods. 

+The experimental results are impressive. The trained model is robust (at Madry’s PGD level), and the total training procedure is fast (6 min for CIFAR-10 and 12 hr for ImageNet).

+ The method is simple, and I guess reproducible. 

+The paper shows surprising facts of a well-known method.

+The paper is generally well-written and easy to follow.

I do have some concerns of the work
- The paper is empirical and the techniques are combinations of previous methods. Even for the surprising fact that Random+FGSM, it has been discussed in several previous papers, for example,  ICLR 2019 Defensive Quantization: When Efficiency Meets Robustness  https://openreview.net/forum?id=ryetZ20ctX. So the main contribution of the paper is limited to show RFGSM works well when combined with optimization tricks like cyclic learning rate. 

-In previous methods claiming random+FGSM can train robust model, their method seems to be slightly different from Alg 3 in page 4 of this paper. The alg in this paper seems to be identical to Madry’s implementation of R-FGSM, which is shown not robust to PGD attacks. See discussions in https://openreview.net/forum?id=rJzIBfZAb and https://openreview.net/forum?id=ryetZ20ctX. I would like the authors to clarify their method to resolve such conflicts and make it clear how R-FGSM can be as robust as PGD as in table 1. 

-The first two paragraphs of section 4.1 seem to be inaccurate. One important trick in the “adversarial training for free” paper is to replay each minibatch m times. It is hard to say how much nonzero initialization helps. According to “universal adversarial training” (https://arxiv.org/pdf/1811.11304.pdf). It may help, but cannot compete with Madry’s PGD training when defending against PGD attacks. 

-I am not sure if using a larger norm 1.25 * \epsilon is a fair comparison. A baseline of PGD training bounded by 1.25 * \epsilon would help. 

-Could the authors combine table 4 and 5 for easy comparison of robust accuracy and training time? Did the authors try the optimization tricks on ImageNet for the baseline free adversarial training method?



================== after rebuttal =================
I change my rating to weak accept. I tend to accept for the following reason
(1) there seems to be no obvious flaw in the authors implementation. I quickly skimmed their code, and looks like a few researchers have tried their code and responded in public discussion. The surprising robustness of RFGSM, though the originality is questionable and the technical difference comparing to previous methods are subtle, seems to hold true. 
(2) The authors work hard to address the comments. 
I still have some concerns, mainly regarding the fairness of experimental comparison. 
(1) As pointed out in public discussion, the success of the proposed RFGSM relies on early stopping. I am wondering if early stopping also helps other methods since it turns out to be some sort of selection procedure.
(2) The authors did not update time in table 1 for CIFAR-10 results, which I consider almost no extra efforts. I am wondering how much more time each method needs from 45% in figure 2 to higher robust accuracy in table 1.
(3) I cannot understand why the proposed method is a particular good fit with cyclic LR and low precision tricks comparing to other methods. 
review_title: Official Blind Review #1

Meta-review: This paper provides a surprising result: that randomization and FGSM can produce robust models faster than previous methods given the right mix of cyclic learning rate, mixed precision, etc. This paper produced a fair bit of controversy among both the community and the reviewers to the point where there were suggestions of bugs, evaluation problems, and other issues leading to the results. In the end, the authors released the code (and made significant updates to the paper based on all the feedback). Multiple reviewers checked the code and were happy. There was an extensive author response, and all the reviewers indicated that their primary concerns were address, save concerns about the sensitivity of step-size and the impact of early stopping.

Overall, the paper is well written and clear. The proposed approach is simple and well explained. The result is certainly interesting, and this paper will continue to generate fruitful debate. There are still things to address to improve the paper, listed above. I strongly encourage the authors to continue to improve the work and make a more concerted effort to carefully discuss the impacts of early stopping.

Decision: Accept (Poster)