Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to look at making neural networks resistant to adversarial loss through the framework of saddle-point problems. They show that, on MNIST, a PGD adversary fits this framework and allows the authors to train very robust models. They also show encouraging results for robust CIFAR-10 models, but with still much room for improvement. Finally, they suggest that PGD is an optimal first order adversary, and leads to optimal robustness against any first order attack.

This paper is well written, brings new ideas and perfoms interesting experiments, but its claims are somewhat bothering me, considering that e.g. your CIFAR-10 results are somewhat underwhelming. All you've really proven is that PGD on MNIST seems to be the ultimate adversary. You contrast this to the fact that the optimization is non-convex, but we know for a fact that MNIST is fairly simple in that regime; iirc a linear classifier gets something like 91% accuracy on MNIST. So my guess is that the optimization problem on MNIST is in fact pretty convex and mostly respects the assumptions of Danskin's theorem, but not so much for CIFAR-10 (maybe even less so for e.g. ImageNet, which is what Kurakin et al. seem to find).

Considering your CIFAR-10 results, I don't think anyone should "suggest that secure neural networks are within reach", because 1) there is still room for improvement 2) it's a safe bet that someone will always just come up with a better attack than whatever defense we have now. It has been this way in many disciplines (crypto, security) for centuries, I don't see why deep learning should be exempt. Simply saying "we believe that our robust models are significant progress on the defense side" was enough, because afaik you did improve on CIFAR-10's SOTA; don't overclaim. 
You make these kinds of claims in a few other places in this paper, please be careful with that.

The contributions in your appendix are interesting. 
Appendix A somewhat confirms one of the postulates in Goodfellow et al. (2014): "The direction of perturbation, rather than the specific point in space, matters most. Space is not full of pockets of adversarial examples that finely tile the reals like the rational numbers".
Appendix B and C are not extremely novel in my mind, but definitely add more evidence. 
Appendix E is quite nice since it gives an insight into what actually makes the model resistant to adversarial examples.


Remarks:
- The update for PGD should be using \nabla_{x_t} L(\theta,x_t,y), (rather than only \nabla_x)?
- In table 2, attacking a with 20-step PGD is doing better than 7-step.  When you say "other hyperparameter choices didn’t offer a significant decrease in accuracy", does that include the number of steps? If not why stop there? What happens for more steps? (or is it too computationally intensive?)
- You only seem to consider adversarial examples created from your dataset + adv. noise. What about rubbish class examples? (e.g. rgb noise)

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: - The authors investigate a minimax formulation of deep network learning to increase their robustness, using projected gradient descent as the main adversary. The idea of formulating the threat model as the inner maximization problem is an old one. Many previous works on dealing with uncertain inputs in classification apply this minimax approach using robust optimization, e.g.: 

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2003/CSD-03-1279.pdf
http://www.jmlr.org/papers/volume13/ben-tal12a/ben-tal12a.pdf

In the case of convex uncertainty sets, many of these problems can be solved efficiently to a global minimum. Generalization bounds on the adversarial losses can also be proved. Generalizing this approach to non-convex neural network learning makes sense, even when it is hard to obtain any theoretical guarantees. 

- The main novelty is the use of projected gradient descent (PGD) as the adversary. From the experiments it seems training with PGD is very robust against a set of adversaries including fast gradient sign method (FGSM), and the method proposed in Carlini & Wagner (CW). Although the empirical results are promising, in my opinion they are not sufficient to support the bold claim that PGD is a 'universal' first order adversary (on p2, in the contribution list) and provides broad security guarantee (in the abstract). For example, other adversarial example generation methods such as DeepFool and Jacobian-based Saliency Map approach are missing from the comparison. Also it is not robust to generalize from two datasets MNIST and CIFAR alone.  

- Another potential issue with using projected gradient descent as adversary is the quality of the adversarial example generated. The authors show empirically that PGD finds adversarial examples with very similar loss values on multiple runs. But this does not exclude the possibility that PGD with different step sizes or line search procedure, or the use of randomization strategies such as annealing, can find better adversarial examples under the same threat model. This could make the robustness of the network rather dependent on the specific implementation of PGD for the inner maximization problem. 

- In Tables 3, 4, and 5 in the appendix, in most cases models trained with PGD are more robust than models trained with FGSM as adversary, modulo the phenomenon of label leakage when using FGSM as attack. However in the bottom right corner of Table 4, FGSM training seems to be more robust than PGD training against black box PGD attacks. This raises the question on whether PGD is truly 'universal' and provides broad security guarantees, once we add more first order attacks methods to the mix. 


review_title: Interesting experimental results, but insufficient to support some of the strong claims made in the paper

final_score: rating: 7: Good paper, accept
review_content: This paper consolidates and builds on recent work on adversarial examples and adversarial training for image classification. Its contributions:

 - Making the connection between adversarial training and robust optimization more explicit.

 - Empirical evidence that:
   * Projected gradient descent (PGD) (as proposed by Kurakin et al. (2016)) reasonably approximates the optimal attack against deep convolutional neural networks
   * PGD finds better adversarial examples, and training with it yields more robust models, compared to FGSM 

 - Additional empirical analysis:
   * Comparison of weights in robust and non-robust MNIST classifiers
   * Vulnerability of L_infty-robust models to to L_2-bounded attacks

The evidence that PGD consistently finds good examples is fairly compelling -- when initialized from 10,000 random points near the example to be disguised, it usually finds examples of similar quality. The remaining variance that's present in those distributions shouldn't hurt learning much, as long as a significant fraction of the adversarial examples are close enough to optimal.

Given the consistent effectiveness of PGD, using PGD for adversarial training should yield models that are reliably robust (for a specific definition of robustness, such as bounded L_infinity norm). This is an improvement over purely heuristic approaches, which are often less robust than claimed.

The comparison to R+FGSM is interesting, and could be extended in a few small ways. What would R+FGSM look like with 10,000 restarts? The distribution should be much broader, which would further demonstrate how PGD works better on these models. Also, when generating adversarial examples for testing, how well would R+FGSM work if you took the best of 2,000 random restarts? This would match the number of gradient computations required by PGD with 100 steps and 20 restarts. Again, I expect that PGD would be better, but this would make that point clearer. I think this analysis would make the paper stronger, but I don't think it's required for acceptance, especially since R+FGSM itself is such a recent development.

One thing not discussed is the high computational cost: performing a 40-step optimization of each training example will be ~40 times slower than standard stochastic gradient descent. I suspect this is the reason why there are results on MNIST and CIFAR, but not ImageNet. It would be very helpful to add some discussion of this.

The title seems unnecessarily vague, since many papers have been written with the same goal -- make deep learning models resistant to adversarial attacks. (This comment does not affect my opinion about whether or not the paper should be accepted, and is merely a suggestion for the authors.)

Also, much of the paper's content is in the appendices. This reads like a journal article where the references were put in the middle. I don't know if that's fixable, given conference constraints.
review_title: The unreasonable effectiveness of gradient descent

Meta-review: This paper presents new results on adversarial training, using the framework of robust optimization. Its minimax nature allows for principled methods of both training and attacking neural networks.

The reviewers were generally positive about its contributions, despite some concerns about 'overclaiming'. The AC recommends acceptance, and encourages the authors to also relate this work with the concurrent ICLR submission (https://openreview.net/forum?id=Hk6kPgZA-) which addresses the problem using a similar approach. 
Decision: Accept (Poster)