Title: Certifying Some Distributional Robustness with Principled Adversarial Training
Authors: Aman Sinha
Hongseok Namkoong
John Duchi
Venue: ICLR 2018
Keywords: adversarial training
distributionally robust optimization
deep learning
optimization
learning theory

Abstract: Neural networks are vulnerable to adversarial examples and researchers have proposed many heuristic attack and defense mechanisms. We address this problem through the principled lens of distributionally robust optimization, which guarantees performance under adversarial input perturbations.  By considering a Lagrangian penalty formulation of perturbing the underlying data distribution in a Wasserstein ball, we provide a training procedure that augments model parameter updates with worst-case perturbations of training data. For smooth losses, our procedure provably achieves moderate levels of robustness with little computational or statistical cost relative to empirical risk minimization. Furthermore, our statistical guarantees allow us to efficiently certify robustness for the population loss. For imperceptible perturbations, our method matches or outperforms heuristic approaches.


TL;DR: We provide a fast, principled adversarial training procedure with computational and statistical performance guarantees.

Review discussion:
This paper proposes a principled methodology to induce distributional robustness in trained neural nets with the purpose of mitigating the impact of adversarial examples. The idea is to train the model to perform well not only with respect to the unknown population distribution, but to perform well on the worst-case distribution in some ball around the population distribution.
In particular, the authors adopt the Wasserstein distance to define the ambiguity sets. This allows them to use strong duality results from the literature on distributionally robust optimization and express the empirical minimax problem as a regularized ERM with a different cost. The theoretical results in the paper are supported by experiments.
Overall, this is a very well-written paper that creatively combines a number of interesting ideas to address an important problem.
This paper applies recently developed ideas in the literature of robust optimization, in particular distributionally robust optimization with Wasserstein metric, and showed that under this framework for smooth loss functions when not too much robustness is requested, then the resulting optimization problem is of the same difficulty level as the original one (where the adversarial attack is not concerned). I think the idea is intuitive and reasonable, the result is nice.
Although it only holds when light robustness are imposed, but in practice, this seems to be more of the case than say large deviation/adversary exists. As adversarial training is an important topic for deep learning, I feel this work may lead to promising principled ways for adversarial training.
In this very good paper, the objective is to perform robust learning: to minimize not only the risk under some distribution P_0, but also against the worst case distribution in a ball around P_0.
Since the min-max problem is intractable in general, what is actually studied here is a relaxation of the problem: it is possible to give a non-convex dual formulation of the problem. If the duality parameter is large enough, the functions become convex given that the initial losses are smooth.
What follows are certifiable bounds for the risk for robust  learning and stochastic optimization over a ball of distributions. Experiments show that this performs as expected, and gives a good intuition for the reasons why this occurs: separation lines are 'pushed away' from samples, and a margin seems to be increased with this procedure.
We thank the reviewers for their time and positive feedback. We will use the comments and suggestions to improve the quality and presentation the paper.
In addition to cleaning up our exposition, we added some content to make our main points more clear. We address these main revisions below.
Our formulation (2) is general enough to include a number of different adversarial training scenarios. In Section 2 (and more thoroughly in Appendix D), we detail how our general theory can be modified in the supervised learning setting so that we learn to defend against adversarial perturbations to only the feature vectors (and not the labels). By suitably modifying the cost function that defines the Wasserstein distance, our formulation further encompasses other variants such as adversarial perturbations only to a fixed small region of an image.
We emphasize that our certificate of robustness given in Theorem 3 applies for any level of robustness \rho. Our results imply that the output of our principled adversarial training procedure has worst-case performance no worse than this data-dependent certificate. Our certificate is efficiently computable, and we plot it in Figure 2 for our experiments. We see that in practice, the bound indeed gives a meaningful performance guarantee against attacks on the unseen test sets.
While the primary focus of our paper is on providing provable defenses against imperceptible adversarial perturbations, we supplement our previous results with a more extensive empirical evaluation. In Appendix A. 4, we augment our results by evaluating performance against L2-norm adversarial attacks with larger adversarial budgets (higher values of \rho or \epsilon). Our method also becomes a heuristic for such large values of adversarial budgets, but we nevertheless match the performance of other methods (FGM, IFGM, PGM) trained against L2-norm adversaries. In Appendix A.
5. 1, we further compare our method——which is trained to defend against L2-norm attacks——with other adversarial training algorithms trained against inf-norm attacks. We also propose a new (heuristic) proximal algorithm for solving our Lagrangian problem with inf-norms, and test its performance against other methods in Appendix A. 5. 2. In both sections, we observe that our method is competitive with other methods against imperceptible adversarial attacks, and performance starts to degrade as the attacks become visible to the human eye.
Again, we appreciate the reviewers' close reading and thoughtful comments.
[I put my reply here as the threads below are now a bit hard to follow.]
Thank you for responding to my comments and making the effort to provide more data. This indeed helps me understand this work better.
I agree that studying the regime of small adversarial perturbation budget epsilon is a very valid research goal. I think, however, that it is important to explicitly mention in the paper that this is the target. Especially, as the proposed methods seem to be inherently restricted to apply to only such a small epsilon regime.
I am not sure though that I agree with the argument why the regime of larger values of epsilon might be less interesting. Yes, some of the larger perturbations will be clearly visible to a human, but some (e.g., the ones that correspond to a change of the background color or its pattern) will not - and we still would like to be robust to them.
After all, security guarantees are about getting "for all", not "for some" guarantees.
Now, regarding being explicit about the constants in the bounds, I agree that many optimization and statistical learning guarantees do not provide not provide explicit constants.
However, I think the situation in the context considered here is fundamentally different.
After all, for example, in the context of generalization bounds, we always have a meaningful way of checking if a given bound "triggered" for a given model and dataset by testing its performance on a validation/test set.
When we talk about robustness guarantee, the whole point is to have it hold even against attacks that we are not able to produce ourselves (but the adversary might).
Then, we really need a very concrete guarantee of the form "(With high probability) the model classifies correctly 90% of the test set against perturbation budget of epsilon <= 0.1”.
In the light of this, providing a guarantee of the form "(With high probability) the model correctly classifies 90% of the test set against perturbation budget of some positive epsilon", which is what the proposed guarantees seem to provide, is somewhat less meaningful. (One could argue that, after all, there is always some positive epsilon for which the model is robust.)
It might be worth noting that, e.g., for MNIST, we currently are able to deliver guarantees of the former (explicit) type.
For instance, there is a recent work of Kolter and Wong (https://arxiv.org/abs/1711.00851).
Although they provide such guarantees via verification techniques and not by proving an explicit generalization bound.
Finally, I am not sure how much bearing the formal NP-hardness of certifying the robustness has here. (I assume you are referring to the result in Appendix B.) Could you elaborate?
Thank you for the detailed follow-up.
We will make the point that we deal with imperceptible changes clearer in the paper. We had emphasized that our work is motivated by imperceptible adversarial perturbations from the second paragraph of the paper. We will make this point even clearer and quantify our statements on performance so that there is no confusion that we mainly consider imperceptible changes.
As we have noted in our previous response, we agree with you in that robustness to larger perturbations is an important research direction. The point we made in our original response is that infinity-norms may not be the most appropriate norm to consider in this perceptible attack setting.
For example, a 1-norm-constrained adversary can change a few pixels in a very meaningful way--with infinity-norms approaching 1--which may be a more suitable model for a perceptible adversary. There are a number of concurrent works on this topic that we believe could lead to more robust learning systems.
It is still open whether distributionally robust algorithms (empirically) allow hedging against large adversarial perturbations. At this point, we believe it would be imprudent to call this class of methods “inherently restricted” to the small perturbation regime;
indeed, any heuristic method (such as one based on projected gradient descent) has the same restrictions, at least in terms of rigorous guarantees. A more thorough study—on more diverse datasets, model classes and hyperparameter settings—should be conducted in order to draw any meaningful conclusions. We hope to contribute to this effort in the future but we invite others as well, since we believe this is an important question for the community to answer.
Our certificate of robustness given in Theorem 3 is efficiently computable for small values of rho, or equivalently, for imperceptible attacks.
Hence, this data-dependent certificate provides a upper bound on the worst-case loss so that you are guaranteed to do no worse than this number with high probability. For the achieved level of robustness (rho hat in our notation), our bounds do imply that we are robust to perturbation budgets of this size.
Hence, we would argue that Theorem 3 is indeed a flavor of result that satisfies the desiderata you described.
There are limitations, and we hope that subsequent work will improve our learning guarantees with a better dependence on model size. This criticism, however, largely applies to most learning-theoretic results applied to deep learning.
As we mentioned in our introduction, we agree that recent advances in verification techniques for deep learning are a complementary and important research direction for achieving robust learning systems. Our understanding of these techniques is that they currently have prohibitive computational complexity, even on small datasets such as MNIST. Our results complement these approaches by providing a weaker statistical guarantee with computational effort more comparable to the vanilla training times.
The motivation of this paper comes from the fact that formal guarantees on arbitrary levels of robustness is NP-hard. We study the regime of small to moderate levels of robustness to provide guarantees for this regime.
Thanks to the reviewers, AC, and senior PC for the valuable feedback; the controversy around the paper has certainly been elucidating. To reflect our understanding of the results in the paper, which we hope others share, we have updated our paper’s title and introduction to make more explicit the claims (and concomitant limitations) of the work, better situating it in relation to other research. It seems that there was confusion about the results in the paper, and we have tried to clear up that our approach seeks a sweet spot balancing the goals of computational efficiency and certification of robustness. In the original submission, we noted in the introduction that "The key feature of the penalty problem (2) is that moderate levels of robustness are achievable at essentially no computational or statistical cost for smooth losses $\ell$.
” We’ve tried to make this facet of the work more apparent throughout the paper, noting the connection between small perturbations/imperceptible changes to moderate levels of robustness. We believe this paper is a step towards efficiently building certifiably stable deep networks, though of course substantial work remains. We note a few possible directions for future research in the paper along with explicit limitations of our work. It is plausible to us that principled guarantees, even for complex deep networks, are possible with reasonable computational tools, and we hope others will continue to focus on these efforts.
We have fixed some typos in the main text as well as the appendices. Thanks to members of the community for their feedback and interest in our paper.

Meta-review:
role: AC
data: comment: You have been contacted now by the Area Chair and the Program Chair and asked to respond to comments by the Area Chair. It is imperative that you respond.
role: Author
data: value: comment: We just received an email notification abut this comment a few minutes ago and somehow did not receive any notification of the original comment uploaded on 21 January. We will upload a response later today.
role: AC
data: value: comment: Sorry for the rush created by this likely OpenReview bug.  A response today would be most appreciated!
role: Author
data: value: comment: Apologies for the (evidently) tardy response. We have now uploaded a response to the area chair's comments (see below).
role: PC
data: comment: This paper attracted strong praise from the reviewers, who felt that it was of high quality and originality.  The broad problem that is being tackled is clearly of great importance.

This paper also attracted the attention of outside experts, who were more skeptical of the claims made by the paper. The technical merits do not seem to be in question, but rather, their interpretation/application. The perception by a community as to whether an important problem has been essentially solved can affect the choices made by other reviewers when they decide what work to pursue themselves, evaluate grants, etc. It's important that claims be conservative and highlight the ways in which the present work does not fully address the broader problem of adversarial examples.

Ultimately, it has been decided that the paper will be of great interest to the community. The authors have also been entrusted with the responsibility to consider the issues raised by the outside expert (and then echoed by the AC) in their final revisions.

One final note: In their responses to the outside expert, the authors several times remark that the guarantees made in the paper are, in form, no different from standard learning-theoretic claims: "This criticism, however, applies to many learning-theoretic results (including those applied in deep learning)." I don't find any comfort in this statement. Learning theorists have often focused on the form of the bounds (sqrt(m) dependence and, say, independence from the # of weights) and then they resort to empirical observations of correlation to demonstrate that the value of the bound is predictive for generalization. because the bounds are often meaningless ("vacuous") when evaluated on real data sets. (There are some recent examples bucking this trend.) In a sense, learning theorists have gotten off easy. Adversarial examples, however, concern security, and so there is more at stake. The slack we might afford learning theorists is not appropriate in this new context. I would encourage the authors to clearly explain any remaining work that needs to be done to move from "good enough for learning theory" to "good enough for security". The authors promise to outline important future work / open problems for the community. I definitely encourage this.






Decision: Accept (Oral)