Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to do adversarial training on multiple L_p norm perturbation models simultaneously, to make the model robust against various types of attacks. 

[Novelty] I feel this is just a natural extension of adversarial training. If we define the perturbation set in PGD to be S, then in general S can be union of perturbation set of several L_p norm, and the resulting algorithm will be MSD (everytime you do a gradient update and then find the worst case projection in S). It would be interesting to study the convergence of this kind of algorithms, since S is no longer convex, the projection is trickier to define. Unfortunately this is not discussed in the paper. 

In terms of experiments, this is an interesting data point to show that we can have a model that is (weakly) robust to L1, L2 and Linf norms simultaneously. However, the results are not surprising since there's more than 10% performance decreases compared to the original adversarial training under each particular attack. So it's still not clear whether we can get a model that simultaneously achieves L1, L2, Linf robust error comparable to original PGD training. 

[Performance] 
- It seems MSD is not always better than others (worst PGD and PGD Aug). For MNIST, MSD performs poorly on Linf norm and it's not clear why.
- There's significant performance drop in clean accuracy, especially MSD on MNIST data. 

[Suggestions]
- As mentioned before, studying the convergence properties of the proposed methods will be interesting. 
- It will be interesting if you can train on a set of perturbation models and make it also robust to another perturbation not in the training phase. For instance, can we apply the proposed method to L{1,inf} in training and generalize to L2 perturbation? 

=====
Thanks for the response. I still have concerns about novelty so would like to keep my rating unchanged. 


review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper adversarially trains models against l_p norms where p is of there different values. They then propose a method which does somewhat better than the obvious way of adversarially training against more than one l_p perturbation.
The motivation for the paper is limited, in that they suggest previous works have suggested adversarial training itself "overfits" to the given l_p norm. This isn't surprising that it works, since the straightforward baseline works. They make it seem surprising by suggesting that ABS suggested adversarial training is doomed and cannot provide robustness to l_1, l_2, l_\infty norms simultaneously. The other motivation is that this is a step toward studying an expanded threat model, but the authors have not demonstrated that the learned representations are any bit more robust to common corruptions (could the authors show the generalization performance on CIFAR-10-C or generalization to unforeseen corruptions?). Without further evidence, we are left to believe this only helps for this narrow threat model. Overall the paper is deficient in creativity and generality, so I vote for rejection.

Small comments:

> take more time than a single norm,  it is a step closer towards the end goal of truly robust models, with adversarial robustness against all perturbations.
Please show model performance on CIFAR-10-C since if the model is more robust, it should hopefully be more robust to stochastic adversaries.

> has claimed that adversarial training “overfits” to the particular type of perturbation used to generate the adversarial examples
Wouldn't this be that l_\infty training fits specifically to l_\infty examples, not that robust optimization cannot handle more than one norm at a time? Who is claiming that?

> First, we show that even simple aggregations of different adversarial attacks can achieve competitive universal robustness against multiple perturbations models without resorting to complex architectures.
I am not sure this was in doubt. The phrase "universal robustness" is misleading.

How were the budgets chosen for l_2 and l_1? Those values seem small.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary of the paper: The paper describes adversarial training aiming to build models that are robust to multiple adversarial attacks - with L_1, L_2 and L_inf norms.  The method is a based on adversarial training against a union of adversaries. That union is created by taking (projected) gradient steps like PGD (Kurakin 2017), but choosing the maximal loss over GD steps for L1, L2, L_inf at each step. 

Strengths: The topic is trendy and interesting. The proposed algorithm is simple and easy to implement. The experimental results demonstrate improvement over several baselines. 

Weaknesses: 
-- I am missing a more systematic comparisons to baseline defenses in the experiments. Figures 2 and 3 should have shown the accuracy as a function of radius also for PGD-aug, PGD-worst, Schott et al. Also what about comparisons to the latest SoTA defenses, e.g. recent baselines from from  
www.robust-ml.org/defenses/. 

-- An implicit expectation from this paper is that it addresses the key issue of  "Defend against one attack but face a different attack". The paper could have done more to advance our understanding of this issue. Specifically: 

The approach improves over baselines for the "all attacks" mode, but under-performs compared with PGDaug and PGDworst when attacked with a single norm (Tab 1). 

While this is expected and probably cannot be avoided, it leaves the reader with an unclear conclusion about risk tradeoffs. It would have been useful to clarify the regime of mixtures of attacks where the various approaches are best. For instance, if one uses a of mix attack samples from the three norms, what mixtures would it be best to defend using MSD, wand what mixtures would it be best to use PGD-aug? or ABS? 


review_title: Official Blind Review #2

Meta-review: Thanks to the authors for submitting the paper and providing further explanations and experiments. This paper aims to ensure robustness against several perturbation models simultaneously. While the authors' response has addressed several issues raised by the reviewers, the concern on the lack of novelty remains. Overall, there is not enough support among the reviewers for the paper to be accepted.
Decision: Reject