Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a robust training against the union of $l_p$ threat models, i.e., training a robust model in l_inf, l_2, l_1, threat models. The key idea is to use a nuclear norm-based training procedure called NCAT. 
strengths_and_weaknesses: Strengths
-----------
- NCAT can achieve SOTA robust accuracy while requiring a single-step attack budget per mini-batch
- NCAT can scale to large datasets


Weaknesses
--------------
- The actual training performance (in terms of computation time) is not reported. Therefore, it is hard to compare the actual performance of NCAT against other robust training methods. 

- Prior work (Sriramanan et al.) 2021 has already used nuclear norms in the context of adversarial training, albeit for single-step attacks.   
questions: Please report the actual computation time for training using NCAT vs. other baseline adversarial training methods.
limitations: yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes an efficient robust training method to achieve adversarial robustness on multiple threat models (l_1,l_2,l_\inf). Specifically, it first study the adversarial training on  l_1 norm where normal adversarial training method could achieve a good performance. By combining the previously proposed nuclear-norm adversarial training and curriculum schedule, it finds it could achieve a better robust accuracy against l_1 adversarial attack. As l_2 and l_inf threat model have a similar behavior on the robustness, the paper then choose to alternatively switch the threat model to do nuclear-norm adversarial training to improve the efficiency. The experiments are conducted on cifar10 and imagenet100 on ResNet18 and wideresnet28. It shows the proposed method could achieve a better worst-case robustness while maintain a good scalabliilty.
strengths_and_weaknesses: Pros:
1. The paper is in general well-written and easy to follow except for the curriculum scheduling part.
2. The experiment result shows the proposed method could achieve a worst-case robust accuracy.

Cons:
1. The novelty of proposed method is quite limited. It is a fairy combination on nuclear norm adversarial training with the curriculum schedule. And curriculum schedule is also discussed in the previous works such as [1]. It is still unclear why the curriculum schedule is useful in the adversarial training or the proposed schedule only works for the l_1 threat model.
2. The main claim of improving the union of threat model is actually improving the adversarial training's performance on the l_1 threat model, which is not equal to improve the performance over the union. In other words, if the union doesn't include l_1 threat model, will the proposed method still outperform the baseline?
3. Although the proposed method is better in terms of the worst-case acc, they are sometimes worse on the average. The main improvement I think is on the training efficiency.

Minor:
1. Typos: line 219 a_\sigma(1)\geq a_\simga(1).. 
2. The dash line in Figure 1 (b)(c) is inconsistent with the legend.

[1] Zhang, Jingfeng, et al. "Attacks which do not kill training make adversarial learning stronger." International conference on machine learning. PMLR, 2020.
questions: 1. Does the weight average on the model parameter matter in Algorithm 1 line 13? I don't see any explanation on that. Also, which parameter is used in the end? theta or w?
2. The curriculum scheduling part is not clear to me. Could you elaborate more on that?
3. If the threat model doesn't include l_1 threat model, will the proposed method still have a good performance?
limitations: The papers has discussed its limitation on provable certificates on different l_p threat models and their union. However, I think one of big limitation is the paper requires the union to have l_1 threat model.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work tackles the lucrative goal of multi-norm perturbation robustness, which is largely constrained by efficient robustness against $L_1$-norm adversaries in the current literature. The authors propose a technique inspired by the Nuclear-Norm regularizer that helps achieve robustness against a union of adversaries efficiently. The curriculum-based method has clear performance gains, especially with complex datasets and large models.
strengths_and_weaknesses: # Strengths
- The paper is well written, with easy-to-follow reasoning for design/algorithmic choices, buildup, and review of relevant literature. Evaluation is also thorough (although I would prefer to see at last least some mention of deviation in results across runs).
- The observation (and perhaps affirmation) of $L_1$ and $L_{\infty}$ being the hardest threat models while aiming for union robustness is nice.

# Weaknesses
As such, I see no visible weaknesses in the paper, apart from minor comments mentioned below (and in the questions section). Good job!

# Minor comments
- Table 1: What does the $^+$ in NCAT$^+$ stand for? Please clarify in the table description.

questions: There is a non-trivial drop in clean-model accuracy when using the proposed method. I believe this is probably a side effect of learning more robust models, thereby limiting weak dataset-specific signals that the model might otherwise be latching on to. Nonetheless, it would be nice if the authors could address this drop better and do some kind of qualitative analysis. For instance, visual/statistical inspection of the (additional) cases where the models go wrong may help identify problematic examples (mislabeled), or specific concepts that seem to be hard if aiming for robustness as well.
limitations: Although societal impact is addressed in Section 7, I do not see much discussion around limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method, NCAT, to obtain models robust wrt $\ell_1$ or multiple $\ell_p$-norms efficiently, that is with single step adversarial training. NCAT combines Nuclear-Norm Adversarial Training (NuAT) with a curriculum for the size of the perturbations seen at training time to prevent catastrophic overfitting in 1-step adversarial training. Further, the paper applies NCAT to the multiple norms scenario, achieving results close to those of more expensive multi-step methods.
strengths_and_weaknesses: Strengths
- The goal of reducing the cost of adversarial training without degrading (too much) the resulting robustness is well established, and to my knowledge this is the first work to extend single step adversarial training to $\ell_1$ and multiple norms.

- The proposed method is simple and effective in the experimental evaluation on both CIFAR-10 (with two architectures) and ImageNet-100. Several additional experiments are presented in the supplements to further analyze and support NCAT.

Weaknesses
- NCAT is built combining several existing techniques (NuAT, curriculum for $\epsilon$, steepest ascent direction for the threat models of $\ell_1$ with the image domain constraints, alternating $\ell_1$ and $\ell_\infty$ when training for multiple norms), which to some extent limits the contributions of the paper.

Overall, I think that the novelty of the method is somehow limited, showing how it's possible to achieve with single step adversarial training robustness in challenging threat models a meaningful contribution.
questions: - While the paper is well-written, I think some more details about the curriculum (and maybe some training details) could belong to the main part, since that's one of the main components of NCAT. Also in Table 1, NCAT+ appears but I couldn't find where it's defined.

- An efficient method is particularly useful when training is particularly expensive, then it'd be interesting to scale NCAT to the full version of ImageNet.
limitations: The limitations are sufficiently addressed.
review_title: null

Meta-review: metareview: This paper received mixed recommendations that range from borderline reject to strong accept. After examining all reviewers' comments, author rebuttal, and the paper itself, I lean toward acceptance mainly because: (i) it seems that the author responses have addressed reviewers' main concerns (although some reviewers did not respond to the rebuttal); (2) the contributions made by the paper, as summarized in the paper and author rebuttal, are significant enough and could potentially facilitate the development of adversarial defense.

Nevertheless, the organization and presentation of the paper needs to be further improved. For instance, the novelty/contributions over some existing works (e.g., [1] [2]), the computational cost of the proposed method, and the possible limitations are insufficiently discussed in the main text of the original submission. I urge the authors to consider these issues and take careful note of the reviewers' comments and suggestions when preparing for the final version.

[1] F. Croce and M. Hein. Adversarial robustness against multiple lp-threat models at the price of one and how to quickly fine-tune robust models to another threat model. 
[2] G. Sriramanan, S. Addepalli, A. Baburaj, and V. B. Radhakrishnan. Towards efficient and effective adversarial training. In NeurIPS 2021

confidence: Less certain
Decision: Accept