Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors attempt to relate adversarial robustness with architectural properties of neural networks such as width and depth. They also provide some theoretical analysis in terms of Lipshitz upper bounds. There are some interesting empirical observations, but no definite conclusions can be drawn. 

main_review: - There are attempts to explain network robustness with Lipschitz upper bound, such as the theoretical results derived in section 3. But the bounds might be loose to explain what happens in experiments. For example, Figure 4b has significant decrease in empirical Lipschitz while maintaining clean accuracy, but there is no corresponding increase in adversarial robustness in Figure 3d. Indeed, the models with different widths at stage-3 all have similar PGD accuracies despite different empirical Lipschitz constant. There can be some other factors that is not captured here. 

- The empirical evaluations on the effect of width, depth, and tradeoff parameter \lambda on adversarial robustness are very thorough. But at the end it still seems unsatisfying as there is no obvious trend or strong conclusions we can draw from it. If we change the dataset from CIFAR to another dataset the conclusions can become different. Or put it in another way, if we replace adversarial robustness (PGD accuracy) with some other quantities of interest (e.g., clean accuracy), we can still talk about the effect of width and depth and \lambda on clean accuracy. Without a clear trend or strong conclusion this type of empirical evaluations can only be treated as data points but not as insights into the problem. 

- This paper is of good quality and very well-written, but the results presented do not seem to advance our understanding on the problem of adversarial robustness and neural architecture by a lot. This is a borderline paper to me. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper conducts an empirical study on the effect of the model architectural on the model robustness. The experimental results show that more parameters do not necessarily lead to more robust model. Reduce capacity of the deeper layers can improve the robustness. With the same architectures, there may exist one optimal architectural configuration for producing the most robust DNN. 
main_review: # Originality
The results of this paper are new and interesting. Readers can learn a lot of new knowledge from the empirical study. The related work is also well discussed.

# Quality
Overall, the paper is clear and the claims can be supported well by the results. Some detailed comments:
1. The study mainly focuses on WRN and CIFAR-10. It is not clear how can the findings generalise to other models or datasets. I appreciated the effort of the authors in applying the similar rules into other datasets (e.g., ImageNet). However, it will be more convincing if more results and analysis can be provided.
2. In section 5, the paper said that they applied the discovered configuration rule on the new models. It is not clear what are these rules and how to conduct such rules in the different models. For example, reduce capacity of deeper layers?  which layers are reduced?
3. The paper claimed that reducing capacity of the deeper layers can improve the robustness. Why? More theoretical analysis would be helpful.
4. In Table 2, it shows that the results from $CW_\infty $ on CIFAR-100 are not as what you expect. Could you provide more analysis?
 
# Clarity
The paper is written clearly and easy to follow.

# Significance
The results are promising and insightful.  Other researchers can benefit from the results 
limitations_and_societal_impact: The societal impact is well discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors evaluate the impact of network width and depth on the robustness of models trained through adversarial training.  Main contributions:
1. Theoretically show that increasing network width and depth leads to lower perturbation stability
2. Empirically evaluates the impact of changing depth and width and various stages of the WRN-34-10 architecture.  The authors find that decreasing the depth and width at the last stage can lead to higher robustness while decreasing depth and width at earlier stages of the network hurts robustness.
3. Find a width configuration that has high robustness on WRN-34-10 and show that this configuration can also be used to improve robustness on a variety of other base architectures.  The authors show that this pattern holds for various datasets and adversarial training techniques.
main_review: Originality: To the best of my knowledge, the design of the experiments is novel.  While previous works have looked into the impact of width and depth [1,2], in those experiments, the authors vary the overall width and depth of the network.  The authors of this paper perform a more in depth experiment in which they modify the width and depth of stages within the model, which allows for a better understanding of where in the architecture more width and depth helps.  On the theoretical result of how Lipschitz constant is upper bounded by some function of width and depth, [2] also provides an upper bound on Lipschitz constant related to network width and depth, so a few sentences comparison between the theoretical results from this paper and [2] should be added.

Quality: There is theory to show that width and depth decreases stability and there is good scope in experiments to show that decreasing width and depth at the last stage of the network improves robustness while earlier stages perform better with higher width and depth.  Additionally, experiments are performed along various architectures, datasets, and adversarial training methods to show that their findings generalize.  In terms of scope of experiments, I do have a few suggestions/questions:
- in section 4.3, the scaling is performed on the discovered width configuration.  What about the impact on scaling for depth?  Additionally, experiments in section 5 use the -R models which is the scaled width configuration.  What about varying the depth configuration for these models?  Ideally, we should see both the depth and width trends found in WRN-34-10 to hold for these other models as well.
- in section 4.4, Figure 4 includes plots of empirical Lipschitz constant and perturbation stability for varying width and depth at only the last layer.  I think it would be helpful to include plots for varying width and depth at earlier layers as well especially since Figure 5 suggests that reducing width at earlier layers does not necessarily reduce the Lipschitzness of the network.  Potentially, reducing depth and width at earlier stages would just decrease the clean accuracy while having little impact on the perturbation stability and empirical Lipschitz which I think ties in with the hypothesis that more parameters are needed in the earlier stages for the model to fit to the adversarial images during training.

Clarity: Overall, the writing is clear.  I feel like the transition between section 3 and 4 is a little awkward though since the first 3 subsections of Section 4 are unrelated to the theory.  A potential solution is to group 4.1-4.3 in 1 section and section 3 and 4.4 within another section.  Some minor comments:
- in Section 3, it would help to add a reference to Appendix A.2 to show that the Gaussian distributed weights assumption holds for an adversarially trained model.
- line 149, missing space between the words "robustness" and "are"
- the trends in Figure 5 are a little difficult to see

Significance: The results help guide researchers on how to design architectures that improve adversarial robustness.  The authors indicate that robustness can be improved even when decreasing the number of parameters by reducing the width/depth at the last stage of the network, which is useful in situations with limited storage.

[1] Gowal, Sven, et al. "Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples." arXiv preprint arXiv:2010.03593 (2020).
[2] Wu, Boxi, et al. "Does Network Width Really Help Adversarial Robustness?." arXiv preprint arXiv:2010.01279 (2020).
limitations_and_societal_impact: See main review for comments.  My main concern is that in Section 5, which empirically shows that the architecture configuration found by the authors generalizes across models/datasets/training techniques, only focuses on varying width while previous sections also analyzed the impact of depth.  These patterns in depth should ideally generalize as well, but there are no experiments to show this.
review_title: null

Meta-review: This paper studies the impact of network width and depth on the robustness of a model through adversarial training. Different from previous works that looked into the impact by varying the overall width and depth of a network, the authors perform stage-level experiments in which they modify the width and depth of different stages within a model. The findings, which indicate that decreasing the depth and width at the last stage can lead to higher robustness while decreasing depth and width at earlier stages of the network hurts robustness, are useful to the community. However, the upper bound on the Lipschitz constant proposed in this work does not seem to explain the empirical results well. We encourage the authors to discuss the consistency between the theoretical and empirical results more if the paper is finally accepted.
Decision: Accept (Poster)