Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the inherent adversarial robustness of subnetowrks inside a full network. It names such subnetworks as Robust Scratch Tickets (RSTs). RSTs have good properties, as they possess robustness even without adversarial training. The paper further studies the existence and properties of RSTs under various aspects. The overall findings are interesting and motivating regarding the architectural robustness.
main_review: Strengths:
+ The analysis of robust subnetworks without training is intriguing and underexplored
+ The proposed method for finding RSTs is well-motivated
+ The experiments are thorough and cover many aspects that are related to robustness
+ The writing is clear and easy to follow

Weaknesses:
- Although the experiments thoroughly evaluated the properties of RSTs, it still remains unclear what makes RSTs robust. For example, what architecture patterns do RSTs exhibit? Is there a pattern that correlates well with network robustness? The answers could further shed light on how to design robust networks from scratch.
- Following the topic on robustness of model architecture, the authors might want to discuss the relationship to recent works on searching robust models with neural architecture search [1, 2]. Especially as indicated in [2], the designed model has much less number of parameters compared to ResNet / MobileNet-V2, but exhibits higher robustness. It would be interesting to also extend experiments to analyze the connections/pattern similarities of RSTs and adversarially searched robust architectures.

Reference:
[1] Intriguing Properties of Adversarial Examples. 2017.
[2] When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks. CVPR 2020.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discovers the existence of robust subnetworks coined as Robust Search Tickets (RSTs) within randomly initialized networks that attain comparable or better performance than the adversarial trained networks. The paper conducts extensive experiments to study the existence of RSTs under different models, datasets and sparsity patterns. Additionally, the paper identifies the poor transferability between RSTs of varying sparsity ratios. To tackle this problem, it proposes a novel method called Random RST Switch (R2S) that randomly switches between different RSTs.
main_review: ## Strengths

* As network pruning is getting adopted in resource-constrained applications, the lottery ticket hypothesis is one approach that has been found effective in over-parameterized deep neural networks. Thus, the paper is well-motivated and showcases an interesting phenomenon of robust search tickets within randomly initialized networks.
* The results are intriguing, especially the comparison between RST and Robust Trained Tickets (RTTs) and the investigation of adversarial transferability between RSTs are interesting.
* The paper is clearly written, easy to follow and provides interesting insights into robust tickets.

***

## Weaknesses
* **Significance and novelty:** One of my primary concerns is the novelty of the work. Ye et al. [1] discuss the lottery ticket hypothesis in the adversarial setting (see Section 4.2). Cosentino and Zaiter et al. [2] analyzed Lottery Ticket Hypothesis with adversarial training. Li and Wang et al. [3] also demonstrate the winning tickets and boosting tickets exist for adversarial training. The paper should compare and highlight the differences with these works.
* **Comparison with existing works:** 
     * The paper highlights existing works that utilize pruning for adversarial robustness but do not provide an empirical comparison with those methods, which makes it challenging to evaluate the efficacy of the proposed method for robustness. I would suggest having the experimental comparison with Ye et al. [1], Madaan et al. [4] and Sehwag et al.[5] to confirm the fact that the proposed method works better than the existing techniques for robustness using pruning. 
    * Additionally, while the proposed method shows slight improvement, Madry et al. is the strongest considered baseline. I would recommend including more robust baselines following RobustBench [6] to highlight the effectiveness of the proposed method over the dense baselines.
* **Limited comparison:** The comparison with dense networks is unfair for Imagenet, in my opinion. For instance, in Table 1, the paper compares ResNet-50/ResNet-110 for RST while using ResNet-18 for the Dense Network. Instead, for a fair comparison, the paper should use the same base network. Furthermore, the paper should have used AutoAttack for all the evaluation results instead of using PGD-20, which is a significantly weaker attack in comparison.

***

## Other Questions and Comments
* RST on Auto Attack does not show improvement over the dense network in Table 1 in the Appendix. Comment.
* It is well known that adversarially trained models overfit to single $\ell_p$ norm. Since RST does not use adversarial training, can it generalize to multiple $\ell_p$ norms and attacks? I believe including these results can further strengthen the paper.
* In Figure 7, please include the transferability from a dense standard, dense adversarial trained model to RST and dense to dense for comparison.
* The proposed acronym RST also conflicts with Robust Self-Training [7] in the robustness literature. The acronym should be altered to avoid confusion in future work.
* The code and the pre-trained models are not provided with the paper or supplementary material. I would encourage the authors to release the code to promote reproducibility and successful comparison for future works.
* I would also suggest the authors report the variance for the Tabular results; it is also good practice for the community.

***
## References

[1] Zhang et al. Prune DNN using Alternating Direction Method of Multipliers (ADMM). ECCV 2020

[2] Cosentino and Zaiter et al. The Search for Sparse, Robust Neural Networks.

[3] Li and Wang et al. Towards Practical Lottery Ticket Hypothesis for Adversarial Training.

[4] Madaan et al. Adversarial Neural Pruning with Latent Vulnerability Suppression. ICML 2020

[5] Sehwag et al. HYDRA: Pruning Adversarially Robust Neural Networks. NeurIPS 2020

[6] Croce, Andriushchenko, Sehwag and Debenedetti et al. RobustBench: a standardized adversarial robustness benchmark.

[7] Carmon et al. Unlabeled Data Improves Adversarial Robustness. NeurIPS 2019
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes the existence of highly robust subnetworks within neural networks with random weights. Further, the paper proposes a simple way to search for these subnetworks, characterizes their properties w.r.t. initialization, model size, attacks, models, search method, and the sparsity structure, and, finally, proposes a technique for ensembling these subnetworks into a more robust model. The experimental framework appears to be sound, and the results are impressive, showing the consistent existence (and high performance) of these subnetworks across several factors.
main_review: —> Strengths:
- The paper’s main subject of study is interesting for a large portion of the DL community (and specially for the robustness community), and its main observation can spur further works that exploit the phenomenon of RSTs.

- The paper’s observation is strongly related to “What's Hidden in a Randomly Weighted Neural Network?”, Ramanujan et al., CVPR 2020 (reference [53] in the paper), and, in my opinion, is a clear contribution upon [53]’s observation: the phenomenon described in [53] also applies for robust networks by incorporating a robustness notion in the searching objective.

- The experimental framework appears to be sound, spanning a sensible amount of factors around the phenomenon being observed, which suggests the results are reliable. The results themselves are significant: the robustness of RSTs is comparable with that of adversarially trained networks. 

- Overall, the paper is well written, clear, and concise. The ideas and sections are well organized, and the results are presented in a convenient manner for the reader.

—> Weaknesses that affect my rating:
- My main concern is regarding the evaluation setup presented in L335 (it is still unclear to me). I understand the motivation behind “ensembling” several models from the same network, but I think this may lead to gradient obfuscation, which should then be studied on its own. To study whether the gains in robustness come from a phenomenon beyond gradient obfuscation, then probably the attacker would have to have access to the “expected” gradient, right? So that gradients are more informative and the adversary can be properly constructed. The first thing that comes to mind is literally the mathematical definition of the gradient under expectation, but perhaps the authors could think of a more reasonable approach that would fit the definition of “adaptive” attack.

- The paper proposes a search strategy for the weights to be pruned (Sec. 3.2). The proposed strategy is sensible (based upon previous works) and the results back the choice for this strategy. However, I think it would also be interesting to study how alternative strategies compare against this one. Have the authors considered alternatives?

- While it is interesting that RSTs exist under all the initializations that were considered (Sec. 4.4), I think a similarly interesting question is what initializations would not allow RSTs to exist. That is, if we initialized with, let’s say, purely normal distribution, would that impede RSTs from existing? I do not mention this experiment as a drawback in the paper, but as a phenomenon that would be worthy of study based on the paper’s findings.

- Section 6.1 reports that the diagonal of the matrix in Fig. 7 corresponds to the lowest values observed. My understanding is that given a single randomly initialized NN one can find *various* RSTs with a given weight remaining ratio. While the figure reports that attacks transfer poorly across NNs of *different* remaining ratios, I wonder how is the transferability across NNs of the same remaining ratio (not the same network topology perhaps, but the same ratio). 



—> Weaknesses that do not affect my rating but should be corrected/answered within reason:
- L68: random (or randomized?)
- L70: example
- L116: it is a bit weird to make reference to an equation that has not been introduced yet (or is not introduced immediately after).
- In Eq. (1), should not the “m” below the argmin have a hat?
- In L120 and L121, “permutation” or “perturbation”?
- In Eqn. (2), in practice, is there also a clipping for usual image pixel values, i.e. [0,1]?
- L137: sparse or sparsity?
- L145: ImageNet
- L161: decent
- Table 1 suggests a possible relation between depth (rather than parameter count) and the robustness of RSTs. Have the authors observed some phenomenon in this direction? I think there may be an interesting observation regarding depth to be made here.
- L182: “achieves” makes reference to RSTs? If so, it should be singular
- Sec. 4.6: another interesting “adversarial search method” would be simple Gaussian noise or uniform noise. It would be interesting to find whether any of these noises could be used as a reliable way of searching for RSTs.
- In Fig. 2(b) there is a regime of the “Remaining Ratio” dimension for which the fine tuning from inherited weights generates a drop in robust accuracy with respect to the original RST, i.e. the green curve is below the red curve. Is there an explanation for this phenomenon? Is this an implementation error?
- In Sec. 5.1 I assume fine tuning means “adversarial” fine tuning, and so perhaps this clarification could be useful for the reader.
- L237: achieves
- In L261, does this setup mean that the images are all added noise coming from a uniform distribution from [-eps,+eps]? Or {-eps,+eps}?
- Section 5.2, L271, studies the role of overparameterization. The conclusion, as reported in L280, is that overparameterization during adversarial training is necessary towards decent robustness of RSTs. I think I understand what the paper means, but is it not semantically incorrect to say that “training is necessary towards the robustness of Robust *Scratch* Tickets”? They are either trained or from scratch, but cannot be both, right? Or is there something I am missing?
- L275: I think there is a missing noun there, probably “accuracy”? Or “robustness” instead of “robust”?
- I think Sec 5.4 could be written in a more convenient manner: the whole section is currently a single sentence.
- L293: in the Appendix.
- L296: an.
- Fig. 7 legend: “accurracies”.
- L346: unwanted space.
- The experimental setup described in L358 is a bit unclear to me. My current understanding is: the networks is adversarial trained on CIFAR10, the last fully connected layer is randomly re-initialized, and then a search for robust networks is performed on this network. Is this correct? If so, I think the wording could improve as to better convey this setup.


limitations_and_societal_impact: Yes, they have.
review_title: null

Meta-review: This paper investigates whether subnetworks with high adversarial robustness can be found by learning a mask over the weights in a randomly initialized large network. Using this approach, the authors demonstrate that robust scratch tickets (RSTs) can be found for a number of datasets and show that multiple RSTs can be ensembled to further increase robustness. All reviewers found the work to be interesting, original, and of high quality, especially after a robust discussion with the authors, and I think this paper will be useful for the broader community. I would strongly encourage the authors to include all the experiments performed for the rebuttal in the final paper. I recommend acceptance. 
Decision: Accept (Poster)