Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: * This paper evaluates the adversarial robustness of 6466 non-isomorphic network designs from NAS-Bench-201. 
* This paper benchmarks robustness measurements based on Jacobian and Hessian matrices for their robustness predictability.
* Neural architecture search on robust accuracies is performed to find the relationship between architecture and robustness under the same model size. 
strength_and_weaknesses: ### Strength 

* This paper presents a large-scale and systematic study of the relationship between robustness and architecture. 
* The predictability of Jacobian and Hessian matrices is an important question. The results of this paper show their usefulness and limitation.

### Weaknesses

* The authors mentioned that "carefully crafting the topology of a network can have a substantial impact on its robustness, where networks with the same parameter count range in mean adversarial robust accuracy from 0.20% − 0.41%." I am not sure whether we can treat 0.2% - 0.41% adversarial accuracy as significant. In addition, what if we use a substantially different architect (but the same number of parameters) like Transformer?  
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: The conclusions reached by this paper are not well supported by quantitative results. Please correct me if I missed anything.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper evaluates NASBench-201 (NAS benchmark) on four adversarial attacks. First, they show that the architecture influences the robustness of the model, then they evaluate all of the NAS search space for various adversarial models. They show how this database can be used for various purposes, including training-free estimates of robustness and robustness-aware neural architecture search. 

strength_and_weaknesses: -= S1 =- The paper extends the NAS benchmark by adding the important dimension of robustness to it. This empirical study is well-motivated, the experiments are presented in a nice and descriptive manner, and this dataset can help other researchers and designers to design more robust models. 

-= S2 =- The paper is very well-written and provide a nice background on various adversarial attacks that can be leveled against a model. Additionally, the results are presented in an easy to understand manner. 

-= S3 =- The three use cases presented for this dataset are also well-motivated and evaluated. 

-= S4 =- Toward the end of the paper, the authors provide a set of empirical analyses on what aspects of the network are highly influential on robustness. These observations can help future researchers and designers of network. 

-= W1 =- The paper can benefit from providing more descriptive captions to the figures. It will be nice if the figures can be read in a standalone manner i.e., they explain what the results are. 

-= W2 =- The paper can benefit from discussing the results a little bit more. For instance, it is not clear what we observe in Figure 5 leads to high diversity of sensitivity to different kinds of corruptions. 

-= W3 =- The paper can benefit from explaining the experimental setup more explicitly, how many times do they repeat their experiments? What machines do they use? What are the hyperparmeters? etc. 
clarity,_quality,_novelty_and_reproducibility: Clearly written, novelty is medium high, and artifacts are provided for reproducibility. 
summary_of_the_review: Well-motivated problem and a nice empirical analyses of an existing benchmark for a new set of metrics. This can be useful in furthering our understanding of network architecture design and robustness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work extends the NAS-Bench-201 dataset to evaluate the neural architecture design and robustness. Moreover, an analysis of how the architectural design affects robustness is conducted.
strength_and_weaknesses: Strengths:
1. The tackled problem is relevant to the ICLR community.
2. Several experiments have been conducted, and comprehensive results are provided. 

Weaknesses:
1. From the technical perspective, no new concepts have been introduced.
2. It would be useful to describe the main methodology through algorithms and schemes.
clarity,_quality,_novelty_and_reproducibility: Clarity: 7/10

Quality: 6/10

Novelty: 5/10

Reproducibility: 8/10
summary_of_the_review: A borderline paper with few technical novel contents but solid evaluation and results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper enriched the NAS-Bench-201 dataset by including the adversarial robustness of 6466 non-isomorphic network designs. Though this is not the first time to show the connection between architecture and adversarial robustness, the dataset provided in this paper can be helpful for the following research in this direction.
justification_for_why_not_higher_score: The authors directly position their work over the existing dataset NAS-Bench-201 and further provide the adversarial robustness results through some adversarial attacks. The overall work could be a bit incremental, but all reviews agree on the usefulness of this augmented dataset in the field. 
justification_for_why_not_lower_score: Adversarial robustness is an important dimension that should be included in the existing NAS dataset. The authors also included interesting use cases to explore and exploit the proposed datasets, which may inspire the following further studies on the relevant datasets.
Decision: Accept: poster