Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper utilizes neural architecture search to discover neural network architectures that are adversarially robust and with low latency at the same time. In particular, they introduce an adversarially robust teacher model for cross-layer distillation in the training and search process to improve the adversarial robustness of the model, and specifically search which layers of the teacher model are used for feature distillation.
strength_and_weaknesses: ==== Strength ====
- The first work that applies knowledge distillation-based NAS for adversarial robustness.
- Intuitively, the proposed method is reasonable.
- The paper writing is clear, the method description is easy to understand.

==== Weakness and Questions ====

1. Unsupported statements and missing discussions.

a) This paper claims in the introduction that RNAS-CL is ``the first NAS method that jointly optimizes accuracy, latency, and robustness against adversarial attacks''. However, quite a few studies have already explored this problem, for example, [1][2][3]. Should remove this statement, and discuss / compare with them.

b) The proposed method adopt ideas from existing methods, should discuss more about these existing work, for example, [4].

2. Weakness and questions about the experiments.

a) Missing comparisons with robustness NAS studies like [1][2][3].

b) Necessary baseline results are missing. Table 1 shows the performance of the architectures discovered by different methods on CIFAR-10. The RNAS-CL-* models are searched with the Wide-ResNet teacher model, but the performance of the teacher model is not mentioned. The performance of Wide-ResNet should be given as baseline results. In particular, I wonder if the searched architecture outperforms the Wide-ResNet model, which is very important to verify the effectiveness of the proposed method;

c) Why are only the results of using WRT as the teacher model on CIFAR-10 reported in Table 1? For ImageNet-100, the results with different models as teacher models are provided in the appendix. How will the teacher model's architecture or performance influence the performance of discovered architecture.

d) Some essential experiment settings are missing. For example, what’s the step number for PGD attack in the paper?

[1] Xie, Guoyang, et al. "Tiny adversarial mulit-objective oneshot neural architecture search." arXiv preprint arXiv:2103.00363 (2021).

[2] Ning, Xuefei, et al. "Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach." arXiv preprint arXiv:2012.11835 (2020).

[3] Yue, Zhixiong, et al. "Effective, efficient and robust neural architecture search." 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.

[4] Li, Changlin, et al. "Block-wisely supervised neural architecture search with knowledge distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Another minor suggestion: You should use ``'' for quotation marks in LaTex.
clarity,_quality,_novelty_and_reproducibility: Clarity is good. The novelty is on the borderline. No code, so reproducibility is doubtful.
summary_of_the_review: My concerns about this paper mainly come from the insufficient experimental verification. I may consider improving the score if the authors could provide the missing experiments and results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a NAS method to find robust neural network that can defend adversarial attracks. It combine knowledge distillation with NAS to pursue better performacne. The authors conduct experiments on multiple tiny datasets, such as CIFAR-10, ImageNet-100.
strength_and_weaknesses: Strength:

-It is interesing to find robust architecture with automatic search. 

-The proposed method use a teacher to enhance the model's performance and robustness, which is reasonable. 

-This paper is well-written and easy to follow.

Weakness:

-Directly combining NAS and knowledge distillation can usually improve performance. The authors should compare the proposed method with other knowledge distillation methods.

- Only tiny datasets are used to evaluate the effectiveness of the proposed method. These datasets have a large gap with the practical scense. Comparesion on large dataset (such as ImageNet) is required to validate the method's effectiveness.
clarity,_quality,_novelty_and_reproducibility: See Strength And Weaknesses
summary_of_the_review: See Strength And Weaknesses
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to obtain a robust and efficient neural architecture by searching the best teacher layers and the number of filters for student network. Experiments are conducted on CIFAR and ImageNet-100, which show good performance over prior methods.
strength_and_weaknesses: Strength:
1. It make senses to automatically search for the best teacher layer to supervise each student layer, and ablation studies justify the effectiveness of this strategy.
2. The paper is well written and easy to understand.

Weaknesses:
1. The knowledge distillation method is quite standard except for the optimal teacher layer searching. For example, there are not much new things in Section 3.1 and distillation loss in Section 3.4.
2. The architecture search method seems to have limited novelty over FBNetv2.
3. The proposed method use knowledge distillation to improve the performance, while the compared methods not. It is not quite fair and no ablation studies to show the performance of the proposed method without knowledge distillation.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to understand. The novelty of the proposed method seems to be limited.
summary_of_the_review: My main concerns lie on the limited novelty of the proposed method and the unfair comparisons with prior work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is proposing a neural architecture search which can satisfy both adversarial robustness and low latency as well as high accuracy. It builds on knowledge distillation methods and also choose which layer to distill. The paper is reviewed by three experts and all agreed on merits and limitations of the paper. The paper is interesting and performs very well. However, it had some issues in terms of related work. It missed some important work which also addressed adversarial robustness and NAS. Moreover, after these works are considered reviewers raised novelty issues. After my personal reading of the paper, I agree with the merits and the weakness of missing related work. However, I do not agree with the novelty issue. I believe the related work issue is still largely unsolved. Authors did provide more results comparing with these method and the results are promising. However, the paper is not revised accordingly. These methods are not mentioned and discussed in detail. Introduction still reads as if these methods do not exist. Hence, the paper needs a major revision in terms of introduction and related work discussing these methods pointed by the reviewers.
justification_for_why_not_higher_score: Authors simply missed a significant set of works which are highly relevant in their submission. Authors mistakenly claimed to be the first to address robustness and NAS. Although authors fixed the claim and added lots of new experiments, the writing still misses these additional references. Both introduction and related work need a major revision addressing these work.
Decision: Reject