RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
Authors: Utkarsh Nath
Yancheng Wang
Yingzhen Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Robust Neural Architecture Search
Knowledge Distillation
Efficient Deep Learning Model

Abstract: Deep Neural Networks are vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the driving tools of deep neural networks, demonstrates superior performance in prediction accuracy in various machine learning applications. However, it is unclear how it performs against adversarial attacks. Given the presence of a robust teacher, it would be interesting to investigate if NAS would produce robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental result evidences the effectiveness of RNAS-CL and shows that RNAS-CL produces small and robust neural architecture.

Under review as a conference paper at ICLR 2023
RNAS-CL: ROBUST NEURAL ARCHITECTURE SEARCH BY CROSS-LAYER KNOWLEDGE DISTIL- LATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Neural Networks are vulnerable to adversarial attacks. Neural Architec- Search (NAS), of the driving tools of deep neural networks, demon- strates superior performance in prediction accuracy various machine learn- ing applications. However, it is unclear how it performs against adversarial at- tacks. Given the presence of a robust teacher, it would be interesting to inves- tigate if NAS would produce robust neural architecture by inheriting robustmess from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL). a novel NAS algorithm that improves the robustness of NAS by learning from robust teacher through cross- layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student/teacher output only in the last layer, RNAS-CL automat- ically searches for the best teacher layer to supervise each student layer. Experi- mental result evidences the effectiveness of RNAS-CE and shows that RNAS- A-SCL produces small and robust neural architecture. The code of RNAS CL available at https: /nnoymuuss 4open. sci www.ennceeei R 06A0/.
INTRODUCTION
Neural Architecture Search (NAS), one of the most promising driving tools with state-of-the-ar performance of deep neural networks in various tasks such as computer vision and natural language processing. has been attracting lot of attention in recent years. NAS automatically searches for neural architecture according to user-specified criteria without human intervention, thus avoiding the time-consuming and burdensome manual design of neural architecture. Earlier studies in NAS are based on Evolutionary Algorithms (EA) (Real et al., 2017) and Reinforcement Learning (RL) (Zoph Le, 2017: Tan et al., 2019). However, despite their performance, they are computationally expensive. It would take them more than 3000 GPU days to achieve state-of-the-ar performance on the ImageNet dataset. Most recent studies (Liu et al., 2019; Cai et al., 2019; Wu et al., 2019; Wan et al., 2020; Nath et al., 2020) encode architectures the weights using gradient descent. Architectures found by NAS exhibit two significant advantages. First, they achieve SOTA performance for various computer vision tasks. Second, the architectures found by NAS are efficient in terms of speed and size. Both advantages make NAS incredibly useful for real-world applications. However, most NAS methods are designed optimize accuracy, parameters, or FLOPs. It is not clear how these architectures perform against adversarial attacks. In this paper, we propose RNAS-CL, a NAS method that jointly optimizes accuracy, latency, and robustness against adversarial attacks without robust training.
weight-sharing super-net and optimize
Adversarial attacks are performed by adding adversarial samples, for example, adding small sophis- ticated perturbations to the clean image, such that the model misclassifies the image. It is widely accepted that deep learning models are susceptible to adversarial attacks (Szegedy et al., 2014). Therefore, it is critical to analyze the robustness of models against adversarial attacks. Adversarial robust models are crucial for security security- sensitive applications such as self-driving cars, health care, and surveillance cameras. For example, self-driving might not recognize signboard after attaching a patch; a surveillance system, an unauthorized person might get access by fooling the DNN model.

---

Under review as a conference paper at ICLR 2023
Hydra
54
tEEN 1220e0
Robrisx
Adverserie Accuracy
IJM, 32341
Amocba Ver
50
EM, 3R3M
Nasfiet
5.24 cocee.
46
aff 25a PAA
2:44 socee
D1-122-4
RACL
10.214 1100
NVV2 ARD
18 -3
+8,236000
RMAS-CL (Our)
13.% 300P1
Clean Accuracy
Figure 1: The figure compares various SOTA efficient and robust methods CIFAR-10. Clean Ac- curacy represents top-1 accuracy on clean images. Adversarial Accuracy represents top-1 accuracy on images perturbed by PGD attack. A larger marker size indicates larger architecture. The numbers in brackets represent the number of parameters and MACs, respectively.
Adversarial training (Goodfellow et al., 2015; Madry al., 2018; Kannan al., 2018; Tramèr al., 2018: Zhang et al., 2019a) is the most standard defense mechanism against adversarial attacks. Here, the models are trained on adversarial examples, which are often generated by fast gradient sign method (FGSM) (Goodfellow et al., 2015) or projected gradient descent (PGD) (Madry et al., 2018). Other types defense mechanisms include models trained by losses or regularizations (Cissé et al., 2017; Hein & Andriushchenko, 2017: Yan et al., 2018 Pang et al., 2020), transforming inputs before feeding to model (Dziugaite et al., 2016; Guo et al., 2018; Xie et al., 2019), and using model ensemble (Kurakin et al., 2018; Liu et al., 2018).
Orthogonal to these methods, recent research (Madry et al., 2018; Guo et al., 2020; Su et al., 2018: Xie & Yuille, 2020; Huang et al., 2021) found intrinsic influence of network architecture adversarial robustness. Inspired by this idea, we propose Robust Knowledge Distillation for Neural Architecture Search (RNAS-CL), to the best of our knowledge, the first NAS method that uses knowledge distilled from a robust teacher model to find a robust architecture. Knowledge distillation transfers knowledge from complex teacher model to small student model. In standard knowledge distillation (Hinton et al., 2015), outputs from the teacher model are used as "soft labels" to train the student model. However, apart from the final teacher outputs, intermediate layers contain rich attention information. Different intermediate layers attend to different parts of the input object (Zagorykko& & Komodakis, 2017).
Hence, we ask the question: can a robust teacher improve the robustness of the student model by providing information about where to look, i.e., where to pay attention? The proposed RNAS-CL gives affirmative answers to the above question. In RNAS-CL, apart from learning from the output of the robust teacher model, each layer in the student learns "where to look" from the layers in the teacher model. However, the teacher and student might have a different number of layers. This leads us to another question regarding how to map a student layer to its corresponding teacher layer that it should learn from. In RNAS-CL apart from searching the architecture of the student model, search for the perfect tutor (teacher) layer for each student layer.
Let consider teacher (T) and student (S) model with ne and n, layers, respectively. T; S, are the i-th teacher and student layer, respectively. In RNAS-CL, each student layer S; is associated with n, gumbel weights, and each gumbel weight corresponds each teacher layer. Intuitively, each gumbel weight indicates the weight of the connection between the student layer and each teacher layer. In the search phase, besides optimizing the architectural weights, we optimize these gumbel weights to find the perfect teacher layer. We hope the teacher to teach "where to pay attention." Therefore, by virtue of our RNAS-CE loss function for each student-teacher layer pair, each student layers learns robustness from properly and automatically chosen teacher layer by maximizing the similarity of its attention map to that of its teacher layer.
1.1 CONTRIBUTIONS
Below the main contributions of this work.
1. Adversarial robust NAS. RNAS-CL optimizes neural architecture to achieve a good tradeoff between robustness and prediction accuracy in differentiable manner. To the best of our knowl-
2

---

Under review as a conference paper at ICLR 2023
Ourgue Probabiitics (9)
Teacher
Cono
Gama
Teacher
KLip-g)
Altention Maps Sai
(1
(10
Jow. for the i. studein layer
Iopur Image
Stadeni
Toe -y logg
Maps
Studeni
Goms
Cona
Gomu
Ouiper Probohiiltics (P)
(iii) (b) Sample Attention Maps
(iv)
RNAS-CL Training Paradigm
Figure 2: (a) Training paradigm based on RNAS-CL. We connect attention maps from each student layer to each robust teacher layer. For each student layer, we search for the optimum teacher layer. Sky represents gumbel weights associated between jth student layer and jth teacher layer. RNAS-CL induces robustness to the student model by searching for the optimum teacher layer. We also search for the number of filters each layer to build an efficient model inspired by FBNetV2 (Wan al., 2020). (b) Sample attention maps corresponding to input Image (i) from low-level (ii), mid-level (iii), and high-level (iv) convolution layers.
edge, RNAS- WNAS-CLIE is the first NAS method that optimizes robustness and prediction accuracy with- out robust training. Leveraging the penalty on model size/inference cost, neural architecture found by RNAS-CL is compact compared to competing NAS methods. We compare RNAS-CL. with other computationally efficient and robust models (Schwag et al., 2020; Ye et al., 2019; Gui et al., 2019; Goldblum et al., 2020; Dong et al., 2020; Huang et al., 2021). Compared to these models, similar sized RNAS-CL models achieve up to ~ 10% higher clean accuracy and up to ~ 5% higher PGD accuracy on CIFAR-10 dataset.
2. Cross-Layer Knowledge Distillation. Our work advances the research of Knowledge Distilla- tion (KD) using NAS. In particular, while conventional KD only uses fixed connections between teacher and student models to guide the student model, RNAS-CL extends the teaching scheme to learnable connections between layers of the teacher and the student models.
2 RELATED WORK
2.1 KNOWLEDGE DISTILLATION
Knowledge Distillation (KD) transfers knowledge from a large, cumbersome model to small model. (Hinton al., 2015) proposes the teacher-student model, where they use the soft targets from the teacher to train the student model. KD forces the student to generalize, similar to the teacher model. Since (Hinton al., 2015), numerous KD variants (Romero et al., 2015; Yim et al., 2017: Zagoruyko & Komodakis, 2017; Li et al., 2019; Tian et al., 2020a; Sun et al., 2019) based on feature map. attention map, contrastive learning have been proposed. (Romero al., 2015) intro- duced intermediate-
hints from the teacher model to guide the student model training. (Romero al., 2015) trained the student model in two stages. First, they trained the student model such that the student's middle layer predicts the output of the teacher's middle layer (hint layer). Next, they fine-tuned the pre-trained student model using the standard KD optimization function. Thanks the intermediate hint, the student model achieved better performance with fewer parameters. Moving a step further. (Yim et al., 2017). (Zagoruyko & Komodakis, 2017) and (Li al., 2019) used informa- tion from multiple teacher layers to guide students' training. (Yim et al.. 2017) computed Gramian matrix between the first and the last layer's output features to represent the flow of problem-solving. (Yim et al., 2017) transferred knowledge by minimizing the distance between student and teacher's s flow matrix. (Li et al., 2019) calculated the inter-layered Gramian matrix and inter-class Gramian matrix to find the most representative layer and then minimized the distance between few of the most representative student and teacher layers. (Zagoruyko & Komodakis, 2017) minimized the distance between teacher and student attention maps at the various block. (Li et al., 2020) distills knowledge from teachers' blocks to supervise students' block-wise architecture search. Contrary to the above methods, which map few teacher-student layers or blocks. We map all student layers to
3

---

Under review as a conference paper at ICLR 2023
teacher layer. We propose RNAS-CL to search for the perfect tutor layer for each student layer. Sim- ilar to (Zagoruyko & Komodakis, 2017), we minimize the distance between mapped student-teaches attention maps.
2.2 NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) is a technique that automatically designs neural architecture with- out human intervention. Given search space, we can find the best architecture by training all archi- tectures from scratch to convergence; however, this is computationally impractical. Earlier studies in NAS were based RL (Zoph & Le, 2017; Tan al., 2019) and EA (Real et al., 2017); how- ever, they required lots of computation resources. Most recent studies (Liu et al., 2019; Cai et al., weight-sharing super-network. Specifically, they
2019; Wu et al., 2019) encoded architectures trained an over-paraninn werr-parameterized network containing all candidate paths. During training, they intro- duced weights corresponding to each path. These weights were optimized using gradient descent to select a single network in the end. The selected network is then trained in standard fashion. Although these methods achieved SOTA performance on various classification tasks, their robust- ness against adversarial attacks is unknown. (Devaguptapu et al., 2021; Guo et al., 2020; Li et al., 2021; Madry et al., 2018; Su et al., 2018: Xie & Yuille, 2020; Huang et al., 2021) 1) found an intrinsic influence of network architecture handcrafted architectures are more robust against adversarial attacks as compared to NAS models. Furthermore, they empirically observed that increase in model size increased the robustness of the model against adversarial attacks. (Guo et al., 2020) discovered that densely connected architectures are more robust to adversarial attacks. Thus they proposed a NAS method that conducts adversar- ial training on super-net and then selects the architecture with dense connections. (Li et al., 2021) dilated the backbone network to preserve its standard accuracy and then optimized the architecture and parameters using the adversarial training. Despite SOTA performance, a major drawback lies in the fact that adversarial training is highly time-consuming and decreases the performance on stan- dard (clean) images. This paper proposes NAS method that optimizes robustness and prediction accuracy without adversarial training.
adversarial robustness. (Devaguptapu et al., 2021) observed
2.3 EFFICIENT AND ROBUST MODELS
Research community has extensively researched building efficient and adversarially robust models individually. However, few works combine both domains, building an efficient and adversarially robust model. (Sehwag et al., 2020) propose to make the pruning technique aware of the robust training objective. They formulate pruning integrate with a robust training objective. (Huang et al., 2021) investigated the impact of network width and depth configurations on the robustness of adversarial trained DNNs. They observed that reducing capacity at last blocks improves adversarial robustness. (Goldblum et al., 2020), propose Adversarially Robust Distillation (ARD), where they encourage student networks mimic their teacher's is output within an e-ball of training samples. Furthermore, there are few NAS methods (Yue et al., 2022; Ning et al., 2020; Xie et al., 2021) that jointly optimises accuracy, latency and robustness. Compared to these methods, similar-sized RNAS-CL models achieve both higher clean and robust accuracy.
empirical risk minimization (ERM) problem and
ROBUST KNOWLEDGE DISTILLATION FOR NEURAL ARCHITECTURE SEARCH
We use knowledge distilled from a robust teacher model search for robust and efficient archi- tecture. Knowledge distillation is the transfer of knowledge from a large teacher model to a small student model. In standard knowledge distillation, outputs from the teacher model are used as "soft labels" to train the student model. However, apart from the final teacher outputs, intermediate fea- tures constitute important attention information. Different intermediate layers "attend" to different parts of the input object. In RNAS-CL, apart from learning from the teacher's soft labels, the method learns from intermediate teacher layers where to pay attention, i.e., each student layer is mapped to a robust teacher layer to learn where to look. In Section 3.1, we discuss how we define attention maps. We hypothesize that learning where to pay attention from a robust teacher will inherently make the student model more robust to adversarial attacks. Now, the teacher and student could have a dif- ferent number of layers, which leads us to the question, how to map a student and a teacher layer? In our method, we search for the perfect tutor for each layer. Furthermore, along with increasing
4

---

Under review as a conference paper at ICLR 2023
the robustness,
also interested in searching for an efficient architecture. In Section 3.2 and 3.3, we discuss our tutor and architecture search algorithm. Similar to other state-of-the-art NAS methods (Liu et al., 2019; Wu et al., 2019; Wan al., 2020), RNAS-CL consists of the searching and training phase. In the search phase, we optimize the architectural weights. In the training phase, train the architecture sampled from the search phase in standard fashion. In Section 3.4, discuss our searching and training optimization objectives.
3.1 ATTENTION MAP
We are interested in learning where to pay attention from a robust teacher model. Let us consider a convolution layer with activation tensor A RC where C the number of channels, and И and W are spatial dimensions. We define a mapping function F RCCHXXW - R that takes A as input and outputs attention map F(A) E R#*W by [F(A)] ko = - -c=1 Ac,h, represents the element of A with channel coordinate and spatial coordinates h and w.
12=1A2 Ach. where
We use activation- based mapping function FF proposed (Zagoruyko & Komodakis, 2017). The mapping function F is applied to activation tensors after each convolution layer to generate an attention map. We visualized few attention maps in Figure 2(b). RNAS-CI intends to find teacher layer, referred to as tutor, for each student layer such that the student layer's attention map is similar to that of its tutor in the teacher model. The student attention map may differ in dimension compared to that of its tutor. To address this issue, interpolate all attention maps common dimension.
3.2
TUTOR SEARCH
As described above, we aim to find a tutor (teacher layer) for each student layer, which teaches where pay attention. However, each student layer can choose any tutor, resulting in an exponen- tially large search space. For example, the search space for student model with 20 layers and a teacher model with 50 layers is of size 5020 In order to address the computational issue, we employ Gumbel-Softmax (Jang et al., 2017) to search for the tutor for each student layer in a differentiable manner. Given network parameter t [V......
. Un] and a constant т. The Gumbel-Softman exp((e;+ta)/+) and 6 ~ N(, 1) is the uniform tion is defined as g(v) = [9i..... 9n. where 9i = >; exp((ei+e.//rl random noise, which is also referred to Gumbel noise. When - 0, Gumbel-Softmal tends to the arg max function. Gumbel-Softmn is a "re-pame re-parametrization trick", that can be regarded as a differentiable approximation the argmax function.
func-
Now consider a teacher T and student S model with n, and n, number of layers, respectively. Aj and A the ht activation tensors of teacher and student layers. In RNAS- each student layer (i) is associated with n, Gumbel weights (gt) such that ge E R xn,, Let 9ij be the Gumbel weight associated with tth student and jth teacher layer. Then the attention loss is defined as
The The
F(A!)
F(A() ||F(A)
L (A,, As)
Σ> 9ú
Ts X Th
||F(A:)]2
112
i=0
where A, and A, are activation tensors for all student and teacher convolution layers. FF is the mapping function as defined in Section 3.1. || ||2 the e-norm. We exponentially decay the temperature of Gumbel-Softma while searching, leading an encoding close to one-hot vector.
3.3 ARCHITECTURE SEARCH
Apart from searching the tutor for each layer, are interested in building efficient architecture with low latency. Inspired by FBNetV2 (Wan et al., 2020), we search for the optimal number of filters, or the number of output channels, for each convolution block. Let A f2, f2, fn) be the be their corresponding outputs for a convolution block. Then
choices filters and {21, 22.,, 2n
the cumulative output is defined as Z Σ Dm1 2i, where g:u is the Gumbel weight corresponding to ensure low latency. The FLOPs
to tth filter choice. The number of FLOPs is optimized proportional to the number of filters, and the cumulative number of filters is a function of Gumbel weights. As result, the FL.Pss can be optimized in differential manner using SGD. Similar to
5

---

Under review as a conference paper at ICLR 2023
tutor search, temperature is exponentially decayed to obtain encoding which is close to one-hot vector. Figure 12 in the appendix illustrates the architecture search process by FBNetV2.
3.4 RNAS-CL Loss
Following the convention of tfate-oftthe-e NAS methods (Liu et al., 2019; Wu et al., 2019; Wan et al., 2020), RNAS-CL has searching and training phases. In the search phase, Gumbel weights and other model parameters are updated at each epoch of SGD, where the Gumbel weights correspond to the intermediate optimised using our RNAS-CL search loss defined by (2).
connection (3.1) and the filter choices (3.3). The weights are
RNAS-CL search loss. Let y be the ground-truth one-hot encoded vector, and be output proba- bilities of the student and teacher network and Ag, A₁ be activation tensors for all student and teacher convolution layers. Then the RNAS-CL search loss is defined as
L(y. 9, A., A.) = (-y logp= [[( 9) + %/ Lam (,),))) A,))nf:
(2)
where KL 9) Σ, Pi log P. is the ity measures. Laun is the attention loss as defined in (1) and 7s is normalization constant. nf represents latency, which is optimized a differential manner following (Wan et al., 2020).
Leiblerr divergence between two probabil-
9,
After the search phase, tutor is selected the j" teacher layer with j arg max, 9ij for each student layer i. addition, the filter choices described in Section 3.3 for neural architecture are decided the one corresponding to the maximum Gumbel weight for each convolution block. We then start the training phase, where the searched architecture trained using the RNAS-CL train loss defined below.
RNAS-CL train loss. Let y be the ground ground-truth one-hot encoded vector, p and be output probabil- ities of the student and teacher network, and A,, A, be activation tensors for all student and teacher convolution layers. Then the RNAS-CL train loss is
L(y.P,q, P, A,, A,) = LCE(V,)) p) + KL(p, 4) + AAtLAAn A.),
where LCE(Y, p) y log p is the cross-entropy, KL(p,)) q) is the KL-divergence, >t is normal- ization constant. Note that, gi in Luwn is one-hot vector. Thus, each student attention map optimized w.r.t. to single tutor layer.
4 EXPERIMENTS
In this section, we conduct experiments on real-world datasets to show the effectiveness of the proposed framework. The experiments section is organized as follows. In Section 4.1, we discuss our experimental setup and implementation details. In Section 4.2, we compare models trained by RNAS-CL against state-of-the-ant the efficient and robust models CIFAR-10. In Section 4.3, we empirically show the effectiveness of cross-connections in improving the adversarial robustness of the model. We further discuss the abbustnessinnducing capacity of teacher layers and compare RNAS-CL models trained on ImageNet-100 in the appendix.
4.1 IMPLEMENTATION DETAILS
In this paper, we evaluate RNAS-CL on two public benchmarks for image classification. (1) CIFAR- 10 collection of 60k images in 10 classes (Krizhevsky, 2009). (2) ImageNet-100= subset of ImageNet- I dataset (Russakovsky et al., 2015) with 100 classes and about 130k: images (Tian et al., 2020c). We use standard data augmentation techniques for each dataset, such as random-resize crop- ping and random flipping. We train different architectures found by RNAS-CL CL on both CIFAR- 10 and ImageNet- 100. On each dataset, we first perform the searching step. We train our model using RNAS-CI. search loss (2). We search for the channel number and the connected teacher layer at each student layer. We conduct experiments with different search spaces and various robust teacher models. In this section, we refer to model by RNAS-CL-X-T where X represents our search space, and T represents the robust teacher model. Detailed search space is provided in Table 7 and Table 8. We use robust teacher model, ResNet-50, ResNet-18, WedeResNet-50. and Nete 34, which are referred to as R-50, R-18, WRT-50, and WRT-34. For example, RNAS-CL-S: -L- represents a model trained in the S3 search space using adversarially robust ResNet-18 model.

---

Under review as a conference paper at ICLR 2023
Method
Clean Ace Without Adversarial
FSGM
PGD
MI-GGSM
* Parims (M)
MACS (M)
Training
DARTS (Liu al., 2019)
97.03
42.48
7.09
0.28
3.3
500"
PC DARTS (Xu et al., 2020)
97.05
49.18
9.84
1,21
3.5
600
RACL (Dong
2020)
97. 97.44
50.53
1,93
4.68
3.6
500
AmaochaNet (Real
1019
07
44.79
500
"NaNNa (Zaph 2018)
07
4733
2020)
76.13
"EIRNAS-C6 YYu al., 2022
43.97
6.76
0.34
DNAS-CL-Si-WREE (Our)
89.4
44.95
34.3
6.64
RNAS-CI -8S. WRT (Our)
90.4
46.72
35.50
40.57
11.02
RNNS C. Our)
60:82
45. 93
37.24
42.27
15:38
CLM RNAS-CL-L-WE WRT-34 (Our)
92:36
50.51
39'81
11:54
"226"
92.6
52.37
41,9
46.66
1210
With Adverssrial Training
Hydra ResNet 18 (Schwag
20200
69
41.6
37.63
"Hydrä ResNet (Schhwa at. 20120)
71%
775:43""
'Hydiä ReaNet 30 (Sclwag al. 20200)
73.9
45.3
85.92
ADV.ADMM Net 18 (Ye al., 2019)
ADDVADMM ResNet 14 (Ye al 2019)
68.8
75:45
"ADV:AD Res 30 (Yo or 2019)
63.
0..25
35592
"RolNeESmalll (Cro0 2t at., 20200
78.05
53.93
33:41
MostictMMsdium (Giso
20200
78
RobNet. arrg (Gio et 20000
78.57
$4.98
49.92
6.80
"AmochaNet (Real en 20019)
83:41
56:40
33
47.60
3.2
"NaaNet (2oph of 2018)
83.66
53.67
55300
"DARTS (Liu at... 2019)
83:75
55.77 75
44.91
$1.63
500
PC-DARTS
52.67
(EXCC et al." 20130)
WY399
57 17444"
"49.34
"54.7
3021
79.85
557:55
5.83
DNC.2TR (Mlinn et at. 20111)
87 27.22
67.12
52.52
DARTS-R (Htunng al.,
MVV2-ARD (Gelbblum ot al. 2020)
46.28
MSRRaKNer CerriiKeeT-i000 T000 [Ning
3000)
3.16
MKSKabNNe:2000 (Ning et al.
60%
"Sarrss ONie ot al." (02T)
76.34
"3TR3"
T.88%
LNAS-CU-S--WRE (Our)
83:45
30.67
43.07
43.98
0.11
6.64
RNAS-CI
(Our)
51.99
RNKS WRT (Our)
40.11
43.24
25.33
00.3
15:58
RNAS WRT 70Wr7
67.20
55:.71
511.6
53.13
3
3228
RNAS-CL-L-WRT3S (Ow)
86.28
61'2"
53.89
55.07
1210
Table I:
The table shows performance of various efficient and robust methods on CIFAR- 10 dataset. Clean Ace represents top-1 accuracy on clean images. FSGM, PGD MI-FGSM represents top-1 accuracy images perturbed by the corresponding attacks. PGD20 represents 20 step PGD attack. represents approximate values. Columns with unreported values are represented by
For both datasets, we use SGD optimizer. For ImageNet-100 100, default values of momentum and weight decay set to 0.9 and 4e 5, respectively. The batch size is set 256. The learning rate initialized as 0.05 and annealed down to zero following a cosine schedule. After the search stage which takes 100 epochs, the searched architecture is trained from scratch using RNAS-CL train loss (3) for 200 epochs. For CIFAR-10, default values of momentum and weight decay are set to 0.9 and 2c - 4, respectively. The batch size is set to 128. We train our model for 100 epochs in both the searching and training phases. The learning rate is initialized as 0.1, and reduced by factor of 10 after the thth and the 9hth epoch. Following the settings of FBNetV2, the temperature (r) in Gumbel- Softmax is initialized as 5.0 and exponentially annealed by e 00005 every epoch in the search phase. The pppee-parameter X, and >, in (2, 3) is selected from candidate set (0.01, 0.1, Both A, and X; are set to 1.0 for all experiments. In the search phase for each batch, we use 80% of the data to optimize the model weights and the remaining 20% data to optimize architectural weights choose five powerful attacks including
0.1, 0.1, 1.0, 10, 100).
which are Gumbel weights For robustness evaluation, FGSM (Goodfellow et al., 2015), MI-FGSM (Dong et al., 2018), PGD (Madry et al., 2018), CW (Carlini & Wagner, 2017) and AutoAttack (Croce & Hein, 2020). Results for CW and AutoAttack are provided in the appendix A.4. Consistent with adversarial literature (Madry et al., 2018; Zhang et al., 201 9b), the adversarial perturbation is considered under Lo0 norm with a total perturbation scale of 8/255 (0.031).
4.2 COMPARE EFFICIENT AND ROBUST CIFAR-10 10 MODELS
In this section, we compare the robustness of our method against other SOTA efficient and robust models. In Table 1, we compare RNAS-CI to both efficient models trained with and without ad- versarial training. All RNAS-CL models are trained with robust WideResNet-34 (Rice et al., 2020) as the teacher model. RNAS-CL. significantly outperforms all models trained without adversarial
7

---

Under review as a conference paper at ICLR 2023
training in terms of adversarial accuracy. While being significantly smaller, our models achieve sig- nificantly higher adversarial accuracy when compared to models trained without adversarial train- achieves more than 28% higher PGD accuracy compared SRAS-CL-S7--W-
ing. For example, ENNS-CL-S7-WRZ-
to most of the other methods. Compared to MVVV2-ARI,
achieves ~ 1% lower PGD accuracy; however, it exceeds MVVV2-ARD by 14.5% in clean accuracy while being 10x smaller. A similar-sized model, for example, KNAS-CL-M-WRT- exceeds both clean and PGD accuracy by 16.5% and 1.43%.
Next, we compare RNAS-CL against adversarially trained robust models. For a fair comparison, after the training stage, we train our RNAS-CL models with the TRADES optimization objective for 20 epochs. For retraining, the cross-entropy term in (3) is replaced by TRADES optimization objective. Adversarially training RNAS-CL. models improve its adversarial accuracy. RNAS-CI. models achieve similar or higher adversarial accuracy compared to other adversarially trained mod- els. However, RNAS-CO CL models are much smaller and achieve significantly higher clean aceuracy. For example, in Table 1, RNAS-CL-M-WRT-34 achieves similar or higher adversarial accuracy than most other methods while being smaller and significantly exceeding in terms of clean accuracy. We also obtain much smaller models using RNAS-CL. Tiny RNAS-CL models exceed their counter- part by more than ~ 12% in terms of clean accuracy. For example, RNAS-CL-SE CL--S-LL---3344 exceeds HYDRA (ResNet-34) by 12.95% while achieving similar adversarial accuracy. Similar results can also be visualized in Figure 1. In Figure 1, RNAS-CL, models are on the top right corner of the plot, representing the models with the highest clean and adversarial accuracy. Results for RNAS-CL models trained with different robust teachers have been added to the appendix A.2.
Comparison against various perturbation budget To further illustrate the effectiveness of RNAS- CL. we compare RNAS-CL with previously proposed defense mechanisms against various pertur- bation budgets. In Figure 3, we compare various methods against PGD and FSGM attacks. For both attacks, RNAS-CL outperforms its counterparts at all perturbations. RNAS-CL significantly outperforms other methods as perturbation size increases. For 6 0.1, RNAS-CI. exceeds other methods by 20% for both PGD and FSGM attacks.
60
60
Hydra RobNett-aagg
Adversanaa oveessanaal Nouresy
50
Adversont Accurasy
50
40
RNAS CL WAT 34 34AS-CL-L-VHT--34
30
10
0.03 0.04 0.05 0.05 0.07 0.08 0.09 0.10
0.03
0.04 0.05 0.06
0.07 0.08 0.09 0.10
Total Perturbation Size
Total Perturbation Size
Adversarial Accuracy CIFAR-10 under
(b) Adversarial Accuracy CIFAR-10 under different FGSM total perturbationsize
different PGD total perturbetionsize
Figure 3: Robustness evaluation under different perturbation sizes for PGD and FGSM attacks.
4.3 ABLATION STUDY
This ablation study demonstrates the significance of student-teacher cross-layer connections in RNAS-CL. We compare four types of training paradigms. In the first training paradigm, we con- duct searching and training using cross-entropy loss without any teacher model. We refer to this as standard. Next, in the second paradigm, we conduct searching and training by minimizing the cross-entropy loss and standard KL Divergence with a robust teacher model. We refer to them as KL-X-T, where X represents the search space and represents the robust teacher model. the third paradigm we search and train using cross-entropy loss and intermediate cross connections (ICC). We refer to them as ICC-X-T. Finally, the fourth model type is RNAS-CI, where we include all three terms, cross-entropy loss, KL Divergence, and cross-layer
-eacchrrt connections.
In Figure 4(a), we compare the attention maps from student models trained using RNAS-CL-I-R-50 against students trained using KL-I-R-50. We compare attention maps for various convolution layers at regular intervals. As expected, adding cross-layer connections obtains attention maps from the student model closer to the teacher model. Each student layer learns where to pay attention from its connected teacher layer. For example, in column (b), the KL-I-R-50 layer attends to various parts of the image, whereas the RNAS-CL. layer learning from 28th teacher layer pays more attention
8

---

Under review as a conference paper at ICLR 2023
1e) (b)
[c)
(d]
Teercher KL4-3000
10" layer
22" layer
35" laye
45" lags
Tracher
Input
layer
15.0
RNAS-CL--8-00
10.0
25 (b
1P (xper())
Nocthr
Atteention map
layers.
Teacher-Studeent Student Connections
Figure 4: (a) KL-I-R-50 represents attention maps from a model trained using cross-entropy loss and knowledge distillation without any cross-layer connections. Teacher and RNAS-CL represent attention maps from the robust teacher (ResNet-50) and RNAS-CL model. Name for each RNAS- CL layer includes its connected teacher layer. For example, in Oth layer (13), 13 represents the corresponding teacher layer. RNAS- CL drives attention maps from student layers closer their cor- responding teacher layer.(b) Illustrations of the number of student layers connected to each teacher layer in RNAS-CL for various student models on the CIFAR-10 dataset.
Adversara Accuracy
Suaddrt BE-SoWaLa
Panted
40
30
76
Adversarte
30
0.000.00 05.0 06 090010
0.0.000...00000 0.000.66. 0.080.090..0
11030.04 thd or Total Size
0.000.00..00 060.576.000
Total Perturbation Size
Perturaation (b)
Perturbation
(a)
[c]
(c)
Figure Adversarial accuracy of various models various perturbation budgets.
central part of the image. Similarly, in column (c), the RNAS-CL layer learns from the teacher model pay attention the central and upper portions of the image. In Figure 5, compare RNAS-CL models against KL-X--T and standard models against PGD attacks at various perturbation budgets on the CIFAR-I0 dataset. The RNAS-CL and ICC models outperform their counterparts, demonstrating the significance of cross-connections.
Robust Teacher layers We hypothesize that few teacher layers are more robust than others and thus should induce more robustness to the student models. In RNAS-CL, each student layer is associated with a teacher layer. Figures 4(b) illustrates the number of student layers connected each robust teacher layer on the CIFAR-10 dataset. For both student models, we observe that 15th and 21 layers of the robust teacher model have significantly more intermediate connections, suggesting that few teacher layers have more robustness-in robustness-inducing capacity than others. Plots for other student models have been added to the appendix A.I.
CONCLUSION
In this paper, we propose Robust Neural Architecture Search by Cross-Layer Knowledge Distillation (RNAS-CL) (LL, novel NAS algorithm that improves the robustness of the student model by learning from a robust teacher through cross-layer knowledge distillation. RNAS- CL optimizes neural archi- tecture achieve good tradeoff' between robustness and clean accuracy a differentiable manner without robust training. RNAS-CL extends conventional knowledge distillation by learning student- teacher cross-connections. We show that models obtained by RNAS-CL outperform all models obtained without robust training in terms of adversarial robustness. We show that adding adversarial training can further increase the adversarial robustness of RNAS-CI models. After robust training, RNAS-CL achieves similar adversarial robustness compared models obtained via robust training while outperforming them in terms of clean accuracy. For future work, we plan to incorporate robust training in the searching stage to further increase the robustness of the model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9163-9171, 2019.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference Learning Representations, ICLR, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy, pp. 39-57. IEEE, 2017.
Moustapha Cissé, Piotr Bojanowski, Edouard Grave, Yann N. Dauphin, and Nicolas Usunier. Parse- val networks: Improving robustness to adversarial examples. In Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML, volume 70 of Proceedings of Machine Learning Research. pp. 854 863. PMLR, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustmess with an ensemble of diverse parameter- free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Chaitanya Devaguptapu, Devansh Agarwal, Gaurav Mittal, Pulkit Gopalani, and Vineeth N Balasub- ramanian. On adversarial robustness: A neural architecture search perspective. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 152-161, 2021.
Minjing Dong, Yanxi Li, Yunhe Wang, and Chang Xu. Adversarially robust neural architectures. arXiv preprint arXiv:2009.00900 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In CVPR, pp. 9185-9193. Computer Vision Foundation JEEE Computer Society, 2018
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of jpg compression on adversarial images. arXiv preprint arXiv: 00853, 2016.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https: /ttgt//tttt///tt hatp:/u.....o////aaaaaaaaaa hotp://////cccaaaa///ooouaeeeee. robustnss................
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3996-4003, 2020.
Ian J. Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR, 2015.
Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model com- pression with adversarial robustness: A unified optimization framework. In Annual Conference on Neural Information Processing Systems, pp. 1283-1294, 2019.
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering adversarial images using input transformations. In 6th International Conference on Learning Representa- tions, ICLR, 2018.
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When NAS meets robustness: In search of robust architectures against adversarial attacks. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR, pp. 628-637, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz. and William J. Dally. EIE: efficient inference engine on compressed deep neural network. In ISCA, pp. 243-254. IEEE Computer Society, 2016.
10

---

Under review as a conference paper at ICLR 2023
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. Annual Conference on Neural Information Processing Sys- tems, pp. 226222266666 2017.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.0255 2015.
Hanxun Huang, Yisen Wang, Sarah M. Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Ex- ploring architectural ingredients of adversarially robust deep neural networks. In Advances in Neural Information Processing Systems, pp. 5545-5559, 2021.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioon with gumbel-softmax. In 5th International Conference on Learning Reprexentations, ICLR, 2017.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.0633 1803.6633,, 2018.
A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Univ. Toronto, 2009.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang. Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS'17 17 Competition: Building Intelligent Systems, pp. 195-231. Springer, 2018.
Changlin Li, Jiefeng Peng. Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xi- aojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1989-1998, 2020.
Hao-Ting Li, Shih-Chieh Lin, Cheng-Yehh Yeh Chen, and Chen-Kuo Chiang. Layer-level knowledge distillation for deep neural network learning. Applied Sciences, 9(10), 2019.
Yanxi Li, Zhaohui Yang, Yunhe Wang, and Chang Xu. Neural architecture dilation for adversarial robustness. Advances Neural Information Processing Systems, 34, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In 7th
International Conference on Learning Representations, ICLR, 2019.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsich. Towards robust neural networks via random self-ensemble. European Conference on Computer Vision, ECCV, pp. 381-397, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR, 2018.
Utkarsh Nath, Shrinu Kushagra, and Yingzhen Yang. Adjoined networks: A training paradigm with applications to network compression. arXiv preprint arXiv:2006. 05624, 2020.
Xuefei Ning. Junbo Zhao, Wenshuo Li, Tianchen Zhao, Yin Zheng, Huazhong Yang, and Yu Wang- Discovering robust convolutional architecture at targeted capacity: multi-shot approach. arXiv preprint arXiv:2012.11835 2020.
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross-entropy loss for adversarial robustness. In 8th International Conference on Learning Rep- resentations, ICLR, 2020.
Jongsoo Park, Sheng R. Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. In 5th International Confer- Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net,200 2017. URL htthttps: /ttps//////p///eee/ee// net feee/forumme forum??n TJPc7.3 rJ3e7.3xx.
11

---

Under review as a conference paper at ICLR 2023
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference Computer Vision (ECCV), pp. 268-284, 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2902-2911. PMLR, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4780-4789, 2019.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104, PMLR, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In 3rd International Conference on Learning Representations, ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211255 2015.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. HYDRA: pruning adversarially robust neural networks. In Annual Conference on Neural Information Processing Systems, 2020.
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve ad- versarial robustness? arXiv preprint arXiv:2 arXiv:2104.09425. 2021.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. robustness the cost of accuracy? A comprehensive study on the robustness of 18 deep image classification models. In European Conference on Computer Vision, ECCV, pp. 644-661, 2018
Dawei Sun, Anbang Yao. Aojun Zhou, and Hao Zhao. Deeply-supervised knowledge synergy. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR, pp. 6997-7006, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties neural networks. In 2nd International Conference Learning Representations, ICLR, 2014.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1820-2828, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In 8th International Conference on Learning Representations, ICLR, 2020a.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Inter- national Conference on Learning Representations, 2020b.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020c.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian J. Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference Learning Representations, ICLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
Frederick Tung and Greg Mori. Similarity-preservinn knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1374,1374, 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, and Joseph E. Gonzalez. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In JEEE/CVF Conference on Com- puter Vision and Pattern Recognition, CVPR, pp. 12962-12971, 2020.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001 .03994, 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 10734-10742, 2019.
Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In 8th Interna- tional Conference on Learning Representations, ICLR, 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2730-2739, 2019.
Guoyang Xie, Jinbao Wang, Guo Yu, Feng Zheng, and Yaochu Jin. Tiny adversarial mulit-objectivy oneshot neural architecture search. arXiv preprint S.KV::2033.00366 2021.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong- Pc-darts: Partial channel connections for memory-efficient architecture search. In ICLR. Open- Review.net, 2020.
Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dnns with improved adver- sarial robustness. In Annual Conference on Neural Information Processing Systems, pp. 417-426, 2018.
Shaokai Ye, Xue Lin, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, and Yanzhi Wang. Adversarial robustness vs. model compression, or both? In IEEE/CVF International Conference on Computer Vision, ICCV, pp. 111-120, 2019.
Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast
optimization, network minimization and transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 7130-7138, 2017.
Zhixiong Yue, Baijiong Lin, Yu Zhang, and Christy Liang. Effective, efficient and robust neural architecture search. 2022 International Joint Conference on Neural Networks (ICNNN,, pp. 1-8. IEEE, 2022.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the per- formance of convolutional neural networks via attention transfer. In 5th International Conference on Learning Representations, ICLR, 2017.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Proceedings of the 36th International Conference on Machine Learning, ICML, volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482. PMLR, 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jor- dan. Theoretically principled trade-off between robustness and accuracy. In ICML, volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482. PMLR, 2019b.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th Inter- national Conference on Learning Representations, ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ROBUST TEACHER LAYERS
In this section, we discuss robustness inducing capacity of teacher layers. We hypothesize that few teacher layers are more robust than others and thus should induce more robustness to the student models. In RNAS- number of student layers connected to each robust teacher layer on the CIFAR-10 and ImageNet- 100 datasets. For all student models on CIFAR-10, we observe that layers 15 and 21 of the robust teacher model have significantly more intermediate connections with the student models. Similarly, for ImageNet-100 100, layers 18, 32, and 40 a few of the dominant robust layers. In Figures 7 and 9, we visualize the most robust teacher layers on CIFAR-10 and ImageNet, respectively.
each student layer is associated with teacher layer. Figures 6 and 8 plot the
Number of cornected student layer Number cf student layer
RNAS-CL-S3
HNAS-CL-SS
17.5
10
15.0
12.5
Number of carrected student leyer Number connected
0.0
0
5
10
3.5
20
25
30
Teacher leyer number
Teaacher layer number
RNAS-CL-S7
RNAS-CL-L
35
40
25
20
20
15
20
0
Teacher layer number
Teacher layer number
Figure 6: Illustrations of the number of student layers connected to each teacher layer in RNAS-CL for various student models the CIFAR- 10 dataset. We choose adversarially trained -ete-ResNet- 34 as the robust teacher model for all the four student models, with one plot for each student model. All student architectures described in Table 7.
Image
15th Layer
21 Layer
Figure 7: Attention map for most robust teacher layers on CIFAR-10 dataset. We chose the same robust teacher model as in Figure 6. The illustrated layers represent teacher layers with maximum number of intermediate connection for various RNAS-CL models (as described in Figure 6).
14

---

Under review as a conference paper at ICLR 2023
Number of correcedd Student toyer
Namber Cornacted studant bayar
RKAS CL/5
3.0
3.0
2.5
2.5
2.5
2.0
2.0
2.0
1.5
2.3
1.0
0.0
0.3
10
20
30
40
50
Taacher layer number
(a)
(b)
(c)
Figure 8: Illustrations of the number of student layers connected each teacher layer in RNAS-CL for various student models on the ImageNet- 100 dataset. We choose adversarially trained Wide- ResNet-50 the robust teacher for all and three students models, with one plot for each student model. All RNAS-CL architectures are described in Table 8.
Image
8th Layer
16th Layer
18" Layer
26 Layer
32th Layer
40th Layer
Figure 9:
Attention maps for most robust teacher layers on ImageNet- 100 dataset. We chose the robust teacher model as in Figure 8. The illustrated layers represent teacher layers with maxi- mum number of intermediate connection for various RNAS-CL models (as described in Figure 8).
A.2 MORE RESULTS ON CIFAR- 100
In this section, we conduct experiments using adversarially trained WRT-34 (Rice et al., 2020), ResNet-50 (Engstrom al., 2019), and ResNet-18 (Sehwag al., 2021) as the robust teacher models on the CIFAR-10 dataset. All RNAS-CI models, while achieving similar clean accuracy, exceed its counterpart by more than 10% in PGD accuracy. RNAS-CL-R50 achieves higher robust accuracy than RNAS-CL-RI8 and RNAS-CL-WRT-S among the teacher models, suggesting that the teacher's architecture has more influence on the student's performance than the teacher's performance. The higher number of teacher layers allows more options for the student layer learn from, leading to better robustness. The teacher models' performance is reported in Table 6.
However, ResNet-50 has the lowest PGD accuracy
A.3 COMPARE EFFICIENT AND ROBUST IMAGENET-IO0 MODELS
We compare RNAS-CL to adversarially robust pruning methods on ImageNet- dataset, with results shown in Table 3. RNAS-CL models are trained with three different robust teachers, ResNet- 18, ResNet-50, and WideResNee-50,, with the ImageNet pre-trained (Engstrom et al., 2019) being the robust teacher. It is observed that RNAS-CL models consistently exceed other models by 25% terms of clean accuracy while exibiting adversarial robustness. Table 3, both Hydra and LWM were adversarially trained using TRADES (Zhang al., 2019a), For a fair comparison, after the regular training stage without TRADES, we retrain our RNAS-CL models with the TRADES optimization objective. We replace the cross-entropy term in (3) by the TRADES optimization
15

---

Under review as a conference paper at ICLR 2023
Method
Clean
PGD
Standard-SS S3
89.92
17.69
Standard-SS
90.76
18.44
Standard-S7
90.98
T9.3
RNAS-CT-S33WWRT
89.4
34.3
RNAS-CT-S3-WRRT-
90.4
35.39
RNAS-CL-S7.WRT33
90.62
37.24
RNAS-CL-S3-R5O
89.39
35.76
RNAS-CL-SS-R5O
90.33
37.32
RNAS CL-S--O
90.41
37.98
RNAS RNAS-CL-S3-PR CL-SCL-
88.47
26.35
RNAS-CL CL-S-88
88.77
25.49
RNAS-CISS7RR8
89.47
27.96
Table 2: Performance of RNAS-CL method trained with various robust teacher models the CIFAR-10 dataset. Standard represents models searched and trained by cross-entropy loss with- out any teacher model.
Method
Clean
PGD
# Params (M)
MACs (M)
Hydra (ResNet-18) 90% (Sehwag al., 2020)
59.96
29.79
1.1
1200
"LWM (ReSNei-1) 90% (Han et 2013)
59.02
27.67
T.1
1200
INAS-CL-
85.22
8.3 "8.3
3.94
241.98
RNAS-CL-1---50
85.98
3..08"
3..66
244.76
RNAS CL-I--L---
85,46
3.36
4.01
255.37
18 TRADES
78.94
29.02
3.94
241.98
RNAS-CL-I-R-55 50 TRADES
79.95
32.44
3.96
244.76
RNAS-CILLWTTSS0 TRADES
79.42""
28.06
4.01
253337
Table Performance of various efficient and robust methods on ImageNet- dataset. Clean Acc and Adv Acc are the same as that in Table 1. All MACs were calculated without special hardware (Han et al., 2016) or special software (Park et al., 2017)
objective. With such training, RNAS-CL achieves similar or higher adversarial accuracy while significantly outperforming Hydra and LWM clean accuracy with only a fraction of MACs.
ImageNet- 1eeeNe 100 Adversarial Accuracy
35
Standard
30
KL-WRT-50
CLAS-CL---RR-50
25
KL-R-50
RNAS-CL---R-50
20
KL-R-18
15
CL-I-R-18
10
0.015
0.025
0.030
Perturbation Budget
Figure 10: Adversarial accuracy of various models at various perturbation budgets on the ImageNet- 100 dataset.
We further study adversarial accuracy at various perturbation budgets for three different teacher models. As illustrated in Figure 10, RNAS-CL exceeds its counterpart in adversarial accuracy various perturbation budgets for all teacher models on the ImageNet- 100 dataset. This demonstrates the significance of cross-layer connections in RNAS-CI.
16

---

Under review as a conference paper at ICLR 2023
A.4 COMPARE CIFAR-10 10 MODEL AGAINST CW AND AUTOATTACK
In this section, we compare RNAS-CL and (Huang et al., 2021) against recent attacks such as CW (Carlini & Wagner, 2017) and AutoAt (Croce & Hein, 2020) on CIFAR-10 dataset. CW attacks were proposed to defeat defensive distillation. In Table 4, we use Los version of CW attack opti- mized by PGD, with maximum perturbation budget set t = 8/255, AutoAttack a parameter-fre ensemble attack currently considered one of the most reliable and widely acknowledged evaluation benchmark in Adversarial Defences.
Method
CW 30
VGG-R (Huang et al., 2021)
46.49
38. 44
DN=121-R (Huang et al. 2021)
"33.07
47.75
"47.07
33..77
RNAS-CL-S5 CL-SS---ST------ WPRT-34((Ou)
48.33
39. 28
RNAS-CL-ST. WRT- W0T--4(33333
47.91
38.36
RNAS-CL-M-WRT-344C
53.52
46.89
322.33"
48:49
Table 4: The table compared performance of (Huang et al., 2021) and RNAS-CL against CW
(Carlini Wagner, 2017) and AutoAttack (Croce Hein, 2020) on CIFAR-IO dataset.
A.5 COMPARISON AGAINST KD VARIENTS
In this section, we compare our methods against various knowledge distillation methods Park al. (2019); Ahn et al. (2019); Tung & Mori (2019); Tian et al. (2020b); Passalis & Tefas (2018). We use Robust WRT-34 as the teacher model for all KD methods and train three different student archi- tectures: RNAS-CL-S3, RNAS-CL-S5, and RNAS-CL-ST In Figure 11, models trained using our paradigm are explicitly on the upper right-most part of the graph. RNAS-CL-S3 architecture trained using RKD performs similarly to the model trained using methods. Apart from this, all models trained using RNAS-CI significantly outperform all other methods in terms of clean and adversarial accuracy.
Similarity
VID
Adversarial Accuracy
RKD
34
CRD
PKD RNAS-CL (Our)
32
87.5
88.0
88.5
89.0
89.5
90.0
90.5
Clean Accuracy
Figure 11: The figure compares various knowledge distillation variants (Similarity (Tung & Mori, 2019), VID (Ahn et al., 2019). RKD (Park et al., 2019), CRD (Tian et al., 2020b), PKD (Passalis & Tefas, 2018)) against RNAS-CL on the CIFAR-10 dataset. Adversarial Accuracy represents top-1 Accuracy images perturbed by 20 step PGD attack. Clean Accuracy represents top-1 Accuracy on clean images. Larger marker size indicates larger architecture. For each method, RNAS-CL-S3,, RNAS-CL-SS,, and RNAS-CL-S7 represented by increasing marker size.
A.6 RESULTS FOR IMAGENET
In this section, we compare our method on the ImageNet dataset. Standard represents the model searched and trained using cross-entropy loss without any teacher model. RNAS-CI represents the
17

---

Under review as a conference paper at ICLR 2023
model trained using training paradigm. Both models further adversarially trained using FastAT (Wong et al., 2020). Table 5, we evaluate the robustness against 10 step PGD attack with 4/255. Models trained with RNAS-CL exceed both in terms of clean and robust accuracy.
Method
Clean
PGD
Standard
53.92
25.45
RNAS- WRT-50
56.1
29.78
Table 5: Robustness results on ImageNet dataset.
A.7 ROBUST TEACHER MODELS
In this section, we report the robustness of adversarially trained teacher model used throughout the paper on CIFAR-10 10 dataset.
Model
Clean
PGD
WRT-34
86.07
58.33
ResNet
84.59
55.54 87.03
ResNet 50
49.25
Table Robustness results for various teacher model on CIFAR-I0 dataset.
A.8 ARCHITECTURE
In this section, we discuss architectures for various proposed super-nets used in RNAS-CL for CIFAR-10 and ImageNet-10) datasets. Table 7 describes the super-nets used for CIFAR-10. We use super-nets with three blocks. Super-nets used for ImageNet-100 are described in Table 8. For ImageNet-100, the number of blocks varies from 3 to 5.
Search Space for CIFAR- 10
Search Space
Depth
Stage
Stage
Stage 3
RNAS-CI-S3
3-3-3
16. 12
32, 28, 24, 20
64, 60, 56, 52
RNASSCIESS
53555
16. 12
28, 24,"20"
64, 60) 36, 56,52 32"
RNAS CL-S7
7-7-7
32, 28, 24, 20
64. 60.
ANAS-CLAM
9.7-1
80, 76
160, 156, 152, 148
128, 124, 120, 116
RN-S-CL-L 9-7-1
160. 156
3300, 316,,,6,,,, 312, 308
252. 248, 244
Table 7: The table describes the search space for CIFAR-10. 10. Depth represents the depth of cach stage. For example, 3-3-3 represents three convolution blocks in each stage. All search spaces have three stages. Stage 1, Stage 2, and Stage represent the filter choices for their respective stages. For example, stage 3 of RNAS-CL-SS for each convolution block, search between 4 output channels (64, 60, 56, 52).
A.9 ARCHITECTURE SEARCH BY FBNETV2
RNAS-CL builds both an efficient and adversarially robust deep learning model. In this work, we the training paradigm of FBNetV2 search for efficient models. Figure 12, we illustrate the searching process for neural architecture at single convolution layer. Each filter choice is attached with Gumbel weight. These Gumbel weights are optimized to select an efficient model.
18

---

Under review as a conference paper at ICLR 2023
Search Space for ImageNet- 100
Search Space
Depth
Stage
Stage
Stage
Stage
Stage
96, 88, 80,
28,
40, 36.
RNAS-CI-IS
3-3-3
72, 64,56, 56,
20, 16
32, 28
48
96, 88, 80,
128 120, 108, 100, 92, 84, 76, 68
28, 24,
40, 36,
RNAS-CI-IM
3-3-3-4
72, 64, 56, 48
20, 16
32,
216, 208, 200,
96, 88, 80,
128 120, 108, 100, 92, 84,
192, 184,176,
28, 24,
40, 36,
RNAS-CL-I
3-3-3-4-4
72, 64, 56,
168, 160, 152, 144,136, 128,
20, 16
32, 28
48
76, 68
120, 108
Table 8: The table describes the search space for ImageNet-100 100. Similar to Table 7, depth represents depth each stage. For ImageNet- 100, we have up to stages. Stage 1, Stage 2, Stage 3, Stage 4, and Stage represent the filter choices for their respective stages. For example, in stage for each convolution block, we search for its channel within output channel options (28, 24, 20, 16).
Convolution Filters of Convolution Block (Student Layer)
9
Xgw
Output
xg
(4)
xg
Mask
Figure 12: Illustration of searching for the neural architecture of each layer of student model using the searching mechanism in FBNetV2. d', represents gumbel weights associated with each mask.
19


Reviews:
[{"review_id": "zC_txVAW48", "reviewer": "Reviewer_4K2f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to obtain a robust and efficient neural architecture by searching the best teacher layers and the number of filters for student network. Experiments are conducted on CIFAR and ImageNet-100, which show good performance over prior methods.", "summary_of_the_review": "My main concerns lie on the limited novelty of the proposed method and the unfair comparisons with prior work.", "strength_and_weaknesses": "Strength:\n1. It make senses to automatically search for the best teacher layer to supervise each student layer, and ablation studies justify the effectiveness of this strategy.\n2. The paper is well written and easy to understand.\n\nWeaknesses:\n1. The knowledge distillation method is quite standard except for the optimal teacher layer searching. For example, there are not much new things in Section 3.1 and distillation loss in Section 3.4.\n2. The architecture search method seems to have limited novelty over FBNetv2.\n3. The proposed method use knowledge distillation to improve the performance, while the compared methods not. It is not quite fair and no ablation studies to show the performance of the proposed method without knowledge distillation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "neaTpNo1Jz", "reviewer": "Reviewer_xKFw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper utilizes neural architecture search to discover neural network architectures that are adversarially robust and with low latency at the same time. In particular, they introduce an adversarially robust teacher model for cross-layer distillation in the training and search process to improve the adversarial robustness of the model, and specifically search which layers of the teacher model are used for feature distillation.", "summary_of_the_review": "My concerns about this paper mainly come from the insufficient experimental verification. I may consider improving the score if the authors could provide the missing experiments and results.", "strength_and_weaknesses": "==== Strength ====\n- The first work that applies knowledge distillation-based NAS for adversarial robustness.\n- Intuitively, the proposed method is reasonable.\n- The paper writing is clear, the method description is easy to understand.\n\n==== Weakness and Questions ====\n\n1. Unsupported statements and missing discussions.\n\na) This paper claims in the introduction that RNAS-CL is ``the first NAS method that jointly optimizes accuracy, latency, and robustness against adversarial attacks''. However, quite a few studies have already explored this problem, for example, [1][2][3]. Should remove this statement, and discuss / compare with them.\n\nb) The proposed method adopt ideas from existing methods, should discuss more about these existing work, for example, [4].\n\n2. Weakness and questions about the experiments.\n\na) Missing comparisons with robustness NAS studies like [1][2][3].\n\nb) Necessary baseline results are missing. Table 1 shows the performance of the architectures discovered by different methods on CIFAR-10. The RNAS-CL-* models are searched with the Wide-ResNet teacher model, but the performance of the teacher model is not mentioned. The performance of Wide-ResNet should be given as baseline results. In particular, I wonder if the searched architecture outperforms the Wide-ResNet model, which is very important to verify the effectiveness of the proposed method;\n\nc) Why are only the results of using WRT as the teacher model on CIFAR-10 reported in Table 1? For ImageNet-100, the results with different models as teacher models are provided in the appendix. How will the teacher model's architecture or performance influence the performance of discovered architecture.\n\nd) Some essential experiment settings are missing. For example, what’s the step number for PGD attack in the paper?\n\n[1] Xie, Guoyang, et al. \"Tiny adversarial mulit-objective oneshot neural architecture search.\" arXiv preprint arXiv:2103.00363 (2021).\n\n[2] Ning, Xuefei, et al. \"Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach.\" arXiv preprint arXiv:2012.11835 (2020).\n\n[3] Yue, Zhixiong, et al. \"Effective, efficient and robust neural architecture search.\" 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.\n\n[4] Li, Changlin, et al. \"Block-wisely supervised neural architecture search with knowledge distillation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nAnother minor suggestion: You should use ``'' for quotation marks in LaTex.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6XvKN0m5Qn9", "reviewer": "Reviewer_k7LD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a NAS method to find robust neural network that can defend adversarial attracks. It combine knowledge distillation with NAS to pursue better performacne. The authors conduct experiments on multiple tiny datasets, such as CIFAR-10, ImageNet-100.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Strength:\n\n-It is interesing to find robust architecture with automatic search. \n\n-The proposed method use a teacher to enhance the model's performance and robustness, which is reasonable. \n\n-This paper is well-written and easy to follow.\n\nWeakness:\n\n-Directly combining NAS and knowledge distillation can usually improve performance. The authors should compare the proposed method with other knowledge distillation methods.\n\n- Only tiny datasets are used to evaluate the effectiveness of the proposed method. These datasets have a large gap with the practical scense. Comparesion on large dataset (such as ImageNet) is required to validate the method's effectiveness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject