SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. For the NAS-Bench-201 [12, 22] search space, we validate our proxy across different tasks (CIFAR-10, CIFAR-100, and ImageNet16-120) and perturbations (FGSM [16], PGD [29], and 15 types of common corruptions [20]). To measure Spearman's  $\rho$  between final accuracies and our proxy values, we use both clean NAS-Bench-201 [12] and robust NAS-Bench-201 [22], which include clean accuracies and robust accuracies, respectively. Finally, we search for the optimal architectures with our proxy in DARTS [26] search space and compare the final accuracies against previous NAS methods [31, 18, 6, 1] on CIFAR-10 and CIFAR-100.
Standard Training & Adversarial Training. For a fair comparison, we use the same training and testing settings to evaluate all the architectures searched with all NAS methods, including ours. 1) Standard Training: We train the neural architectures for 200 epochs under SGD optimizer with a learning rate of 0.1 and weight decay 1e-4, and use a batch size of 64 following [31]. 2) Adversarial Training: We train the neural architectures with  $l_{\infty}$  PGD attacks with the epsilon of 8/255, step size 2/255, and 7 steps. We evaluate the robustness against various perturbations, which are adversarial attacks (FGSM [16], PGD [29], CW [5], DeepFool [32], SPSA [38], LGV [17], and AutoAttack [7]) and common corruptions [20]. More experimental details are described in Supplementary A.
4.2 Results on NAS-Bench-201
Standard-Trained Neural Architectures. We verify our proxy's capability to search for high-performing neural architectures across various tasks and perturbations by employing Spearman's  $\rho$  to measure the rank correlation between the proxy values and final accuracies [1, 30, 11]. While existing clean zero-shot NAS works [1, 30] display weaker correlations with clean accuracies than using the number of parameters (#Params.) as a proxy, our proxy exhibits significantly stronger correlations with clean accuracies across all tasks. Specifically, it shows improvements of  $5.92\%$  and  $1.95\%$  on CIFAR-10 and CIFAR-100, respectively, compared to best-performing baselines (Table 2).
Furthermore, CRoZe shows remarkable Spearman's  $\rho$  for robust accuracies obtained against adversarial perturbations and corrupted noises across tasks. Notably, our proxy outperforms the SynFlow [1] by  $6.41\%$  and  $8.77\%$  in an average of Spearman's  $\rho$  for adversarial perturbations and common cor
Table 2: Comparison of Spearman's  $\rho$  between the actual accuracies and the proxy values on CIFAR-10, CIFAR-100, and ImageNet16-120 in NAS-Bench-201 search space. Clean stands for clean accuracies and robust accuracies are evaluated against adversarial perturbations (FGSM) [16] with various attack sizes  $(\epsilon)$  and common corruptions [20]. Avg. stands for average Spearman's  $\rho$  values with all accuracies within each task. Bold and underline stands for the best and second, respectively. All models are standard-trained on clean images.
Proxy Type	CIFAR-10	CIFAR-100	ImageNet16-120
Clean	FGSM	Common Corruption (CC.)	Clean	FGSM	Common Corruption (CC.)	Clean	FGSM
ε = 8	ε = 4	Weather	Noise	Blur	Digital	Avg.	ε = 4	Weather	Noise	Blur	Digital	Avg.	ε = 4	Avg.
FLOPs	0.726	0.753	0.740	0.665	0.138	0.232	0.473	0.532	0.699	0.661	0.674	0.253	0.492	0.607	0.557	0.680	0.611	0.634
#Params.	0.747	0.756	0.739	0.674	0.131	0.229	0.489	0.502	0.720	0.654	0.685	0.240	0.489	0.618	0.568	0.683	0.627	0.655
Plain [1]	-0.073	-0.059	-0.055	-0.041	0.035	0.048	-0.032	-0.025	-0.242	-0.172	-0.169	0.055	-0.110	-0.182	-0.137	-0.231	-0.241	-0.236
Grasp [1]	0.551	0.657	0.673	0.641	0.250	0.226	0.426	0.532	0.550	0.649	0.565	0.311	0.468	0.518	0.510	0.543	0.601	0.572
Fisher [1]	0.383	0.466	0.499	0.411	0.179	0.221	0.244	0.343	0.382	0.575	0.425	0.325	0.447	0.396	0.425	0.324	0.394	0.359
GradNorm [1]	0.637	0.760	0.774	0.540	0.239	0.183	0.375	0.460	0.638	0.789	0.654	0.357	0.562	0.613	0.602	0.573	0.649	0.611
SynFlow [1]	0.777	0.778	0.751	0.673	0.188	0.181	0.554	0.557	0.769	0.683	0.703	0.215	0.439	0.641	0.575	0.751	0.695	0.723
NASWOT [30]	0.687	0.522	0.513	0.479	-0.078	0.012	0.434	0.367	0.708	0.520	0.525	-0.023	0.319	0.534	0.431	0.698	0.730	0.714
CRoZe	0.823	0.826	0.801	0.743	0.190	0.236	0.565	0.598	0.784	0.693	0.747	0.251	0.504	0.682	0.610	0.765	0.696	0.731
(a) Search on Clean Inputs
Figure 3: Search with our proxy on CIFAR-10 in NAS-Bench-201 search space. Our proxy can reduce the sampling costs in searching for neural architectures against both clean (left) and perturbed samples (right).
(b) Search on Adversarially Perturbed Inputs
ruptions on CIFAR-10, respectively (Table 2). Our results on multiple benchmark tasks with diverse perturbations highlight the ability of our proxy to effectively search for robust architectures that can consistently outperform predictions against various perturbations. Importantly, our proxy is designed to prioritize generalizability, and as a result, it exhibits consistently enhanced correlation with final accuracies for both clean and perturbed samples. This result indicates that considering generalization ability is effective in identifying robust neural architectures against diverse perturbations but also leads to improved performance for clean neural architectures.
Adversarial-Trained Neural Architectures. We also validate the ability of our proxy to precisely predict the robustness of adversarially-trained networks, specifically for adversarial perturbations. Adversarial training [29] is a straightforward approach to achieving robustness in the presence of adversarial perturbations. To assess the Spearman's  $\rho$  of robustness in adversarially-trained networks, we construct a dataset consisting of final robust accuracies of 500 randomly sampled neural architectures from NAS-Bench-201 search space that are adversarially-trained [29] from scratch. The final robust accuracies are obtained for 7 different adversarial attacks, including FGSM, PGD, CW, DeepFool, SPSA, LGV, and AutoAttack.
Notably, our proxy showcases the highest overall correlation for the 7 different adversarial attacks, supporting the ability of our proxy to search the architectures that are robust against diverse perturbations. Considering the trade-off between the clean and robust accuracy in adversarial training [43], we employ the harmonic robustness score (HRS) [9] to evaluate the overall performance of the adversarially-trained models. When comparing the correlations between clean performances and proxy values, existing clean zero-shot NAS methods, i.e., SynFlow and Grasp, demonstrate higher correlations in the FGSM or PGD, but their correlations with clean accuracies are poorer than GradNorm and Fisher, respectively. This result shows that clean zero-shot NAS methods tend to search for architectures that are more prone to overfitting to either clean or robust tasks (Table 1, 2). In contrast, CRoZe consistently achieves higher Spearman's  $\rho$  for both clean and robust tasks, ultimately enabling the search for architecture with high HRS due to consideration of alignment in gradients.
4.3 End-to-End Generalization Performance on DARTS
In this section, we evaluate the effectiveness of CRoZe in rapidly searching for generalized neural architectures in the DARTS search space and compare it with previous clean one-shot NAS [42, 6], clean zero-shot NAS [37], and robust NAS [31, 18] in terms of performance and computational cost.
Efficient Sampling with CRoZe. The efficiency of our proxy lies in evaluating neural architectures without iteratively training models. As a result, most search time is dedicated to sampling the
Table 3: Comparisons of the final performance and search time of the searched network in DARTS search space on CIFAR-10 and CIFAR-100. All NAS methods are executed on a single NVIDIA 3090 RTX GPU.
Task	NAS Method	Training-free NAS	Params (M)	Search Time (GPU sec)	Standard-Trained
Clean	CC.	FGSM	HRS
CIFAR-10	PC-DARTS [42]	3.60	8355	95.14 (± 0.16)	73.57 (± 0.89)	17.74 (± 3.44)	29.76 (± 4.81)
DrNAS [6]	4.10	46857	94.18 (± 0.33)	71.79 (± 0.74)	15.63 (± 2.78)	26.71 (± 4.01)
RobNet [18]	5.44	274062	94.94 (± 0.26)	72.49 (± 0.45)	14.69 (± 1.15)	25.42 (± 1.71)
AdvRush [31]	4.20	251245	93.44 (± 0.41)	71.43 (± 1.07)	14.94 (± 2.56)	25.67 (± 3.87)
GradNorm [1]	✓	4.69	9740	92.95 (± 0.67)	63.03 (± 6.33)	15.64 (± 0.15)	26.78 (± 0.24)
SynFlow [1]	✓	5.08	10138	92.15 (± 1.25)	70.74 (± 2.69)	12.65 (± 1.46)	22.22 (± 2.32)
CRoZe	✓	5.52	17066	94.34 (± 0.08)	74.11 (± 1.40)	20.51 (± 1.32)	33.07 (± 2.30)
CIFAR-100	PC-DARTS [42]	3.60	8355	76.95 (± 0.67)	50.02 (± 0.05)	7.68 (± 0.47)	13.96 (± 0.80)
DrNAS [6]	4.10	46857	75.49 (± 1.66)	49.18 (± 1.31)	8.14 (± 0.22)	14.69 (± 0.35)
RobNet [18]	5.44	274062	76.40 (± 0.30)	49.77 (± 0.82)	7.50 (± 0.84)	13.65 (± 1.39)
AdvRush [31]	4.20	251245	74.48 (± 1.35)	48.27 (± 1.42)	7.46 (± 0.81)	13.55 (± 1.37)
GradNorm [1]	✓	3.83	9554	68.52 (± 0.80)	43.69 (± 2.22)	6.87 (± 1.58)	12.44 (± 2.61)
SynFlow [1]	✓	4.42	9776	76.23 (± 0.73)	49.60 (± 0.78)	9.01 (± 0.56)	16.11 (± 0.91)
CRoZe	✓	4.72	17457	75.46 (± 0.81)	49.33 (± 0.38)	9.75 (± 0.78)	17.26 (± 1.22)
candidate architectures. To thoroughly explore the sampling costs involved in identifying good neural architectures, we conduct experiments in the NAS-Bench-201 search space with two representative sampling-based search algorithms: random search (RAND) and evolutionary search (AE).
Our experiments show that CReZe can rapidly identify an architecture with  $91.5\%$  clean accuracy and  $50\%$  FGSM robustness, even early in the search. By leveraging the initial pool of architectures from CReZe with high proxy values (RAND+warmup and AE+warmup), our proxy cuts sampling costs and rapidly identifies high-performing architectures for clean inputs (Figure 3a). Moreover, by focusing sampling around the architecture with the highest proxy values of CReZe (AE+warmup+move), we can find even more robust architectures using fewer sampled models ( $< 50$ ) (Figure 3b). In essence, CReZe effectively identifies high-performing neural architectures for both clean and perturbed inputs in the NAS-Bench-201 search space.
End-to-End Performance. We validate the final performance of the neural architectures discovered by CRoZe (Table 3, 4) and compare the search time and performance with existing NAS methods including robust one-shot NAS (RobNet [18] and AdvRush [31]), clean one-shot NAS (PC-DARTS [42] and DrNAS [6]), and clean zero-shot NAS (SynFlow, GradNorm [1]). For a fair comparison between clean zero-shot NAS (SynFlow, GradNorm) and CRoZe, we sample the same number of candidate architectures (e.g., 5,000) using the warmup+move strategy in the DARTS search space. We report the average and standard deviation of accuracies evaluated over 3 different neural architecture searches. All experiments are conducted on a single NVIDIA 3090 RTX GPU to measure search costs.
In standard training, the searched network by our proxy surpasses the networks by robust one-shot NAS methods, RobNet and AdvRush, in terms of robust accuracy against FGSM on CIFAR10, improving by  $5.82\%$  and  $5.57\%$ , respectively (Table 3). CRoZe also shows the highest HRS accuracy,  $7.40\%$  better than AdvRush on CIFAR-10, indicating its efficiency in mitigating the trade-off between clean and robust accuracy, while reducing the search cost by 14.7 times.
Table 4: Final performance of the searched networks that are adversarially-trained on CIFAR-10.
NAS
Method	Clean	Robustness
PGD	CW	SPSA	LGV	AA.	Avg.
PC-DARTS	86.02	52.16	9.85	86.00	79.32	48.82	55.23
DrNAS	86.45	54.66	6.44	85.86	79.64	52.40	55.80
RobNet	80.53	50.62	22.91	80.85	77.79	46.34	55.70
AdvRush	85.98	53.89	6.68	84.81	79.61	51.88	55.37
GradNorm	81.61	49.86	12.02	77.18	73.27	46.69	51.61
SynFlow	77.08	45.95	26.50	75.78	74.14	42.45	52.96
CROZe	84.28	52.17	19.13	83.43	76.85	48.14	55.94
Compared to the previously best-performing clean zero-cost proxy, SynFlow, CRoZe finds architectures with superior performance in clean, common corruptions, and FGSM scenarios, showcasing the effectiveness of our proxy in identifying generalized architectures. Moreover, the architecture chosen by CRoZe outperforms clean one-shot NAS methods in HRS accuracy, addressing vulnerability against both perturbations and clean inputs effectively (Figure 2a). When we adversarially train the searched architectures, CRoZe achieves the highest average robustness against 5 adversarial attacks at much lower search costs compared to robust NAS on CIFAR-10 (Table 4, Figure 2b).
Ablation of Each Component of CRoZe. Our proxy consists of three components: feature, parameter, and gradient consistency. To analyze the importance of considering all factors for evaluating the robustness of the given neural architecture, we conduct an ablation study in both the
Table 5: Comparisons of the final performance of the searched network in NAS-Bench-201 and DARTS search space on CIFAR-10. All models are standard-trained. Bold and underline stands for the best and second.
Proxy components	NAS-Bench-201	DARTS
Feature	Parameter	Gradient	Clean	CC.	FGSM	HRS	Avg.	Clean	CC.	FGSM	HRS	Avg.
✓	-	-	93.30	55.62	44.30	60.08	63.33	94.37	72.26	16.87	28.62	53.03
✓	✓	-	93.70	56.93	45.80	61.53	64.49	94.99	74.06	16.82	28.58	53.61
✓	-	✓	93.70	56.93	45.80	61.53	64.49	94.30	74.91	16.67	28.33	53.55
-	✓	✓	93.70	56.93	45.80	61.53	64.49	94.34	74.46	15.71	26.93	52.86
✓	✓	✓	93.70	56.93	45.80	61.53	64.49	94.45	74.63	22.38	36.19	56.91
Figure 5: Spearman's  $\rho$  in our proxy and each consistency component between the architectures with single-step trained states and two of fully-trained and 10-step trained states, respectively, against clean and perturbed images.
NAS-Bench-201 search space and the DARTS search space. On the NAS-Bench-201 search space, all ablation proxies identified the same candidate architecture except for the feature-based proxy. To compare the ablation proxies thoroughly, we extend our examination to the DARTS search space, which contains a significantly larger number of candidate architectures (e.g.,  $10^{19}$ ) compared to the NAS-Bench-201 search space. As shown in Table 5, architecture identified solely by feature consistency exhibits better average performance compared to those discerned by proxies without feature consistency, emphasizing the influential role of feature consistency in evaluating robustness. Notably, the architecture selected by our proxy showcases the highest average performance. This indicates that both parameter and gradient consistency can bolster feature consistency, offering more insight into the learning trajectory, as elaborated in Section 3.3. Overall, considering all components is crucial to evaluate the robustness within a single batch.
Feature Variance of the Robust Models. Our proxy searches for architectures that can generalize to all types of perturbed inputs, which are not overfitted to a single type of perturbation, as described in Section 3.1 (Eq. 2). To validate this, we analyzed the feature variance of neural architectures that demonstrated the highest performance against a single type of perturbation, namely FGSM and PGD, compared
Figure 4: Feature variance across perturbations.
to an architecture selected with the CRoZe proxy from a pool of 300 standard-trained and adversarially-trained models (Figure 4). Features were obtained from 18 different perturbations, including clean, PGD, FGSM, and 15 types of common corruptions. Remarkably, the neural architectures selected with our proxy exhibited the smallest standard deviations between the features obtained from the 18 different perturbations, in both standard and adversarially-trained scenarios. Conversely, the architecture with the best PGD robustness demonstrated 2.48 times larger variations in adversarially-trained scenarios, indicating a higher risk of overfitting to specific perturbations. These results provide compelling evidence of the effectiveness of CRoZe in identifying robust architectures that can consistently extract features and generalize across diverse perturbations.
Assessment of CRoZe Predictiveness. To empirically validate whether our proxy obtained from a random state accurately represents the proxy in the fully-trained model, we conducted an analysis using Spearman's  $\rho$ . We randomly sampled 300 architectures from the NAS-Bench-201 search space and trained them on the entire dataset, using both the full training and reduced training only of 10 steps. The Spearman's  $\rho$  between the proxy value obtained from a single-step trained state and the 10-step trained state shows a strong correlation of 0.995. Even after full training, the correlation remained high at 0.970 (Figure 5). These suggest that our proxy, derived from estimated surrogate networks, can significantly reduce the computational costs associated with obtaining fully-trained models. Furthermore, each component of CRoZe consistently demonstrates a high correlation with
the fully-trained states, supporting the notion that the high correlation computed with the final state is not solely a result of the collective influence of components, but rather an accurate estimation.
Validation of CRoZe for Estimating Fully-Trained Neural Architectures. To investigate whether the architectures selected with our proxy possess the desired robustness properties discussed in Section 3.1, we conduct an analysis. From a pool of 300 samples in the NAS-Bench201 search space, we select top-5 and bottom-5 architectures based on our proxy, excluding
Figure 6: Comparisons of feature consistency in fully-trained states.
those with a robust accuracy of less than  $20\%$ . We evaluate the performances of these architectures that are standard-trained (ST) and adversarially-trained (AT) against clean inputs and three distinct types of perturbed inputs: adversarial perturbations (i.e., FGSM and PGD), and common corruptions.
Notably, the top-5 networks show impressive average clean accuracies of  $89.62\%$ , and  $84.36\%$  on CIFAR-10, while the bottom-5 networks achieve significantly lower average clean accuracies of  $61.34\%$ ,  $51.64\%$  in standard training and adversarial training, respectively (Table 6). Moreover, the top-5 networks exhibit an average robust accuracy of  $54.95\%$  against perturbations, compared to only  $28.50\%$  for the bottom-5 group. Furthermore, by considering feature con
Type	Standard-Trained (ST)
Clean	CC.	PGD	FGSM	Avg. Rob.
Top-5	89.62	77.68	26.09	61.08	54.95
Bottom-5	61.34	49.13	19.35	17.01	28.50
Adversarily-Trained (AT)
Clean	PGD	HRS(PGD)	FGSM	HRS(FGSM)
Top-5	84.36	40.55	54.76	66.48	74.34
Bottom-5	51.64	22.59	31.36	37.20	43.04
Table 6: Comparisons of the performance.
sistency between clean and perturbed inputs, as defined in Eq. 2, the top-5 group exhibited similar features, whereas the bottom-5 group showed dissimilar features (Figure 6). This supports that architectures with consistent features across inputs are more likely to make accurate predictions under perturbations. Overall, our analysis provides strong evidence that our proxy can search for generalizable robust architectures that accurately predict the final accuracies under diverse perturbations.
While neural architecture search (NAS) is a powerful technique for automatically discovering high-performing deep learning models, previous NAS works suffer from two major drawbacks: computational inefficiency and compromised robustness against diverse perturbations, which hinder their applications in real-world scenarios with safety-critical applications. In this paper, we proposed a simple yet effective lightweight robust NAS framework that can rapidly search for well-generalized neural architectures against diverse tasks and perturbations. To this end, we proposed a novel consistency-based zero-cost proxy that evaluates the robustness of randomly initialized neural networks by measuring the consistency in their features, parameters, and gradients for both clean and perturbed inputs. Experimental results demonstrate the effectiveness of our approach in discovering well-generalized architectures across diverse search spaces, multiple datasets, and various types of perturbations, outperforming the baselines with significantly reduced search costs. Such simplicity and effectiveness of our approach open up new possibilities for automatically discovering high-performing models that are well-suited for safety-critical applications.
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-00153), the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00256259), the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD190031RD), and Google Research Grant and Google Cloud Research Credits program with the award (WLKY-E43Y-19JJ-A0E2). We thank Jin Myung Kwak, Eunji Ko, Yulmu Kim, Sohyun An, and Hayeon Lee for providing helpful feedback and suggestions in preparing an earlier version of the manuscript. We also thank the anonymous reviewers for their insightful comments and suggestions.
[36] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
[37] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6389, 2020.
[38] Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In ICML, pages 5025-5034. PMLR, 2018.
[39] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2020.
[40] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020.
[41] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, 2020.
[42] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. International Conference on Learning Representations, 2020.
[43] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, 2019.
[44] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2423-2432, 2018.
[45] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017.
[46] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.