Zero-cost Proxy for Adversarial Robustness Evaluation
Authors: Yuqi Feng
Yuwei Ou
Jiahao Fan
Yanan Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Architecture Search
Adversarial Robustness
Zero-cost Proxy
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Deep neural networks (DNNs) easily cause security issues due to the lack of adversarial robustness. An emerging research topic for this problem is to design adversarially robust architectures via neural architecture search (NAS), i.e., robust NAS. However, robust NAS needs to train numerous DNNs for robustness estimation, making the search process prohibitively expensive. In this paper, we propose a zero-cost proxy to evaluate the adversarial robustness without training. Specifically, the proposed zero-cost proxy formulates the upper bound of adversarial loss, which can directly reflect the adversarial robustness. The formulation involves only the initialized weights of DNNs, thus the training process is no longer needed. Moreover, we theoretically justify the validity of the proposed proxy based on the theory of neural tangent kernel and input loss landscape. Experimental results show that the proposed zero-cost proxy can bring more than $20\times$ speedup compared with the state-of-the-art robust NAS methods, while the searched architecture has superior robustness and transferability under white-box and black-box attacks. Furthermore, compared with the state-of-the-art zero-cost proxies, the calculation of the proposed method has the strongest correlation with adversarial robustness. Our source code is available at https://github.com/fyqsama/Robust_ZCP.

Published as a conference paper at ICLR 2025
ZERO-COST PROXY FOR ADVERSARIAL ROBUSTNESS EVALUATION
Yugi Feng, Yuwei Ou, Jiahao Fan, Yanan Sun' College of Computer Science, Sichuan University Feng770 62.3 3ma com; ouyuwel @stu. edu. cn; (fanjh, ysun ) @scu. bacu.edu.com en
ABSTRACT
Deep neural networks (DNNs) easily cause security issues due to the lack of ad- versarial robustness. An emerging research topic for this problem to design adversarially robust architectures via neural architecture search (NAS), i.e., robust NAS. However, robust NAS needs to train numerous DNNs for robustness estima- tion, making the search process prohibitively expensive. In this paper, we propose a zero-cost proxy to evaluate the adversarial robustness without training. Specifically, the proposed zero-cost proxy formulates the upper bound of adversarial loss, which can directly reflect the adversarial robustness. The formulation involves only the initialized weights of DNNs, thus the training process is no longer needed. More- over, we theoretically justify the validity of the proposed proxy based on the theory of neural tangent kernel and input loss landscape. Experimental results show that the proposed zero-cost proxy can bring more than 20x speedup compared with the oftteoof-hheart robust NAS methods, while the searched architecture has superior robustness and transferability under white-box and black-box attacks. Furthermore, compared with the state-of of-the-art zero-cost proxies, the calculation of the proposed method has the strongest correlation with adversarial robustness. Our source code is available https: //g/ //gtttttttta thub. com/ fyqsama/ RRobust_CCP.
INTRODUCTION
Deep neural networks (DNNs) are vulnerable to adversarial attacks (Szegedy et al., 2014), bringing potential hazards to security-critical systems that apply DNNs (Fykholt et al., 2018; Sharif et al., 2016). To address this problem, an emerging topic is to design adversarially robust DNNs via neural architecture search (robust NAS) (Ou et al., 2024). Current robust NAS methods need to evaluate the adversarial robustness of large number of DNNs select out the robust ones. However, the mainstream evaluation methods (e.g., training from scratch or weight-sharing technique (Pham et al., 2018)) are inefficient due to the training process of DNNs, and this problem becomes sharper when performing adversarial training to evaluate the adversarial robustness (Goodfellow et al., 2014). This because the adversarial examples are generated during adversarial training, making it much more time-consuming than the standard training which does not need to generate adversarial examples.
Recently, a few works (Lukasik et al., 2023; Ha et al., 2024) try to reduce the evaluation cost via zero- cost proxies (Abdelfattah et al., 2021). Specifically, zero-cost proxies surrogate tasks evaluating DNNs without training. Zero-cost proxies were initially designed for natural accuracy evaluation, but now few works also try use them for evaluating adversarial robustness. For example, Lukasik al. (Lukasik et al., 2023) show that some zero-cost proxies designed for natural accuracy evaluation can also be used for adversarial robustness evaluation. Ha et al. (Ha et al., 2024) design another zero-cost proxy, i.e., CRoZe, to evaluate the adversarial robustness. Although these works have taken the first step towards zero-cost proxies for adversarial robustness evaluation, they still require adversarial example generation and lack theoretical analysis, increasing the computational cost and making the users do not know why the proxies can reveal robustness. order to further accelerate the adversarial robustness evaluation with theoretical guarantee, we design a robust zero-cost proxy without adversarial example generation along with theoretical justifications in this paper.
"Coresponding author.

---

Published as a conference paper at ICLR 2025
Specifically, we propose zero-cost proxy that describes the upper bound of adversarial loss (i.e., loss on adversarial examples), which is negatively correlated to adversarial robustness. Meanwhile, the upper bound of adversarial loss is represented by the initialized weights of DNNs, and can be directly used to evaluate adversarial robustness without adversarial example generation. Furthermore, we mathematically prove the validity of the proposed zero-cost proxy based on the theory of neural langent kernel (NTK) (Jacot et al., 2018) and input loss landscape (Zhao et al., 2020).
In order to show the efficiency and effectiveness of the proposed zero-cost proxy, we perform the proposed zero-cost proxy along with NAS, searching for adversarially robust architectures. Surprisingly, when the proposed zero-cost proxy is adopted, there is a 20x speedup compared with the state-of tfrt robust NAS methods (Dong et al., 2020; Hosseini et al., 2021; Cheng et al., 2023), while the derived architectures are still competitive under both white-box and black-box attacks.
Moreover, we comprehensively analyze the correlation between the output of the proposed zero- cost proxy and the adversarial robustness on a recently constructed dataset (Jung et al., 2023), which we call JAS-Bench-201-R for convenience. NAS-Bench-201-R is the first to evaluate full NAS-Bench-201 search space (Dong & Yang. 2020) under various adversarial attacks with various parameters, and it can be well used for evaluating the performance of the zero-cost proxy. However, NAS-Bench-201-4 can only be used to validate the zero-cost proxy when attacks are weak, it does not work when attacks are strong. This is because their networks not adversarially trained, and their adversarial accuracy may drop to near zero when suffering strong attacks. Such data is not informative. Therefore, to evaluate the zero-cost proxy more comprehensively. called Tiny-RobutBeench, mapping adversarially trained networks to their adversarial accuracy. Experimental results on HAS-Bench-201-R show that the proposed zero-cost proxy performs well when the adversarial accuracy is practically meaningful (i.e., not drop to near zero). Furthermore, experimental results on Tiny-RobustBent show that the output of the proposed zero-cost proxy has the strongest correlation to adversarial robustness among state-of-the-aa zero-cost proxies.
construct dataset
Our contributions can be summarized as follows: 1) We propose a zero-cost proxy to evaluate the adversarial robustness without training, and the relationship between the proxy and adversarial robustness is theoretically proven. 2) We apply the proposed zero-cost proxy to NAS, and the search cost is dozens of times less than state-of-the-ar robust NAS methods, while the searched architecture is competitive in terms of white-box attacks, black-box attacks, and transferability. Massive outstanding architectures are analyzed to guide the future design of adversarially robust architectures. 3) We comprchensively evaluate the correlation between the output of the proposed zero-cost proxy and the adversarial robustness under different adversarial attacks in different search spaces. Moreover, novel dataset named Tiny- Tiny-RobustBench is open sourced to promote the development of this field.
2 RELATED WORK
2.1 ADVERSARIAL ATTACKS AND DEFENSES
According to whether or not attackers have full access to DNN parameters, adversarial attacks can be divided into white-box and black-box attacks. Common white-box attacks include FGSM (Goodfel- low et al., 2014), PGD (Madry et al., 2018). BIM (Kurakin et al., 2018), C&W (Carlini & Wagner, 2017), and so on. One commonly used black-box attack is the transfer-based attack (Papernot et al., 2017), which generates adversarial examples utilizing substitute DNN. Recently, AutoAttack (Croce & Hein, 2020), which is an ensemble of white-box and black-box attacks, becomes popular for fair adversarial robustness evaluation.
To defend DNNs from adversarial attacks, the adversarial defense techniques have emerged. One popular defense technique is adversarial training (Goodfellow et al., 2014). Other adversarial defense techniques include defensive distillation (Papernot et al., 2016), obfuscated gradients (Athalye et al., 2018), data compression (Dziugaite et al., 2016), and so on. In this paper, we apply the adversarial training for its effectiveness, and test the adversarial accuracy under the adversarial attacks.
2.2 NEURAL TANGENT KERNEL (NTK) AND ROBUSTNESS
NTK (Jacot et al., 2018) is defined as the Gram matrix of gradients, and can be used to analyze the convergence and generalization of the model. There have been some studies investigating the
2

---

Published as a conference paper at ICLR 2025
model robustness based the NTK theory. For example, Wu et al. (Wu et al., 2021) study the relationship between network width and adversarial robustness, and explain it based on NTK. Tsilivis et al. (Tsilivis & Kempe, 2022) study how to generate adversarial examples using NTK. Singla er al. (Singla et al., 2021) show that shift invariance may reduce adversarial robustness based on the NTK theory. Gao al. (Gao al., 2019) show some properties of adversarial training. which also proven based on NTK. However, these studies are still far from our work. We aim to conduct quantitative analysis to compare the adversarial robustness of different architectures, especially when different primitives (e.g., convolutional and pooling layers) and topologies (how to connect different layers) are involved, while previous studies could not.
2.3 ZERO-COST PROXY
Zero-cost proxy technique that evaluates DNNs using a substitute task requiring little compu- tational cost. For instance, Abdelfattah et al. (Abdelfattah et al., 2021) study a series of zero-cost proxies including grad norm, snip, grasp. synflow, fisher, and jacob cov (Mellor al., 2021). Zhang et al. (Zhang & Jia, 2021) propose GradSign, which predicts the DNN performance through local optima density. Recently, some advanced zero-cost proxies have attracted much attention for their strong interpretability. Their correlations to the final performance of DNNs are mathematically proven, based on the theory of NTK. For example, Xu et al. (Xu et al., 2021) 1) prove that the mean of the NTK matrix is negatively correlated to the training loss, and propose MGM as a zero-cost proxy. Shu et al. (Shu et al., 2022a) also derive the upper bound of training loss based on NTK and use it as zero-cost proxy. Besides, Shu et al. (Shu et al., 2022b) develop a novel framework that consistently boosts existing zero-cost proxies.
The aforementioned zero-cost proxies are designed for the prediction of natural accuracy. Recently, Lukasik et al. (Lukasik et al., 2023) show that these zero-cost proxies can also be used for adversarial robustness evaluation. Ha et al. (Ha et al., 2024) design another zero-cost proxy, named CRoZe, to evaluate the robustness of architectures against diverse perturbations. Although these works have taken the first step towards zero-cost proxies for adversarial robustness evaluation, they still need to costly generate adversarial examples and do not provide theoretical analysis about why their proxies can reveal robustness. In this paper, we aim design zero-cost proxy which is theoretically guaranteed and no longer needs adversarial example generation, thus the efficiency is further achieved.
3
METHODOLOGY
3.1 PRELIMINARIES
We present the preliminaries of NTK and input loss landscape, which form the basis of the proposed zero-cost proxy.
Proposition 1 (Xu et al. (2021)). Suppose fo, RRd + R defines DNN, with its weight parameters at the 1-th epoch. Suppose x and y denote image samples and the corresponding label vector, respectively. For any t. 0, Eq. (1) holds:
ly fe. (x) S <exp(-Amit (o,))
(=)
(1)
where Amin (Oo.) (m.. of two gradient vectors with different input samples x and x', formulated as Eq. (2):
is the minimum eigenvalue of the NTK Oo,- The NTK is essentially the dot product
Oo, (I, x') (Vofe, (x')),
(2)
where Vefe, means to the Jacobian of DNN prediction.
Mok et al. (Mok et al., 2022) show that Eq. (1) can further transformed to Eq. (3) because the NTK remains constant regardless of training under some assumptions:
ly fo. 11) < exp(-Amin )t) Ily- - fo, (1.
The assumptions involved in this transformation are detailed in Appendix A.I.
Proposition (Zhao et al. (2020). Suppose fo defines a DNN. with its weight parameters 9. Suppose ()) denotes the classification loss with weight parameters 0 and input Consider the oracle
3

---

Published as a conference paper at ICLR 2025
adversarial loss maz ( 6) of the model t the path connecting two independently trained models, where denotes a perturbation to x confined by an e-ball induced by a vector norm
Il Let denote the normalized inner product in absolute value for the largest eigenvector of the
input Hessian and V. (=))(((),,))) i.e., v, =
Then under some assumptions, we have
(a 6) Amaz (H) as
3.2 DESIGN OF THE ZERO-COST PROXY
In this section, we present the proposed zero-cost proxy for adversarial robustness evaluation, and the theoretical analysis is provided in the next section. The design of zero-cost proxy for adversarial robustness evaluation would be more challenging compared with designing a zero-cost proxy for the natural accuracy evaluation, because the adversarial robustness is dependent the natural accuracy and requires extra adversarial robustness-relate calculation.
The proposed zero-cost proxy is formulated as Eq. (4):
M
N
N fop (o:))
0 Oo, ff (T) x
1(x + h2*) 1(x)
R - expl
Σ
MN m=
00
h
i=1
where R is the calculation of the zero-cost proxy, M is the number of layers, N is the number of data samples, 8t is the sampled parameters from the m-th layer, [(x) = (a x), 2"
sign(V. [=.gv(V:.(bbb=]]]] and l controls the scale of the loss landscape on which the smoothness is induced.
Eq. (4) can be directly used efficiently evaluate the adversarial robustness of DNNs. The efficiency mainly comes from two aspects. First, the proposed zero-cost proxy only needs to iterate samples instead of generating adversarial examples. This is because the NTK is calculated only based on different samples and x' as shown in Eq. (2). These samples can be obtained by simply iterating samples batch, thus there is no need to generate adversarial examples. Consequently, the proposed zero-cost proxy is computational efficient, because the repetitive calculation for the gradient or the time-consuming optimization for the adversarial example generation are no longer needed. Second, the proposed zero-cost proxy only needs the initial weights On of the DNN. Because there is no training process included, the efficiency further guaranteed.
3.3 THEORETICAL ANALYSIS
In this section, we carry out the theoretical analysis to justify the validity of the proposed zero-cost proxy. Specifically, our motivation comes from the upper bound of adversarial loss. This is because the upper bound of adversarial loss negatively correlated to adversarial robustness, and can be used as proxy for adversarial robustness evaluation. From this perspective, consider the upper bound of adversarial loss |yy - fe, (I) where E denotes the adversarial examples generated using v. By replacing x in Eq. (3) with 2, Eq. (5) holds:
ll - ( < (f,,)))mn(ee) (
(5)
where θ,, denotes the NTK matrix computed by adversarial examples and a' generated using x and x'. Similar to Eq. (2), the NTK matrix computed by adversarial examples can be formulated as Eq. (6):
(8, (Vofo, (+), (T')
(6)
Eq. (5) shows that the final adversarial loss is upper bounded by the right term, which is negatively correlated to adversarial robustness and can be used for adversarial robustness evaluation. However, both the minimum eigenvalue the NTK matrix Amin (0p,) ) and the adversarial loss initialization lly- feo (2)1l3 in the right term require the generation of adversarial examples ê for further calculation, which is computationally expensive and violates the principle of "zero-cost".. To avoid the generation of adversarial examples, we are committed to finding two substitutes of Amsn (θ,,) and ly - fo, (+)
For the substitute of the adversarial loss initialization || fe, (.) Proposition gives feasible solution, i.c., lly fo. (&)ll ~ Xmna (x)). The validity of this solution is discussed in detail in Section 3.5. For higher efficiency, following previous research (Mok et al., 2021), we approximate
4

---

Published as a conference paper at ICLR 2025
Amaz (Ho, (x)) as Eq. (7):
1(x + hz*) 1(x)
Amaz Hgg (x)) 20
h
where 1(x) V (10 =). 2"
signl V ((). [|sig|[ (().
and h controls the scale of the loss landscape on
which the smoothness is induced.
As for the substitute of the minimum eigenvalue the NTK matrix A ). we find that Amin ( is positively correlated to natural accuracy but negatively correlated to adversarial ac- curacy. This could be explained by the competing relationship between both indicators ( (Tsipras et al., 2018; Zhang al., 2019). More discussion and theoretical analysis about this phenomenon can be found Section 4.2.3 and Appendix A.2. Therefore, we simply use ( to replace
-Amin (0o,), which avoids the generation of adversarial examples. Meanwhile, following previous research (Xu et al., 2021), we calculate Amin (9o,) via Eq. (8) for higher efficiency:
1
N ΣVΣ 1=1
M
fea (z:)) afe, (f))(11)
Amin (@v.)
(8)
MN
300
303
where M is the number of network layers, and N is the number of data samples. 0th is the sampled parameters from the mn-th layer where the length of (th is 50 in implementation.
in Eq. (5) with Amaz (Ho, (z)) and Amin (Ge, formulated by Eq. (7) and Eq. (8), respectively. Mcanwhile, we add minus sign before the right term of Eq. (5) to make it positively correlated to adversarial robustness, which consistent with most zero-cost proxies. Consequently, we obtain Eq. (4) provided in the previous subsection, which can be used to evaluate the adversarial robustness of DNNs.
Finally, we replace lyy fe, (2)1l2 and Amin (Oo,) Am
3.4 COMPUTATIONAL COMPLEXITY ANALYSIS
The computational complexity of the proposed zero-cost proxy is determined by the number of both layers and data samples. Given the architecture with AM layers, because our proxy needs to traverse all layers once in the evaluation, the computational complexity for this step is ( Meanwhile, because our proxy requires two traversals the data for the calculation of each layer. Considering the number of data samples is N, the computational complexity for this step is O According the above analysis, the overall computational complexity of the proposed zero-cost proxy is OMM
3.5 DISCUSSION
In this section, will justify the validation of adopting the largest eigenvalue Xmaz (Hop (x)) to approximate the upper bound of adversarial loss. As empirically stated by Moosavi et al. (Moosavi- Dezfooli al., 2019), in Proposition 2 is equal to 0.43 before adversarial fine-tuning, and increases to 0.90 after the adversarial fine-tuning on CIFAR-10. 10. It is demonstrated that the condition where is approaching 1.0 can be approximately satisfied in practice, especially for the architectures adversarially trained. As result, adopting the largest eigenvalue Amaz (Ho, (x)) to approximate the upper bound of adversarial loss practical.
EXPERIMENTS
We study the following research questions:
RQI: Is the proposed zero-cost proxy superior in robust accuracy and cost-efficiency?
RQ2: Is the proposed zero-cost proxy superior in evaluating adversarial robustness of architectures?
4.1 RQI: SUPERIORITY IN TERMS OF ROBUST ACCURACY AND COST-EFFICIENCY
4.1.1
EXPERIMENTAL. SETTINGS
Benchmark Datasets Following the conventions of the robust NAS community (Guo et al., 2020; Mok et al., 2021; Ou et al., 2024), CIFAR-10 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al.,
5

---

Published as a conference paper at ICLR 2025
2011), Imageee (Le & Yang, 2015), and ImageNet (Deng et al., 2009) are used as benchmark datasets.
Peer Competitors We choose architectures commonly used in the robust NAS community (Mok et 2021; Ou et al., 2024) as our peer competitors, including ResNet-18 18 (He et al., 2016), DenseNet- 121 (Huang et al., 2017), DARTS (Liu et al., 2019), PDARTS (Chen et al., 2019). RobNet-free (Guo et al., 2020), RACL (Dong et al., 2020), DSRNA (Hosseini et al., 2021), and WsrNet (Cheng et al., 2023). Besides, we also choose some training-free NAS methods, including GradNorm (Chen et al., 2018), SynFlow (Tanaka al., 2020), and CRoZe (Ha et al., 2024). The experimental results of these training-free free NAS methods are cited from reference (Ha et al., 2024).
Parameter Settings Neural architectures are stacked by cells, with an initial channel number of 36. After the search, the best architecture is adversarially trained. Following advanced settings in the robust NAS community (Mok et al., 2021; Ou et al., 2024), the adversarial training is performed using seven-step PGD with a step size of 0.01 and total perturbation of 8/255. SGD is used to optimize networks, with the momentum of 0.9 and the weight decay of 1 x 10 The learning rate is set to 0.1 initially, and decayed by factor of 0.1 at the 100-th epoch. The batch size is set to 64. The search cost is measured by GPÚ days (number of GPUs used x total running time (days)) using the NVIDIA RTX 2080Ti GPU.
4.1.2 PERFORMANCE UNDER WHITE-BOX ATTACKS
To apply the proposed zero-cost proxy to NAS and explore its performance under white-box attacks, construct a simple NAS algorithm, randomly sampling 1,000 architectures from the DARTS search space, and adopting the architecture with the highest calculated score as our search result. The searched architecture is adversarially trained and evaluated under several white-box attacks. The results are shown in Table 1. The best and second- best results are in bold and underlined, respectively.
Evalantion results of adversarially trained models CIBAR- under white box
Category
Model
With Training?
Parses (M)
FL.OPs
(M)
Natural Aoc. (96)
PGSM
(96
PGT"1 (9)
PGT
(95
PCTT
(96
APCIDeR
(96
(%)
AA
Search Cost
(GPU Days)
Hand-Crrfted
ResNeri 18 DonseNet-121 121
11.2
7.0
59.83
37.67
85.95%
$4.09%
58, 46%
54.64%
50.49%
45.36%
49.929)
45.53%
44,54%
40.11%
47.46%
43.22%
DARTS PDARTS
3.3
547.44
K5 17%
56 74%
50.45%
40.78%
48.32%
46 7979
1.0
Sundard NAS
GradNomaa SysFlow
3.4
550.75
85.37%
59.12%
49
51.32%
50.91%
49.96%
48.52%
696
0.3
0.1
5.1
77.08%
45.95%
43.45%
0.1
RobNert-fee
V
5.6
800.40
85%00%
99 2256
$2.00%
51 14%
50.41
48. 56%
>3.3
RACL
3.6
568.80
83.97%
59 2913
$2.18%
$1.22%
51.25%
43,591
0.5
Robust NAS
WarNer
DSRNA
V
2.0
3.0
336.25
454,30
80.93%
13.94%
$1.19%
12%
47.17%
19.15%
46.61%
48 13.89%
48.54%
44.87%
43.91%
4.0
0.4
CRoiee
28%
$2.17% 09.21%
48.1445
0.2
3.4
555.54
60.20%
52.75%
52.51%
52.25%
49.97%
0.017
reporred
toe prosided
acconding peinciplos. Spectiiallly estimate loqger 3.3 GPU
shoc metlod
whea gzsensting alvessaia eumple eviluation
metiod (Bender 2015).
The expetinctall soults thou trssingg-fee NAS malbeak an cied frum roforma: (fl 2004 and ihe experimandd asting of PGD" aula felloweal with the total pertartation scale 57255 and the size of 5/2550 (wbich is differem tiem PUD*) and PGD Symbol 'n' means unmessured.
As can be observed, the proposed method achieves the second best natural accuracy. Comparing the adversarial accuracy, the proposed method surpasses other peer com- petitors under simple attacks (e.g., FGSM and PGD). To our surprise, the adversar- ial accuracy of the proposed method also appears to be the best among all com- petitors under some stronger attacks (e.g., APGDCE and AA), which demonstrates the effectiveness of the proposed zero-cost proxy for adversarial robustness evaluation. The biggest highlight of the proposed method lies in its search efficiency, which is at least 20 times higher than previous methods that require training, and also about at least 6 times higher than previous training- -fee methods. Additional experimental results under white-box attacks can be found in Appendix B. and B.2.
Table 2: Results of adversarially trained models on CIFAR-10 under transfer- based black- attacks.
Target
DSRNA
RACL
AdvRush
Ours
Cost
Source
DSRNA
64.96%
66.86%
64.409
0.4
RCAL
62.37%
66.37%
64.77%
0.5
AdvRush
62.26%
64.22%
64.24%
0.7
Ours
61.24%
64.19%
66.12%
0.017

---

Published as a conference paper at ICLR 2025
4.1.3 PERFORMANCE UNDER BLACK-BOX ATTACKS
To explore the performance of the searched architecture under black-box attacks, we perform transfer- based attacks, attacking target models using adversarial examples generated by source models. The results are shown in Table 2. In each column, when our architecture is used as source model (in bold), the accuracy of its target model is always the lowest, indicating that our architecture generates the strongest transfer-based black-box attacks. When considering each model pair, we can further compare the adversarial robustness between every two models. For example, when considering the model pair Ours ++ DSRNA, Ours + DSRNA achieves the attack success rate (i.e., 100% adversarial accuracy) of 76%, while DSRNA - Ours achieves the attack success rate of 35.60% indicating that our architecture has stronger adversarial robustness under the black-box attack. Similarly, our architecture has stronger adversarial robustness to transfer- based black- box attacks than RACL. Additional experimental results under black- box settings can be found in Appendix B.3.
4.1.4 TRANSFFRABILITT TO OTHER DATASETS
The architecture searched on CIFAR-
10 is transferred to SVHN and Tiny- ImageNet-200 to show its transfer- ability. The results are shown in Ta- ble 3. When transferred to SVHN, our architecture performs best un- der all metrics. When transferred to Tiny-ImageNet-2000 it demonstrates a
Table 3: Evaluation results of adversarially trained models on SVHN and Eny-ImageNet-200
Datasets
Model
Natural
PGD
R8 PDARTS
92.06%
88.73%
69.51%.
SVHN
95.10%
93.01% %
89.58
Ours
95.79%
95.14%
91.64%
significant improvement. Compared with ResNet-18, 18, our architecture is far ahead with 17.6 17.66% higher natural ac- curacy, 16. 16.73% higher FGSM accu- racy, and 1.78% higher PGD accuracy. Compared with PDARTS, the natural and FGSM accuracy of our architecture are also significantly higher, though its PGD accuracy is lower. Additional experimental results and analysis regarding the transferability can be found in Appendix B.4.
ResNet- 18
36 36.26%
16.08%
13.94%
Tiny-ImageNet
PDARTS
45.94%%
24.36%
22.74%
Ours
53.92%
32.81 %
15.72%
4.1.5 PERFORMANCE ON THE LARGER-SCALE DATASET
In order to evaluate the proposed zero-cost proxy on the larger-scale dataset, we have performed experi- ments on ImageNet (Deng et al., 2009) and compared the proposed zero-cost proxy with ofate-of-the-art robust zero- cost proxy (i.e., CRoZe (Ha et al., 2024)) and robust NAS methods (i.e., RACL (Dong et al., 2020) and Ad- vRush (Mok et al., 2021)). To perform the evaluation, tures derived by both proxies for 30 epochs, with the learning rate decayed by the factor of 0.1 at 20-th and 25-th epochs. Besides, because the vanilla adversarial training on ImageNet will cost thousands of GPU days, we used the fast adversarial training technique (Wong et al., 2020) for acceleration. The experimental results are presented in Table 4. It clearly observed that the proposed zero-cost proxy demonstrates the best performance in terms of natural accuracy and accuracy under adversarial attacks. The results indicate that the proposed zero-cost proxy is still effective on larger-scale dataset, further demonstrating the effectiveness of the proposed zero-cost proxy.
Table 4: Experimental results of the proposed zero-cost proxy and state of the arts on ImageNet.
With Training?
Natural Acc. (%)
FGSM (%)
PGD
Methods
(%)
RACL (Dong al., 2020) AdvRush (Mok et al., 2021)
$1,59%.
18.15%
10.49%
51.54%
18.42%
10.74%
CRoZe (Ha al., 2024)
49.95%
16.54%
9.67%
Ours
52.71%
19.88%
11.96%
train the architec-
4.1.6 PERFORMANCE IN LARGER SEARCH SPACE
In addition the results in DARTS search space (Liu al., 2019), we also perform experiments in WideResNet (WRN)-like search space (Zagoruyko, 2016; Li et al., 2021), to show the effectiveness of the proposed zero-cost proxy the more complex search space. The results are presented Table
7

---

Published as a conference paper at ICLR 2025
As can be seen from Table 5, the proposed zero-cost proxy can still achieve the best performance in the WRN-like search space, along with the lowest search cost. Although the search space becomes larger and more complex, the search cost of our proxy just shows slightly increment compared with that in DARTS search space (from 0.017 GPU days to 0.019 GPU days). This phenomenon further demonstrates the efficiency of the proposed zero-cost proxy. More experimental results about the computational cost and the scalability can be found in Appendix B.5 and B.6, respectively.
4.1.7 ABLATION STUDY OF THE PROPOSED CERO-COST PROXY
To show the individual role of the pro- posed zero-cost proxy in robust NAS, we compare it with the mainstream weight-sharing technique adopted by peer competitors. Specifically, we ad- versarially train super-networs of the DARTS search space, evaluate ran-
Table 5: Experimental results in WRN-like search space. The
derived architectures are adversarially trained on CIFAR-10.
Natural
FGSM
PGD
Search Cost
Models
Acc. (%)
(9)
(%)
(GPU Days)
NADAR (Li et,,, 2021)
86.23% 87.34%
60.46%
53.43%
0.5 0.019
domly sampled architectures using the
Ours
61.93%
53.89%
shared weights, and train the best ar- chitecture from scratch. The results are shown in Table 6.
The total cost (cost of training super- network + search cost) of the pro- posed zero-cost proxy is 500 times less than the weight-sharing techmique. Meanwhile, given that both methods have their own advantages in terms of natural accuracy and adversarial accuracy, we use Harmonic Robust- ness Score (HRS) (Devaguptapu et al., 2021) for further comparison. HRS is recently- introduced metric that considers both natural accuracy and adversarial accuracy. The higher the HRS, the better the architecture. The proposed zero-cost proxy gets higher HRS score, which means the architecture searched by the proposed zero-cost proxy is better.
Table 6: Ablation study of the proposed zero-cost proxy. The cost is measured in GPU Days.
Performance
Estimation
Cost of Training
Super-network
Search
Cost
Natural
Acc.
PGD
HRS
Weight-sharing
84.34%
The proposed
0.0
0.017
83.67%
52.00%
64.14
zero-cost proxy
4.1.8 ARCHITECTURE INGREDIENTS OF ADVERSARIALLY ROBUST NEURAL NETWORKS
To analyze the architecture ingredients of adversarially robust networks, we randomly sample 1,000 architectures from the DARTS search space and obtain the top 100 architectures according to their scores. The statistical data is shown in Figure 1. On the whole, most architectures in the top 100 tend to employ a large number of learnable primitives, but few primitives without learnable parameters. Under this overall impression, we further analyze each trend line in detail.
When analyzing the trend line correspond- ing to skip connections, we learn that most top 100 architectures are without skip con- nections, and the number of architectures decreases dramatically as the number of skip connections increases. This is a spe-
Number of Neural
80
avo__ool_ ____pp0__333 3x3
--x dil_conv_5+5
max_pool_3x3
dil_onw_333
40
sep_conv_5es
wcppsooy..vccccooo 3x3
20
skip connection
cial case in which the proposed proxy may
misjudge the adversarial robustness. Com- monly, the architectures without skip con- nections will not perform well in terms of adversarial robustness, because they may meet with training difficulties such as gra- dient vanishing (Glorot& & Bengio, 2010). The proposed method may misjudge such neural architectures as good ones, because it evaluates the adversarial robustness of neural networks through their weights at initialization but ignores the effect of skip connections.
3
8
Number of Specified Primitives Normal Cell
Figure 1: Analysis on the architecture ingredients of Top-100 neural architectures.
8

---

Published as a conference paper at ICLR 2025
Similarly, both 3 x 3 max pooling and average pooling have similar tendencies to skip connections, indicating that they have little effect on the adversarial robustness. Differently, as the horizontal value gets large, the trend lines corresponding to learnable primitives first rise and then fall, indicating that all types of learnable primitives contribute to adversarial robustness. But the trend line of 3 x dilated convolutions falls earlier, indicating that robust architectures prefer x 5 dilated convolutions, 3 x 3 separable convolutions, and 5 x 5 separable convolutions more than x 3 dilated convolutions. Furthermore, more analysis of architecture ingredients are placed in Appendix B.7.
4.2
RQ2: SUPERIORITY IN TERMS OF EVALUATING ADVERSARIAL. ROBUSTNESS
4.2.1
EXPERIMENTAL SETTINGS
Datasets NAS-Bench-201-R (Jung et al., 2023): NAS-Bench--20-R is the first test all pretrained NAS-Bench-201 models (containing 6,466 unique architectures) under various adversarial attacks, i... FGSM. PGD, APGD, and Square, with various attack parameters. It is effective evaluate zero-cost proxies under weak attacks. However, it cannot work when attacks are strong, because their models are not adversarially trained, resulting in near zero accuracies when suffering strong adversarial attacks. Such data is not informative.
Roy-tRobsttench: To effectively evaluate zero-cost proxies when the attacks are strong, we create a dataset called Bon-Roobut,ench, which maps 223 adversarially trained networks to their adversarial accuracy. The neural architectures are sampled from the DARTS search space (Liu et al., 2019), which is the most popular one in the robust NAS community (Guo al., 2020; Mok et al., 2021). The adversarial training method also follows the advanced settings in the robust NAS community (Mok et al., 2021: Ou et al., 2024). Please note that acquiring the dataset is expensive, because the adversarial training method with the advanced settings takes more than 10 times as long as the standard training, which may be obstacle to relevant researchers. We believe that creating such dataset is also one of our contributions, and the dataset is open-sourced for further research. More details of RonyyRobuttBenc
be found in Appendix C.I.
Peer Competitors We take state-of-the-ar zero-cost proxies as peer competitors, i.e., grad
grad_norm norm (Chen et al., 2018), snip (Lee et al., 2018), grasp (Wang et al., 2020), synflow (Tanaka et al., 2020), fisher (Turner al., 2019), jacob_ccoroooooaaa (Mellor al., 2021), and MGM (Xu et al., 2021).
Parameter Settings Our zero-cost proxy has only five parameters in Eq. (4), Following the settings in MGM (Xu et al., 2021), the number of layers M and the number of data samples N are and 25, respectively. is set to 50. t that represents the prediction epoch is set to 5 x 109, which plays the role of balancing the two parts divided by the multiple sign in Eq. (4) in practice. The batch size is set to 8. The parameter studies are presented in Appendix C.2.
11
4.2.2 CORRELATION TO VARIOUS ADVERSARIAL ACCURACY ON
We calculate Kendall's Tau correla- tion of zero-cost proxy NAS- Bench-201-R. The results are shown in Figure The proposed zero-cost proxy is positively correlated to the adversarial accuracy of these four at- tacks (r = 0.38) when the attacks are weak (e 0.1 0.1/255). When the at- tacks get stronger (e gets larger), gets larger under the FGSM attack, in- dicating the proposed zero-cost proxy makes more accurate evaluations un- der stronger FGSM attacks. How- ever, the correlations become slightly weaker under stronger PGD attacks, and even vanish under Square attacks and APGD attacks.
Allece Mechool
FCOM
333
0/41
0:15
0.47
0:41
0.4
PGD
3331
021
0.17
1026
03
0.32
025
0.3
0.2
Square
233
0.2
Total Parturbation Scale
Figure Kendall's Tau correlation coefficient of the pro- posed zero-cost proxy under various attacks with various parameters in the MAS-Bennh-2011 Dataset. Symbol e rep- resents the total perturbation scale of these attacks.
This phenomenon not in line with practice. According to recent research (Mok et al., 2021; Ou al., 2024) well as practice, networks performing well under one attack tend to keep

---

Published as a conference paper at ICLR 2025
outstanding under other attacks (though not absolute). It curious that the proposed zero-cost proxy shows positive correlations to FGSM attacks but no correlation to Square and APGD attacks. We conjecture this is because the networks in the NAS-Bench-201--R dataset are not adversarially trained, so they generate invalid predictions when attacks get stronger. For example, under PGD) attacks with 8.0/255, the adversarial accuracy of these networks is near zero, and the situation worsens under Square and APGD. Consequently, we recommend testing correlations with adversarially trained networks using the proposed Tiny-RobustBence dataset, which also line with our practical needs.
4.2.3 CORRELATION ADVERSARIAL ACCURACY OF ADVERSARIALLY TRAINED NETWORKS IN BONSTROBUTT DATASET
To show the effectiveness of the proposed zero- cost proxy in evaluating adversarially trained networks, we evaluate all networks in the Tiny- RobustBench dataset using the proposed proxy and obtain their score rankings. The results are visualized in Figure 3. Figure 3a is scatter plot,
41.5
80.5
where each point represents a network corre-
sponding to its score ranking and its adversarial accuracy, and the solid blue line is the result of linear regression. From the solid blue line, we can see that the larger the score ranking (repre- senting a lower score), the lower the adversarial accuracy. Figure 3b is a histogram, where 223 networks are classified into four groups accord- ing to their score rankings, and each bar repre- sents their average adversarial accuracy. This figure shows that the average adversarial accuracy decreases when the score becomes lower. From these two figures, we come to the same conclusion that the proposed zero-cost proxy is positively correlated to adversarial robustness, which is in line with the theory during the design.
50
-00
150
200
165
Score Ranking
Score Raniking
(a) Scatter plot
(b) Histogram
Figure 3: Visualization of the correlation between
the score rankings and adversarial accuracy.
4.2.4 COMPARISON WITH PREVIOUS ZERO-COST PROXIES
To show the superiority of our zero-cost proxy. we compare it with chosen peer competitors, and the re- sults are visualized in Figure 4. Our proxy reaches the highest Kendall's Tau correlation coefficients, indi- cating the proposed proxy is most correlated to adver- sarial robustness. Interestingly, MGM was originally designed to be positively correlated to natural accu- racy, but now shows the strongest negative corre latioation to adversarial accuracy. This is because MGM is de- signed according to the upper bound of the natural loss, while the natural loss always conflicts with the adversarial loss (Tsipras et al., 2018). Furthermore, more comparisons between the proposed zero-cost proxy and MGM are presented in Appendix C.3.
(Ours)
Proxics
0:33
Kencall's Tau
0.23
0.15
0.15
0.10
-001-0.
0.29
0.0
great norm
unga
grasp
fisheer
Jacib day
Synffo
-0.1
Figure 4: Kendall's Tau correlation coef- ficient of zero-cost proxies in the Tiny- RobustBench dataset.
CONCLUSION
In this work, we recognize that existing techniques for adversarial robustness evaluation are computa- tionally expensive. To address the problem, we design a zero-cost proxy, evaluating the adversarial robustness of DNNs according to their initialized weights without training. The proposed zero-cost proxy is shown to be effective to NAS, descending the search cost dozens of times less than state-of- the-art robust NAS methods. Meanwhile, the proposed zero-cost proxy shows the strongest correlation to adversarial robustness among existing zero-cost proxies. In addition, we also open- a dataset promote the development of the community.
called Tiny-RobustBent
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grant 62276175 and Innovative Research Group Program of Natural Science Foundation of Sichuan Province under Grant 2024NSFTD0035.
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight NAS. International Conference on Learning Representations, 2021. URL https::/opeereviie net
/ /pdf 1aaiffdddd00MyMy--- 0cmMMy8JJ5g
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283. PMLR, 2018.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning, pp. 550-559. PMLR, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pp. 39-57. IEEE, 2017.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1294 1303, 2019.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning. pp. 794 803. PMLR, 2018
Zhi Cheng, Yanxi Li, Minjing Dong. Xiu Su, Shan You, and Chang Xu. Neural architecture search for wide spectrum adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 442-451, 2023.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference Machine Learning, pp. 2206-2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Chaitanya Devaguptapu, Devansh Agarwal, Gaurav Mittal, Pulkit Gopalani, and Vineeth Balasub- ramanian. On adversarial robustness: A neural architecture search perspective. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 152-161,20 2021.
Minjing Dong, Yanxi Li, Yunhe Wang, and Chang Xu. Adversarially robust neural architectures.
arXiv preprint arXiv:2009.00900 .0000,000020000 2020.
Xuanyi Dong and Yi Yang. NAS-Bench-201. Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2020. URL https:// openreview net
Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A study of the effect of JPG compression adversarial images. arXiv preprint arXiv: 1608000808.0 2016.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-wodd attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
11

---

Published as a conference paper at ICLR 2025
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason D Lee. Convergence of adversarial training in overparametrized neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, arXiv preprint 1412::442.66572 2014.
Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, and Dahua Lin. When NAS meets robustness:
In search of robust architectures against adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 631-6410 2020.
Hyeonjeong Ha, Minseon Kim, and Sung Hwang. Generalizable lightweight proxy for robust nas against diverse perturbations. Advances in Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Ramtin Hosseini. Xingyi Yang, and Pengtao Xie. DSRNA: Differentiable search of robust neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6196-6200 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Steffen Jung, Jovita Lukasik, and Margret Keuper. Neural architecture design and robustness: A dataset. In International Conference on Learning Representations, 2023. URL https: openreviees .et /ptd 2ad=ptdd11llllg1
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
URL http: /www cs www.rroottt..u.krroooooee
far .tml..
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pp. 912112. Chapman and Hall/CRC, 2018.
Ya Le and Xuan Yang. Tiny ImageNet visual recognition challenge. CS 231N. 7(7):3, 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv: 1FX0.0800000234. 2018.
Yanxi Li, Zhaohui Yang, Yunhe Wang, and Chang Xu. Neural architecture dilation for adversarial
robustness. Advances Neural Information Processing Systems, 34:29578-22 295599 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search.
International Conference on Learning Representations, 2019. URL https: //pper/iiee......e.
Jovita Lukasik, Michael Moeller, and Margret Keuper. An evaluation of zero-cost proxies-from neural architecture performance to model robustness. arXiv preprint arXiv:2307.09335, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks. The 6th International Conference on Learning Representations, 2018. URL https: hwtp///ppeeeeeeeeeee/ee/eeeeeeee net ppd::idddJiIBBeaa....
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Jisoo Mok, Byunggook Na, Hyeokjun Choe, and Sungroh Yoon. AdvRush: Searching for adversar- ially robust neural architectures. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp.
2021.
Jisoo Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from a practical perspective: Can it be trusted for neural architecture search without training? In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 11861-1877, 2022.
Seyed-Mohsen Moosavi- -ezaol,,,, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http:://tttleeeeee stanford. edu/ housenumbers nips2011111 www.hhuuuuuuubbeee pdf.
Yuwei Ou, Yuqi Feng, and Yanan Sun. Towards accurate and robust architectures via neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5967-5976, 2024.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, pp. 987. IEFE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506-519, 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095-4104 PMLR, 2018.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1528-1540, 2016.
Yao Shu, Shaofeng Cai,, Zhongxiang Dai, Beng Chin Ooi, and Bryan Kian Hsiang Low. NASI: Label- and data-agoostic neural architecture search at initialization. In International Conference on Learn- ing Representations, 2022a. URL https: http////ppeeeeeeeeee/eeeeeeeeeet net http://wwwwee/oooo.mddeeee. http:/orondddd-iiiloede
v.
Yao Shu. Zhongxiang Dai. Zhaoxuan Wu, and Bryan Kian Hsiang Low. Unifying and boosting gradient- training-free free neural architecture search. Advances in Neural Information Processing Systems, 35:33001-33015, 2022b.
Vasu Singla, Songwei Ge, Basri Ronen, and David Jacobs. Shift invariance reduce adversrria robustness. Advances in Neural Information Processing Systems, 34: 14:18881187,, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. URL https: arxiv. org/ /ttp:///////// 1112. 6199. pdf.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks with- out any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 0166777-3899 2020.
Nikolaos Tsilivis and Julia Kempe. What can the neural tangent kernel tell us about adversarial robustness? Advances in Neural Information Processing Systems, 35:181 16-18130. 2022.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference on Learning Representations, 2018. URL https: pectts::/oppeeee http///ppp///veeeeeeee www.weeweeeeeeee net wwte///ftttteeeeee
13

---

Published as a conference paper at ICLR 2025
Jack Turner, Elliot Crowley, Michael O O'Boyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher- guided block substitution for network compression on a budget. arXiv preprint PrXiv:1906.04110 2019.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 2020.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL https: http:///pppeeeeeeeeee
net
Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, and Quanquan Gu. Do wider neural networks really help adversarial robustness? Advances in Neural Information Processing Systems, 34:7054-7067, 2021.
Jingjin Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. KNAS: green neural architecture search. In International Conference on Machine Learning, pp. 11613-11625. PMLR, 2021.
Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv: 1605.07144 2016.
A Zela, T Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Hutter. Understanding and robustifying differentiable architecture search. In International Conference Learning Representations, 2020.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Zhihao Zhang and Zhihao Jia. Gradsign: Model performance inference with theoretical insights. arXiv preprint arXiv:21 10.08616, 2021.
Pu Zhao, Pin-Ye Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. International Conference on Learning Representations, 2020. URL https: / arxiv. org/ pdf 2ttp//005.000000 pdf.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
The appendix contains additional detailed information not covered the main article. The organiza- tion of the appendix is presented follows:
Appendix A: This section contains additional theoretical analysis of the proposed zero-cost
proxy.
Appendix B: This section contains additional experimental results for RQ1
Appendix C: This section contains additional experimental results for RQ2.
Appendix D: Some visualizations of the neural architectures derived by the NAS method along with the proposed zero-cost proxy.
A ADDITIONAL THEORETICAL ANALYSIS OF THE PROPOSED ZERO-COST PROXY
A.1 ASSUMPTIONS OF THE PROPOSED ZERO-COST PROXY
In this section, discuss the assumption in terms of the infinite-width DNN parameters. In partieular, when transferring Eq. (1) to Eq. (3), the assumption of infinite-width DNN parameters valid. As evidenced by the previous literature Jacot al. (2018), in the infinite-width limit, the NTK becomes deterministic at initialization and stays constant during training. Consequently, when replacing O, Eq. (1) with Qg, in Eq. (3), because the invariance NTK in the infinite-width limit, these two NTKs can be directly replaced by each other. Therefore, the transformation from Eq. (1) to Eq. (3) keeps valid for the infinite-widd width DNN parameters.
A.2 VALIDITY OF APPROXIMATING -Amin(Oe, WITH Amin (mmn)
In this section, we theoretically justify the validity of Approximating -Amin (0o.) with Amin (O%o). Suppose and denote image samples and the corresponding label vector, and & denotes the adversarial examples generated based on the input samples z. We can obtain the lower bound of Amia ( (Qo,) based on Eq. (3):
Amin (Qa,) 2 In ly fa, (x)
1
(9)
yy for (x)
Similarly, the lower bound of Amin (Ge,) can be obtained based on Eq. (5):
ly (
ln Based on Eq. (9) and Eq. (10), we can obtain the inequality of Amin (Poo) and (min(0g,) by adding the two inequalities together:
Amin (Q
(10)
1y fer (
Aenin (O%)) Amin (G 2 In ly fo, (x) Sa (8)1
(11)
Bly fo. (+) fo. (
To form the equivalence of Amin (Po.) and Aymin (6o,). ). introduce margin E 2 0, and then the equivalence of >min (0%,) and Amin (0%)) can be formed Eq. (12) based Eq. (11):
fe, (+) 1 In 1ly fo, lilly So, (
fa, (
Amin (0%)) Amii (O%) +
E.
(12)
According the theoretical findings of Mok et al. (2022), Amin ( (0%,) and Amin (Q%,) satisfy the upper bound shown in Eq. (13) and Eq. (14):
Amin (0p.) -
Dee
(13)
15

---

Published as a conference paper at ICLR 2025
Amin (0%) =
Σ (e.
(14)
where & denotes the number of cigenvalues. Mcanwhile, because the loss of the network with trained weights 8, is often smaller than that of the network with randomly initialized weights 0g because of the training, the losses |lv fe, (z) and y fo. (.) are often larger than l1 fe, (z)
and lly fe, (i) Consequently, In 1 (x))l liv (8)1l
Fao (x))l So, (#)1 is often larger than 0. Therefore, the
upper bound of be formulated Eq. (15):
Σ Au (+a.)))
(15)
Based on Eq. (12) and Eq. (15), conclude that there exists margin satisfying 0 S $ V >x (0, V Ex (,,,),,) such that Amin (Go,) and Amin (Ge,) negatively correlated. Therefore, the validity of approximating Amin (Qo.) with Amin (0o,) justified.
B ADDITIONAL EXPERIMENTAL RESULTS FOR RQ1
B. COMPARISONS UNDER WHITE-BOX ATTACKS ON CIFAR- 100 AND IMAGENET
In addition the comparisons on CIFAR-10 10, we have also performed comparisons on CIFAR-100 and ImageNet. Please note that part of the comparison on ImageNet has been presented in Table 4. In this section, additional results on ImageNet are presented under more adversarial attacks (i.c., PGD and AA) in terms of the models chosen in Table I, in order to achieve more comprehensive comparison. The experimental results are shown in Table 7.
Table 7: Comparisons with state of the arts on CIFAR- 100 and ImageNet under the white-box attacks.
Dataset
Model
Training- Training-Free? Natural Acc.
FGSM
PGD
PGD
AA
ResNet-18 18 (He et al., 2016)
$5. 15.172
25.65%
21.08%
19.98%
18.02%.
DenseNet-IP 121 (Huang et al., 2017)
X
61.71% %
34.28%
27.30%
27.07%
24.55%
DARTS (Liu al., 2019)
59.14%
30.35%
25.66%
25.40%
22.65%
PDARTS (Chen et ul., 2019)
58.41
34.81%
29.11%
28.87%
24.07%
RACL (Dong al., 2020)
X
59.18%
32.04%
26.61%
26.20%
22.92%
CIFAR-100
DSRNA (Hosseini
2021)
57.44%
35.03%
28.11%
27.97%
25.20%
WsrNet (Cheng ul., 2023)
57.81 %
28.08%
23.27%
23.01%
21.57%
GradNorm (Chen al., 2018)
58.66%
32.87%
28.339
28.05%
25.58%
SynFlow (Tanaka
2020)
55.66%
30.08%
25.24%
24.90%
22.46%
CroZe (Ha et al., 2024)
V
59.23%
30.31%
26.16%%
26.02%
22, 82%
Ours
59.39%
35.73%
32.10%
31.88% %
29.95
ResNet-18 (He et al., 2016)
X
47.38%
17.88%
8.88%
8.37%
7.91%
DenseNet-121 (Huang al., 2017)
44.
12.51%
3.74%.
3.14%
3.72%
DARTS (Liu et al., 2019)
X
50.58%
17.45%
10.07%
9.44%
8.53%
PDARTS (Chen et al., 2019)
X
$1.56%
18.10%
10.18%
9.40%
8.77%
RACL (Dong
2020)
$1.59%
18.15%
10.49%
9.82%
8,99%
ImageNet
DSRNA (Hosseini et al., 2021)
43.32%
13.04%
7.88%
7.49%
6.47%
WarNet (Cheng al., 2023)
50.93%
17.58%
10.15%
9.48%
8.77%
GradNorm (Chen
201
$1.34%
18.30%
10.71%
9.90%
9,25%
SynFlow (Tanaka et al., 2020)
51.47%
18.90%
10.73%
9.93%
9.42%
CroZe et al., 2024)
49.95%
16.54%
9.67%
9.10%
8.36%
Ours
52.71 %
19.88
11.96 %
10.94%
10.38%
As can be observed from Table 7, the proposed zero-cost proxy achieves the second-best natural accuracy CIFAR- 100. Meanwhile, the proposed zero-cost proxy demonstrate the ttate-oflle the-art under all the adversarial attacks chosen. Furthermore, the proposed zero-cost proxy still achieves the highest natural accuracy and adversarial robustness ImageNet comparing with the models chosen in Table 1. In summary, the experimental results Table show the effectiveness the proposed zero-cost proxy multiple datasets besides CIFAR-10.
B.2 MORE STATISTICAL RESULTS OF THE PROPOSED ZERO-COST PROXY
In order to evaluate the performance of the proposed zero-cost proxy more comprehensively. we have conduct experiments CIFAR-10 obtain the statistical results. Specifically, we have the
16

---

Published as a conference paper at ICLR 2025
proposed zero-cost proxy twice CIFAR-10, and then adversarially trained the derived architectures following the settings presented in Section 4.1.1. Then, the proposed zero-cost proxy is compared with the widely-compann 2022a)), and the experimental results are reported in the "mean value + standard deviation" format in Table 8.
zero-cost proxy (i.e., NASWOT (Mellor et al., 2021) and NASI (Shu et al.,
Table 8: Statistical results of the proposed zero-cost proxy and widely-compared zero-cost proxies CIFAR-10.
Model
Natural Acc. (%)
PGD (%)
NASWOT (Mellor et al., 2021)
80.89% + 2.33%
51.53% = 0.89%
NASI (Shu et al., 2022a)
78.75% + 0.18%
50.07% = 0.26%
Ours
82.90% + 0.77% %
51.98% % = 0.02%
As shown in Table 8, the proposed zero-cost proxy still achieves highest mean values in terms of the natural accuracy and adversarial robustness. The results demonstrate that the proposed zero-cost proxy has better ability than NASWOT and NASI on discovering neural architectures with better adversarial robustness. In addition, please note that the results shown in Table I are reported based on the single run. This is because there is a convention in the robust NAS community to report the best architecture for comparison (Guo et al., 2020; Mok et al., 2021). But we still present the statistical results in this section for more complete evaluation of the proposed zero-cost proxy.
B.3 COMPARISONS UNDER BLACK-BOU ATTACKS ON CIFAR-100 AND IMAGENET
In order to make the comparison under black-box attacks more comprehensive, the experimental results carried out multiple datasets (i.e., CIFAR-100 and ImageNet) in terms of the baselines chosen in Table 2. Besides, two more recent methods (i.e., WsrNet (Cheng et al., 2023) and CroZe (Ha et al., 2024)) are also contained in the experiments. The experimental results are shown in Table 9,
Table 9: Comparisons with state of the arts on CIFAR-100 and ImageNet under the black- box attack.
Target
Dutaset
DSRNA
RACL
AdvRush
WsrNet
CroZe
Ours
Source
DSRNA (Hosseini al., 2021)
41.74%
45.04%
42,73%
43.13%
45.72%
RCAL (Dong al., 2020)
42.90%
44.04%
42.84%
44.33%
46.87%
AdvRush (Mok al., 2021)
41.83%
39.60%
41.66%
43.30%
45.53%
CIFAR-100
WsrNet (Cheng al., 2023)
44.66%
43.30%
47.13%
46.38%
47.68%
CroZe (Ha et al., 2024)
41.98%
41.68%
45.06%
43.34%
46.15%
Ours
39.90%
40.43%
43.08 %
40.89 %
41.76%
DSRNA (Hosseini et al., 2021)
31.50%
31.74%
29.38%
27.97%
36.69%
RCAL (Dong et al., 2020)
23.77%
27.48%
27.49%
26.51%
32.72%
AdyRush (Mok al., 2021)
23.65%
27.22%
27.37%
26.23%
32.54%
ImageNet
WsrNet (Cheng et al., 2023)
22.45%
27.97%
28, 13%
25.75%
33.405
CroZe (Ha et al., 2024)
22.41%
28. 14%
28, 17%
26.94%
33.71%
Ours
21.72%
24.43%
24.49%
24.61
23.82%
As shown in Table our model achieves the highest attack success rate (100% the test accuracy, highlighted in bold) for all the target models except RACL on CIFAR- 1000 Meanwhile, our model also achieves the highest attack success rate for all the models on ImageNet. Furthermore, when our model is set as the target model, it demonstrates the best adversarial robustness among all the baselines chosen. conclusion, the proposed zero-cost proxy is still effective under the black-box attacks across multiple datasets.
B.4 ADDITIONAL EXPERIMENTAL RESULTS OF THE TRANSFERABILITY
To further evaluate the transferability of the proposed zero-cost proxy, we present the experimental results of models in Table in terms of the transferability. Specifically, these models directly transferred and trained on SVHN and Inny-ImaeeNet-- 200 for evaluation. The experimental results shown in Table 10.
17

---

Published as a conference paper at ICLR 2025
Table 10: Additional results of adversarially trained models on SVHN and Tiny-ImageNeet200.
Datasets
Model
Natural Acc.
FGSM
PGD
DenseNet-121 (Huang al., 2017)
93.72%
89.68%
72.62%
DARTS (Liu et al., 2019)
94.90%
90.01%
77.58%
RobNet-free (Guo et al., 2020)
92.45%
89.33%
85.30%
DSRNA (Hosseini et al., 2021)
91.58%
91.27%
84.94%
WsrNet (Cheng et al., 2023)
94.97%
76.67%
84.20%
SVHN
GradNorm (Chen et al., 2018)
95.16%
92.51%
90.53%
SynFlow (Tanaka et al., 2020)
95.52%
91.53%
76.96%
CroZe (Ha et al., 2024)
93.19%
66.36%
48.11% 18.11%
Ours
95.79% %
95.14%
91.64 %
DenseNet-121 (Huang et al., 2017)
46.26%
22.88%
19.11
DARTS (Liu et al., 2019)
45.94%
24.36%
21.74%
RobNet-free (Guo et al., 2020)
44.24%
25.44%
23.85%
DSRNA (Hosseini al., 2021)
44.42%
28.52%
24.32%
WsrNet (Cheng et al., 2023)
48.62%
22.65%
19.86%
Tiny-ImaeeNet-200
GradNorm (Chen et al., 2018)
49.17%
16.02%
11.35%
SynFlow (Tanaka et al., 2020)
50.96%
12.80%
8.13%
CroZe (Ha al., 2024)
48.06%
10.17%
6.08%
Ours
53.92%
32.81%
15.72%
As can be seen from Table 10, when comparing with the models beyond ResNet-18 and PDARTS in Table 3, our architecture still achieves decent natural accuracy and adversarial robustness, demonstrat- ing superior transferability. Furthermore, we have also provided justification for the transferability grounded in our theoretical contributions.
Justification for the transferability The transferability of the proposed zero-cost proxy can be attribute to the design of the zero-cost proxy. In particular, when evaluating the adversarial robustness not specified. Mcanwhile, the initial weights 9 of
R of the architecture, the input samples the network is randomly determined, without any specific constraints. Consequently, the proposed zero-cost proxy can find the inherently robust neural architectures, without the dependence specific data or weights. As result, the architecture derived on a certain dataset based on the proposed zero-cost proxy be well transferred to other datasets.
B.5 MORE COMPARISON OF THE SEARCH COST
In order to evaluate the efficiency of the proposed zero-cost proxy more comprehensively, we carry out more experimental results in terms of the search cost. Specifically, we compare the proposed zero-cost proxy with the sfate-of-the art robust zero-cost proxy CRoZe (Ha et al., 2024). Besides, the benchmark dataset chosen are CIFAR- 10 and ImageNet, and the search spaces are DARTS space (Liu al., 2019) and WRN space (Zagoruyko, 2016; Li al., 2021). The experimental results are presented in Table 11.
Table 11: The comparisons in terms of the search cost on different datasets and search spaces. The search cost measured in GPU days.
Datasets
Search Spaces
Methods
Cost
DARTS Space
CRoZe (Ha et al., 2024)
0.2
CIFAR-10
DARTS Space
Ours
0.017
WRN Space
Ours
0.019
ImageNet
DARTS Space
Ours
0.036
As can be seen from Table 11, when the search space changed from DARTS space to WRN space, the search cost of the proposed zero-cost proxy increases slightly (from 0.017 GPU days to 0.019
18

---

Published as a conference paper at ICLR 2025
GPU days). Meanwhile, when the datasets change from CIFAR-10 to ImageNet, the proposed zero-cost proxy demonstrates the similar behavior. Besides, no matter which dataset or search space is adopted, the proposed zero-cost proxy always achieves higher efficiency than CRoZe (Ha et al., 2024), demonstrating the efficiency of the proposed zero-cost proxy. This phenomenon can be mainly attributed the fact that CRoZe needs generate adversarial examples achieve the evaluation. In contrast, the proposed zero-cost proxy does not need to generate adversarial examples in evaluation, thus the efficiency is achieved.
B.6 SCALABILITY THE REINFORCEMENT LEARNING
In order to demonstrate the scalability of the proposed zero-cost proxy to other kinds of NAS methods, we integrate our proxy into the reinforcement learning (RL))-ased NAS method (i.e., NASNet (Zoph et al., 2018)) to perform experiments. Specifically, the proposed zero-cost proxy is adopted to replace the training-based performance validation RL-based NAS, and the reward is determined according to the output of the proxy. Then, the derived architectures are forward to adversarial training for evaluation. The experimental results are shown in Table 12.
Table 12: Experimental results in terms of the scalability the RL-based NAS method. In this of experiments, the proposed zero-cost proxy is integrated into the NASNet (Zoph et al., 2018) to search for robust neural architectures. The search cost is measured in GPU days.
Model
Natural Acc. (%)
FGSM (%)
PGD (%)
Cost
NASNet (Zoph et al., 2018) NASNet + Our Proxy
80.61%
54.19%
50.25%
2,000
87.35 %
62.21 %
53.23%
0.027
As be seen from Table 12, the introduction of the proposed zero-cost proxy brings improvements in terms of both natural accuracy and adversarial robustness. Mcanwhile, the search cost is significantly reduced (from 2,000 GPU days 0.027 GPU days). This because the proposed zero-cost proxy replaces the costly architecture training process in the performance evaluation. In summary, the experimental results demonstrate the effectiveness of the proposed zero-cost proxy in the RL-based NAS method, thus the scalability is proved.
10
add noise
0
sep_3x3 sep_5x5 dil_3x3
dil_5x5 max_poolavg_pool
skip
DARTS search space
deteriorating search space
Figure 5: Appearing times of each option in deteriorating search space.
B.7 ABILITY IDENTIFY "BAD" ARCHITECTURE OPTIONS
So far, all experiments are conducted on the DARTS search space for its effectiveness. We are curious the proposed method can perform well in deteriorating search space. For this purpose, We add noise to 3 x 3 separable convolutions (one of the popular options shown in Section 4.1.8) make a bad option that is harmful to the DNN performance (Zela et al., 2020), and repeat the proposed method for three times. The appearing times of each option shown in Figure 5. As shown in the figure, the appearing times of noisy 3 separable convolutions decrease sharply, indicating that the proposed method has the ability identify bad architecture options and will not choose them.
19

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTAL RESULTS FOR RQ2
C.I MORE DETAILS OF THE CONSTRUCTED TINYROBUUTBEENCH
In this section, we will present more details about the proposed Liny-RobuttBench dataset. The details mainly comes from two aspects, i.e., the training details of the architectures and the data distribution the whole dataset. In particular, the training details are presented in Table 13.
Table 13: The training details for the architectures in
Items
Values
Total Training Epoch
105
Initial Learning Rate
0.1
Learning Rate Decay Policy Learning Rate Decent Factor
Stepped Decent
0.1
The Index of Epoch for Learning Rate Decent
99
Momentum
0.9
Weight Decay
0.0001
Adversarial Loss Perturbation Rate
PGD
8/255
Number of Steps
7
Step Size
0.01
In addition, we also present the data distribution terms of the architectures the dataset. In particular, we statistically record both adversarial robustness and natural accuracy of the architectures Tiny-RobustBench, and distribution presented Figure can be seen, the distribution relatively dispersed. Such distribution is beneficial for evaluate the performance of zero-cost proxies, because both the better case and the worse case adversarial robustness are included.
43
Adversarial Robustnes (%)
42
40
39
74
Natural Accuracy (%)
Figure 6: The data distribution of the architectures in Tiny- Biny-RobustBench. Specifically, the distribution contains two aspects, i.c., the adversarial robustness and the natural accuracy.
C.2 HYPPE-PARAMMTTR STUDIES
In order to evaluate different settings of the hyper-parameters and h in Eq. (4), we have performed hyper-parameter studies for both and h on the Tiny-RobustBench dataset. Specifically, the hyper- parameter h is fixed to 50, and the hyper-parameter is set to 5 10 and 108 in order to evaluate the impact of on the performance of the proposed zero-cost proxy. Besides, in order to explore the impact of h on the performance of the proposed zero-cost proxy. the hyper byper-parameter parameter t fixed to 5 x 108, and the
10a, 5 x 105, x 107,
is set to 0.5, five. 50, x 10 and 5 10 Please note that / and h are fixed to 10 and 50 because this is the default setting
20

---

Published as a conference paper at ICLR 2025
indicated in Section 4.2.1. The experimental results of the hyper-paramete studies are shown in Tables 14 and 15. In order to evaluate the impact on architecture ranking and architecture search, we report the KTau correlation of the proxy and the accuracy under PGD" attack of the architecture with the highest rank.
Table 14:
Experimental results of the hyper-parameter study for the hyper-parameter t. In the experiments, the hyper-parameter is fixed to 50, and the Tiny- Tiny-RobustBench dataset used.
(/- 50)
TTau
PGD" (%)
x 10* (x 0.01)
0.29
42.93%
10 (x 0.1)
0.32
42.93%
5 x 108 (Default)
0.33
42.93%
5 x 107 (x 10)
0.34
42.93%
10 (x 100)
0.26
42.93%
Table 15: Experimental results of the hyper-parameter study for the hyper-parameter h. In the
experiments, the hyper-parameter / is fixed to 5 x 10", and the Tiny-RobustBench dataset is used.
(t = x 10%)
KTauu
PGD7 (%)
0.5 (x 0.01)
0.33
42.93%
(x 0.1)
0.33
41.90%
50 (Default)
0.33
42.93%
x 102 (x 10)
0.33
42.93%
10 (x 100)
0.33
42.93%
As show in Table 14, when the hyper-parameter / decreases to 5 10a increases 106 the KTau correlation suffers significant drop. In contrast, the KTau correlation almost keeps unchanged when & decreases the architecture ranking performance of the prro-cost-prox, will be negatively affected when the hyper-parameter is too large or small. Moreover, the architecture with the highest rank keeps the
10" or increases to 5 10". Based on these facts, we can conclude that
the same when the f.
f is changing, indicating the architecture search is not sensitive to
Furthermore, the results of the hyper-panne parammteteeter study for h are presented in Table 15. As can be seen, the KTTuu correlation remains the same when h changed. Meanwhile, architecture with the highest rank also keeps unchanged except when h is set to five. Giving all these facts, we can come to the conclusion that the performance of the proposed zero-cost proxy is not sensitive to the dyper-paramette h.
In summary, the stability of the proposed zero-cost proxy is demonstrated based on the experimental results shown in Tables 14 and 15, because the performance of the proposed zero-cost proxy is almost not affected when both seper-paraamters change in a large range.
C.3 FURTHER COMPARISONS WITH MGM PROXY
As shown in Figure in Section 4.2.4, the MGM proxy (Xu et al., 2021) demonstrates strong negative correlation with the adversarial robustness of urchitectures. Motivated by this fact, we are curious about whether the negative version of the MGM proxy is effective to search for robust architectures. Therefore, we directly use the negative version of MGM proxy to search for architectures in DARTS search space, and then the derived architecture is adversarially trained for evaluation. The experi- mental results presented in Table 16, where the natural accuracy and the accuracy under 2GD adversarial attack reported.
clear shown in Table 16 that the proposed zero-cost proxy outperforms the negative version of MGM. Specifically, the proposed zero-cost proxy achieves the improvement of +3.23 and +2.93 for natural accuracy and the accuracy under 26 adversarial attack, respectively. Therefore, the superiority of the proposed zero-cost proxy is further demonstrated.
21

---

Published as a conference paper at ICLR 2025
Table 16: Comparisons between the negative version of the MGM proxy (Xu al., 2021) and the proposed zero-cost proxy. The architectures are searched in DARTS search space and adversarially trained on CIFAR-10.
Model
Natural Acc. (%)
PGD (%)
Negative MGM (Xu et al., 2021) Ours
80.44%
49.07% %
83.67 %
52.00 %
D ARCHITECTURE VISUALIZATION
The architectures searched by the proposed zero-cost proxy are presented Figures and 8. Specifi- cally, cells searched in DARTS search space. The normal cells are presented in Figures 7a and 8a, and the reduction cell are presented in Figures 7b and 8b.
cory 2id
cony
HAp only Jidl
akio_ cunment
Hap, coby 5u5
3x5
U4-1]
comy
(a) Normal Cell
C
comeect
(b) Reduction Cell
Figure 7: CIFAR-10.
The normal cell and the reduction cell searched by the proposed zero-cost proxy on
(a) Normal Cell
Ann caayy Jelk skip connect
c. (k-2)
Aap, cray Aod
(k-1)
A0p, cumyy
Xva pool _S33
(b) Reduction Cell
Figure The normal cell and the reduction cell searched by the proposed zero-cost proxy on ImageNet.
22


Reviews:
[{"review_id": "E6dQQDZyl0", "reviewer": "Reviewer_rBjP", "summary": "This paper targets a lightweight proxy to assess the adversarial robustness of Neural Architecture Search (NAS) networks. The proposed proxy is represented as the product of two terms (Eq. 4). The first term, based on the intuition that adversarial and natural accuracy are reversely correlated, is introduced to consider adversarial accuracy and is experimentally validated. The second term is proposed through a theoretical approximation using the Neural Tangent Kernel. The proposed proxy is experimentally validated.", "questions": "- In the preliminatries, 3.1., the formulation is bit awkward; if the function $f_{\\theta_t}$ is a function $\\mathbb{R}^d \\rightarrow \\mathbb{R}$, then the output will be a single scalar, but in the equations the l2-norm calculation is applied … I also checked the reference paper (Xu et al., (2021)), but it is slightly different. \n\n- The overall experiments on Kendall’s tau seems to be not high (Fig.2,Fig.4). (Relatively high, but not significant in absolute terms). Can the authors explain this in further details?\n\n- The cost analysis is done with the GPU search day metric. How is the cost calculated? Under which GPU?\n\n[Post-Rebuttal] I have reviewed the author's rebuttal and the feedback from other reviewers. Most of my concerns have been addressed, and given the paper's contributions, I would like to maintain my current rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is easy to read through, well presented.\n- The paper proposes relatively strong method for the proxy of searching adversarially robust architectures.\n- The paper includes numerous validations including performance under white-box, black-box attacks, datasets, numerous datasets, and ablation studies.", "weaknesses": "1.  To read through, I felt that the paper has to focus on two points : \n- Is the proposed method superior in terms of cost-efficiency? (zero-cost)\n- Is the proposed method superior in terms of robust accuracy?\n- For now, RQ1 and RQ2 looks a bit similar(line 255-256), before looking at the experimental results.\n\n2. I feel the cost-efficiency related explanation is slightly short. The authors could include detailed explanations for (line 183-185.) such as , the reason the authors chose to iterate samples over generating adversarial samples itself and why is it efficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7woR7mkT20", "reviewer": "Reviewer_jcgN", "summary": "This paper proposes a zero-shot proxy that requires no training and depends only on the initial neural network weights to find the robust architecture. Using this zero-shot proxy reduces the neural architecture search speed drastically. Experiments are performed on multiple datasets with varying resolution and number of classes.", "questions": "While the contribution in terms of speed reduction is significant for the proposed method, my concern is the increase in adversarial robustness as compared to the baselines is very minimal in the majority of the cases.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The reduction in the search cost is very significant. \n- The paper is well-written with typos.  \n- From Table 3, the transferability results are good. \n- A novel dataset will be released by the authors, which will help advance research in the robust NAS direction.   \n- Comparison with the latest state-of-the-art methods.", "weaknesses": "- While the search cost is reduced significantly, the increase in the adversarial robustness is minimal (less than 1% in most cases for Table 1). A similar observation can be derived from the results in Table 4.    \n- Minor typos: \"8/2550\" instead of \"8/255\" on line 301, \"We\" instead of \"we\" on line 63.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70mRI4R95U", "reviewer": "Reviewer_kdEp", "summary": "This paper introduces a zero-shot proxy applied directly to the initial weights of a deep neural network (DNN), serving as an upper bound for adversarial loss. This approach enhances the adversarial robustness of the network without requiring adversarial examples. Theoretical justification for the zero-shot proxy is provided based on the Neural Tangent Kernel (NTK) and the input loss landscape. Experimental results demonstrate a 20x speedup over state-of-the-art robust neural architecture search (NAS) methods, with no loss in performance against black-box and white-box attacks.", "questions": "Please refer to Weaknesses Section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The preliminaries on the Neural Tangent Kernel (NTK) and input loss landscape are clearly presented, making the methodology easy to understand. All symbols and equations are well-defined. \n\n+ A robust theoretical justification for the proposed zero-shot proxy is provided by establishing the corresponding loss as an upper bound for adversarial loss.\n\n+ The efficiency of the search process using their technique is clearly demonstrated in the experimental section.\n\n+ The authors introduce a novel dataset called Tiny-RobustBench, which may prove valuable to the research community.", "weaknesses": "- In the theoretical analysis section,  $\\lambda_{\\min}(\\hat{\\Theta_{\\theta_o}})$ is approximated based on empirical observations to $-\\lambda_{\\min}(\\Theta_{\\theta_o})$, which weakens the theoretical foundation of the work. I wonder if the authors could establish the relationship between the two NTKs in strict mathematical terms, such as showing that the approximation holds true within a specific margin of error with a certain probability. Currently, the analysis relies on an empirically driven approximation, which undermines its theoretical robustness.\n- The performance of the black-box and white-box attacks is demonstrated using a single dataset (CIFAR-10) in Tables 1 and 2. The authors should consider providing performance comparisons across multiple datasets for the chosen baselines in both tables. Additionally, Table 2 should clearly present a comparison of search efficiency.\n- I am curious about the rationale for selecting only specific models (ResNet-18 and PDARTS) for comparison in Table 3. Would it be possible to include models from Table 1 to evaluate their transferability in Table 3?\n- There is inadequate justification for why their approach exhibits better transferability in Table 3. It would be beneficial to provide justification grounded in their theoretical contributions, particularly highlighting the unique advantages that their proposed techniques offer to enhance transferability.\n- The authors should discuss the assumptions made in their analysis and their validity within the context of the study. For instance, when transforming from Equation 1 to Equation 3, it would be helpful to clarify the assumptions involved and how they apply in their setting. Specifically, I am uncertain whether the assumption of infinite-width DNN parameters is valid; if it is not, what implications does that have for this paper? I encourage the authors to include these details, even if they are standard, as doing so would enhance the paper's readability and facilitate the assessment process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)