SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyeonjeong Ha $^{1*}$ , Minseon Kim $^{1*}$ , Sung Ju Hwang $^{1,2}$
$^{1}$ Korea Advanced Institute of Science and Technology (KAIST),  $^{2}$ DeepAuto.ai {hyeonjeongha, minseonkim, sjhwang82}@kaist.ac.kr
Recent neural architecture search (NAS) frameworks have been successful in finding optimal architectures for given conditions (e.g., performance or latency). However, they search for optimal architectures in terms of their performance on clean images only, while robustness against various types of perturbations or corruptions is crucial in practice. Although there exist several robust NAS frameworks that tackle this issue by integrating adversarial training into one-shot NAS, however, they are limited in that they only consider robustness against adversarial attacks and require significant computational resources to discover optimal architectures for a single task, which makes them impractical in real-world scenarios. To address these challenges, we propose a novel lightweight robust zero-cost proxy that considers the consistency across features, parameters, and gradients of both clean and perturbed images at the initialization state. Our approach facilitates an efficient and rapid search for neural architectures capable of learning generalizable features that exhibit robustness across diverse perturbations. The experimental results demonstrate that our proxy can rapidly and efficiently search for neural architectures that are consistently robust against various perturbations on multiple benchmark datasets and diverse search spaces, largely outperforming existing clean zero-shot NAS and robust NAS with reduced search cost. Code is available at https://github.com/HyeonjeongHa/CRoZe.
Neural architecture search (NAS) techniques have achieved remarkable success in optimizing neural networks for given tasks and constraints, yielding networks that outperform handcrafted neural architectures [2, 24, 28, 33, 42]. However, previous NAS approaches have primarily aimed to search for architectures with optimal performance and efficiency on clean inputs, while paying less attention to robustness against adversarial perturbations [16, 29] or common types of corruptions [20]. This can result in finding unsafe and vulnerable architectures with erroneous and high-confidence predictions on input examples even with small perturbations [31, 22], limiting the practical deployment of NAS in real-world safety-critical applications.
To address the gap between robustness and NAS, previous robust NAS works [31, 18] have proposed to search for adversarially robust architectures by integrating adversarial training into NAS. Yet, they are computationally inefficient as they utilize costly adversarial training on top of the one-shot NAS methods [26, 4], requiring up to  $33 \times$  larger computational cost than clean one-shot NAS [42]. Especially, Guo et al. [18] takes almost 4 GPU days on NVIDIA 3090 RTX GPU to train the supernet, as it requires performing adversarial training on subnets with perturbed examples (Figure 2, RobNet). Furthermore, they only target a single type of perturbation, i.e., adversarial perturbation [16, 29], thus, failing to generalize to diverse perturbations. In order to deploy NAS to real-world applications
*Equal contribution. Author ordering is determined by coin flip.
Common Corruption
(a) Clean NAS and Robust NAS vs CRoZe.
Figure 1: Generalizable lightweight proxy for robust NAS against diverse perturbations. While previous NAS methods search for neural architectures primarily on clean samples (Clean NAS) or adversarial perturbations (Robust NAS) with excessive search costs and fail to generalize across diverse perturbations, our proposed proxy, namely CRoZe, can rapidly search for high-performing neural architectures against diverse perturbations. Specifically, CRoZe evaluates the network's robustness in a single step based on the consistency across the features  $(z$  and  $z^r)$ , parameters  $(\theta$  and  $\theta^r)$ , and gradients  $(\nabla_{\theta}\mathcal{L}$  and  $\nabla_{\theta^r}\mathcal{L})$  between clean and robust network against clean and perturbed inputs, respectively.
(b) Consistency-based zero-cost proxy (CRoZe).
that require handling diverse types of tasks and perturbations, we need a lightweight NAS method that can yield robust architectures without going over such costly processes.
To tackle this challenge, we propose a novel and lightweight Consistency-based Robust Zero-cost proxy (CROzE) that can rapidly evaluate the robustness of the neural architectures against diverse semantic-preserving perturbations without requiring any iterative training. While prior clean zero-shot NAS methods [1, 30] introduced proxies that score the networks with randomly initialized parameters [23, 39, 27, 37] without any training, they only consider which parameters are highly sensi
(a) Standard-Trained
(b) Adverserially-Trained
Figure 2: Final performance of the searched network in DARTS search space on CIFAR-10 through clean one-shot NAS, robust NAS, clean zero-shot NAS and our CRoZe.
tive to clean inputs for a given task, as determined by measuring the scale of the gradients based on the objectives and thus yield networks that are vulnerable against perturbed inputs (Figure 1a).
Specifically, our proxy captures the consistency across the features, parameters, and gradients of a randomly initialized model for both clean and perturbed inputs, which is updated with a single gradient step (Figure 1b). This metric design measures the model's robustness in multiple aspects, which is indicative of its generalized robustness to diverse types of perturbations. This prevents the metric from being biased toward a specific type of perturbation and ensures its robustness across diverse semantic-preserving perturbations. Empirically, we find that a neural architecture with the highest performance for a single type of perturbation tends to exhibit larger feature variance for other types of perturbations (Figure 4), while our proxy that considers the robustness in multiple aspects obtains features with smaller variance even on diverse types of perturbations. This suggests that our proxy is able to effectively discover architectures with enhanced generalized robustness.
We validate our approach through extensive experiments on diverse search spaces (NAS-Bench 201, DARTS) and multiple datasets (CIFAR-10, CIFAR-100, ImageNet16-120), with not only the adversarial perturbations but also with various types of common corruptions [20], against both clean zero-shot NAS [1, 30] and robust NAS baselines [31, 18]. The experimental results clearly demonstrate our method's effectiveness in finding generalizable robust neural architectures, as well as its computational efficiency. Our contributions can be summarized as follows:
- We propose a simple yet effective consistency-based zero-cost proxy for robust NAS against diverse perturbations via measuring the consistency of features, parameters, and gradients between perturbed and clean samples.
- Our approach can rapidly search for generalizable neural architectures that do not perform well only on clean samples but also are highly robust against diverse types of perturbations on various datasets.
- Our proxy obtains superior Spearman's  $\rho$  across benchmarks compared to existing zero-shot NAS methods and identifies robust architectures that exceed robust NAS frameworks by  $5.57\%$  with 14.7 times less search cost within the DARTS search space on CIFAR-10.
Robustness of DNNs against Perturbations. Despite the recent advances, deep neural networks (DNNs) are still vulnerable to small perturbations on the input, e.g., common corruptions [20], random noises [10], and adversarial perturbations [3, 36], which can result in incorrect predictions with high confidence. To overcome such vulnerability against diverse perturbations, many approaches have been proposed to train the neural network to be robust against each type of perturbation individually. To learn a rich visual representation from limited crawled data, previous works [21, 8] utilized a combination of strong data augmentation functions to improve robustness to common corruptions and random Gaussian noises. Furthermore, to overcome adversarial vulnerability, widely used defense mechanisms [16, 29, 32] generate adversially perturbed images by taking multiple gradient steps to maximize the training loss and use them in training to improve the model's robustness.
Neural Architecture Search. Neural architecture search (NAS) leverages reinforcement learning [45, 2, 44] or evolutionary algorithms [34, 25, 13, 35] to automate the design of optimal architectures for specific tasks or devices. However, those are computationally intensive, making them impractical to be applied in real-world applications. To address this, zero-shot NAS methods [1, 30] have emerged that significantly reduce search costs by predicting the performance of architecture at the initialization state only with a single batch of a given dataset. Despite the improvement in NAS, previous zero-shot NAS methods, and conventional NAS methods aim only to find architectures with high accuracy on clean examples, without considering their robustness against various perturbations. In particular, SynFlow [1] lacks data incorporation in its scoring mechanism, potentially failing to find the network that can handle diverse perturbations. NASWOT [30] search models with low activation correlation across two different inputs, emphasizing their distinguishability. However, the criterion contradicts the requirements of robustness, which necessitates finding a network that maintains similar activations for both clean and perturbed inputs. As a result, models found with previous NAS methods often lead to incorrect predictions with high confidence [31, 22] even with small imperceptible perturbations applied to the inputs. A new class of NAS methods [18, 31] that considers robustness against adversarial perturbations has emerged. Yet, they require adversarial training of the supernet, which demands more computational resources than conventional NAS [34, 13] due to repeated adversarial gradient steps. Furthermore, adversarial robust NAS often overfit to a single type of perturbation due to only considering the adversarial robustness. Thus, there is a need for a lightweight NAS approach that can achieve generalized robustness for safe real-world applications.
Our ultimate goal is to efficiently search for robust architectures that have high performance on various tasks, regardless of the type of perturbations applied to the input samples. To achieve this goal, we propose a Consistency-based Robust Zero-cost proxy (CRoZe) that considers the consistency of the features, parameters, and gradients between a single batch of clean and perturbed inputs obtained by taking a single gradient step. The proposed proxy enables the rapid evaluation of the robustness of the neural architectures in the random states, without any adversarial training (Figure 1).
3.1 Robust Architectures
Formally, our goal is to accurately estimate the final robustness of a given neural architecture  $\mathcal{A}$  and a single batch of inputs  $B = \{(x,y)\}$ , without training. Here,  $x\in X$  is the input sample, and  $y\in Y$  is its corresponding label for the given dataset  $\mathcal{D}$ . In the following section, the network  $f_{\theta}(\cdot)$  consists of an encoder  $e_{\psi}(\cdot)$  and a linear layer  $h_{\pi}(\cdot)$ , which is  $\mathcal{A}$  that is parameterized with  $\psi$  and  $\pi$ , respectively. The straightforward approach to evaluating the robustness of the network is measuring the accuracy against the perturbed input  $x^{\prime}$  with unseen semantic-preserving perturbations, as follows:
$$
\text {A c c u r a c y} = \frac {1}{N} \sum_ {n = 1} ^ {N} \zeta (\underset {c \in Y} {\arg \max } \mathbb {P} \left(h _ {\pi} \circ e _ {\psi} \left(x ^ {\prime}\right) = c\right) = y), \tag {1}
$$
where  $\zeta$  is the Kronecker delta function, which returns 1 if the predicted class  $c$  is the same as  $y$ , and 0 otherwise,  $x'$  is a perturbed input, such as one with random Gaussian noise, common types of corruptions [21], or adversarial perturbations [29, 16] applied to it. Specifically, to have a correct prediction on the unseen perturbed input  $x'$ , the model needs to extract similar features between  $x'$  and  $x$ , assuming that the model can correctly predict the label for input  $x$  as follows:
$$
\left\| e _ {\psi} (x) - e _ {\psi} \left(x ^ {\prime}\right) \right\| \leq \epsilon , \tag {2}
$$
where  $\epsilon$  is sufficiently small bound. Thus, a robust model is one that can extract consistent features across a wide range of perturbations. However, precisely assessing the accuracy of the model against perturbed inputs requires training from scratch with a full dataset, which incurs a linear increase in the computational cost with respect to the number of neural architectures to be evaluated.
3.2 Estimating Robust Network through Perturbation
In this section, we explain details on preliminary protocols before computing our proxy. Due to the impractical computation cost to obtain a combinatorial number of fully-trained models in a given neural architecture search space (i.e.,  $10^{19}$  for DARTS), we propose to utilize two surrogate networks which together can estimate the robustness of fully-trained networks within a single gradient step. The two surrogate networks are a clean network  $f_{\theta}$  with the randomly initialized parameter  $\theta$  and a robust network  $f_{\theta^r}$  with the robust parameter  $\theta^r$ , which is determined with a parameter perturbations from  $f_{\theta}$ . Then, the obtained  $\theta^r$  is used to generate the single batch of perturbed inputs for our proxy.
Robust Parameter Update via Layer-wise Parameter Perturbation. We employ a surrogate robust network to estimate the output of fully-trained networks against perturbed inputs. To make the perturbation stronger, we use a double-perturbation scheme that combines layer-wise parameter perturbations [41] and input perturbations, both of which maximize the training objectives  $\mathcal{L}$ . This layer-wise perturbation allows us to learn smoother updated parameters by min-max optimization, through which we can obtain the model with the maximal possible generalization capability [41, 14] within a single step. Specifically, given a network  $f$  is composed of  $M$  layers,  $f_{\theta} = f_{\theta_M} \circ \dots \circ f_{\theta_1}$ , with parameters  $\theta = \{\theta_1, \ldots, \theta_M\}$ , the  $m^{th}$  layer-wise parameter perturbation is done as follows:
$$
\theta_ {m} ^ {r} \leftarrow \theta_ {m} + \beta * \frac {\nabla_ {\theta_ {m}} \mathcal {L} \left(f _ {\theta} (x) , y\right)}{\| \nabla_ {\theta_ {m}} \mathcal {L} \left(f _ {\theta} (x) , y\right) \|} * \| \theta_ {m} \|, \tag {3}
$$
where  $\beta$  is the step size for parameter perturbations,  $\| \cdot \|$  is the norm, and  $\mathcal{L}$  is the cross-entropy objective. This bounds the size of the perturbation by the norm of the original parameter  $\| \theta_{m}\|$ .
Perturbed Input. On top of the perturbed parameters (Eq. 3), we generate perturbed input images by employing fast gradient sign method (FGSM) [16], which is the worst case adversarial perturbation to the input  $x$  as follows:
$$
\delta = \epsilon \operatorname {s i g n} \left(\nabla_ {x} \mathcal {L} \left(f _ {\theta^ {r}} (x), y\right)\right), \tag {4}
$$
where  $\delta$  is a generated adversarial perturbation that maximizes the cross-entropy objective  $\mathcal{L}$  of given input  $x$  and given label  $y$ . Then, we utilize the perturbed inputs  $(x' = x + \delta)$  to estimate the robustness of the fully-trained model. Although CRoZe is an input perturbation-agnostic proxy (Supplementary B.2), we employ adversially perturbed inputs for all the following sections.
3.3 Consistency-based Proxy for Robust NAS
We now elaborate the details on our proxy that evaluate the robustness of the architecture with the two surrogate networks: the clean network that is randomly initialized and uses clean images  $x$  as inputs, and the robust network parameterized with  $\theta^r$  which uses perturbed images  $x + \delta$  as inputs.
Features, Parameters, and Gradients. As we described in Section 3.1, we first evaluate the representational consistency between clean input  $(x)$  and perturbed input  $(x^{\prime})$  by forwarding them through the encoder of clean surrogate network  $f_{\theta}(\cdot)$  and robust surrogate network  $f_{\theta^{r}}(\cdot)$ , respectively, as follows:
$$
\mathcal {Z} _ {m} \left(f _ {\theta} (x), f _ {\theta^ {r}} \left(x ^ {\prime}\right)\right) = 1 + \frac {z _ {m} \cdot z _ {m} ^ {r}}{\| z _ {m} \| \| z _ {m} ^ {r} \|}, \tag {5}
$$
where  $z_{m}$  and  $z_{m}^{r}$  are output feature of each network  $f_{\theta}(\cdot)$  and  $f_{\theta^r}(\cdot)$ , respectively, from each  $m^{th}$  layer. Especially, we measure layer-wise consistency with cosine similarity function between clean and robust features. The higher feature consistency infers the higher robustness of the network.
However, the proxy solely considering the feature consistency within a single batch can be heavily reliant on the selection of the batch. Therefore, to complement the feature consistency, we propose incorporating the consistency of updated parameters and gradient conflicts from each surrogate network as additional measures to evaluate the robustness of the network. To introduce these concepts, let us first denote the gradient and updated parameter of each surrogate network. The gradient  $g$  from the clean surrogate network  $f_{\theta}$  and robust surrogate network  $f_{\theta^r}$  against clean images  $x$  and perturbed images  $x'$ , respectively, are obtained as follows:
$$
g = \nabla_ {\theta} \mathcal {L} \left(f _ {\theta} (x), y\right), \quad g ^ {r} = \nabla_ {\theta^ {r}} \mathcal {L} \left(f _ {\theta^ {r}} \left(x ^ {\prime}\right), y\right), \tag {6}
$$
where  $g$  and  $g^{r}$  are the gradients with respect to cross-entropy objectives  $\mathcal{L}$  for clean images  $x$  and perturbed images  $x'$ , respectively. Then, we can acquire single-step updated clean parameters  $\theta$  and robust parameters  $\theta^{r}$  calculated with gradients  $g$  and  $g^{r}$  and learning rate  $\gamma$ , respectively as follows:
$$
\theta_ {1} \leftarrow \theta - \gamma g, \quad \theta_ {1} ^ {r} \leftarrow \theta^ {r} - \gamma g ^ {r}. \tag {7}
$$
Since each surrogate network represents the model for each task, i.e., clean classification and perturbed classification, the parameters and gradients of each surrogate network correspond to the updated weights and convergence directions for each task. Thus, the network that has high robustness will exhibit identical or similar parameter spaces for both classification tasks. However, as acquiring parameters of a fully-trained network is impractical, we estimate the converged parameters with the single-step updated parameters  $\theta_{1}$  and  $\theta_1^r$ . Accordingly, since the higher similarity of single-step updated parameters may promote the model to converge to an identical or similar parameter space for both tasks, we evaluate the parameter similarity as one of our proxy terms as follows:
$$
\mathcal {P} _ {m} \left(\theta_ {1}, \theta_ {1} ^ {r}\right) = 1 + \frac {\theta_ {1 , m} \cdot \theta_ {1 , m} ^ {r}}{\| \theta_ {1 , m} \| \| \theta_ {1 , m} ^ {r} \|}. \tag {8}
$$
Moreover, each gradient of the surrogate networks represents the converged direction of given objectives for each task, which is cross-entropy loss of clean input and perturbed input (Eq. 6). We employ the similarity of these gradients to assess the difficulties of optimizing architecture for both tasks. When the gradient directions are highly aligned between the two tasks, the learning trajectory for both tasks becomes more predictable, facilitating the optimization of both tasks easily. In contrast, orthogonal gradient directions lead to greater unpredictability, hindering optimization and potentially resulting in suboptimality for both clean or perturbed classification tasks. Therefore, to evaluate the stability of optimizing both tasks, we measure the absolute value of gradient similarity as follows:
$$
\mathcal {G} _ {m} (g, g ^ {r}) = \left| \frac {g _ {m} \cdot g _ {m} ^ {r}}{\| g _ {m} \| \| g _ {m} ^ {r} \|} \right|. \tag {9}
$$
Consistency-based Robust Zero-Cost Proxy: CRoZe. In sum, to evaluate the robustness of the given architecture, we propose a scoring mechanism that evaluates the consistencies of features, parameters, and gradients between the clean network  $f_{\theta}$  and the robust network  $f_{\theta^r}$  that are obtained with a single gradient update. The robustness score for a given architecture is computed as follows:
$$
\operatorname {C R o Z e} \left(x, x ^ {\prime}; f _ {\theta}, f _ {\theta^ {r}}\right) = \sum_ {m = 1} ^ {M} \mathcal {Z} _ {m} \times \mathcal {P} _ {m} \times \mathcal {G} _ {m}. \tag {10}
$$
That is, we score the network  $f_{\theta}$  with a higher CRoZe score as more robust to perturbations. In the next section, we show that this measure is highly correlated with the actual robustness of a fully-trained model (Table 1).
We now experimentally validate our proxy designed to identify robust architectures that perform well on both clean and perturbed inputs, on multiple benchmarks. We first evaluate Spearman's  $\rho$  between robustness and the proxy's values across different tasks and perturbations in the NAS-Bench-201
Table 1: Comparison of Spearman's  $\rho$  between the actual accuracies and the proxy values on CIFAR-10 in the NAS-Bench-201 search space. Clean stands for clean accuracies and robust accuracies are evaluated against diverse adversarial attacks [16, 5, 32, 38, 17, 7]. * denotes the HRS value of each attack and AA. indicates AutoAttack. Avg. stands for average Spearman's  $\rho$  values with all accuracies. Bold and underline stands for the best and second, respectively. All models are trained on adversarially-perturbed images.
Proxy Type	Clean	FGSM	PGD	FGSM*	PGD*	CW	DeepFool	SPSA	LGV	AA.	Avg.
FLOPs	0.670	0.330	0.418	0.531	0.515	0.189	0.364	0.196	0.347	0.365	0.393
#Params.	0.678	0.341	0.429	0.541	0.526	0.182	0.371	0.209	0.355	0.375	0.401
Plain [1]	-0.042	-0.007	-0.012	-0.016	-0.016	0.072	0.009	0.009	0.010	0.015	0.002
Grasp [1]	0.470	0.324	0.341	0.392	0.375	0.179	0.393	0.249	0.401	0.397	0.352
Fisher [1]	0.482	0.226	0.276	0.335	0.334	0.234	0.242	0.092	0.239	0.244	0.270
GradNorm [1]	0.659	0.336	0.400	0.490	0.478	0.264	0.421	0.149	0.401	0.405	0.400
SynFlow [1]	0.635	0.355	0.420	0.519	0.498	0.202	0.397	0.196	0.387	0.383	0.399
NASWOT [30]	0.600	0.332	0.381	0.437	0.438	0.240	0.250	0.197	0.265	0.280	0.342
CRoZe	0.723	0.417	0.501	0.602	0.588	0.220	0.454	0.240	0.449	0.458	0.465
search space, comparing it to clean zero-shot NAS methods (Section 4.2). We then evaluate the computational efficiency and final performance of the chosen architecture using our proxy in the DARTS search space, comparing it to existing robust NAS methods, which are computationally costly (Section 4.3). Lastly, we analyze the proxy's ability to accurately reflect the fully trained model's behavior in a single step, as well as the capacity of the chosen robust architecture to consistently generate features, aligned gradients, and parameters against clean and perturbed inputs (Section 4.4).
Baselines. We consider three types of existing NAS approaches as our baselines. 1) Clean one-shot NAS [42, 6]: One-shot NAS methods for searching architectures only on clean samples. 2) Clean zero-shot NAS [1, 30]: Zero-shot NAS with proxies that evaluate the clean performance of architectures without any training. 3) Robust NAS [18, 31]: One-shot NAS methods for searching architectures only on adversarially perturbed samples.