Title: MeCo: Zero-Shot NAS with One Data and Single Forward Pass via Minimum Eigenvalue of Correlation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Architecture Search

Zero-Cost Proxy

Evaluation Strategy

Feature Map
Abstract: Neural Architecture Search (NAS) is a promising paradigm in automatic architecture engineering. Zero-shot NAS can evaluate the network without training via some specific metrics called zero-cost proxies. Though effective, the existing zero-cost proxies either invoke at least one backpropagation or depend highly on the data and labels. To alleviate the above issues, in this paper, we first reveal how the Pearson correlation matrix of the feature maps impacts the convergence rate and the generalization capacity of an over-parameterized neural network. Enlightened by the theoretical analysis, we propose a novel zero-cost proxy called $\mathsf{MeCo}$, which requires only one random data for a single forward pass. We further propose an optimization approach $\mathsf{MeCo_{opt}}$ to improve the performance of our method. We design comprehensive experiments and extensively evaluate $\mathsf{MeCo}$ on multiple popular benchmarks. $\mathsf{MeCo}$ achieves the highest correlation with the ground truth (e.g., 0.89 on NATS-Bench-TSS with CIFAR-10) among all the state-of-the-art proxies, which is also fully independent of the data and labels. Moreover, we integrate $\mathsf{MeCo}$ with the existing generation method to comprise a complete NAS. The experimental results illustrate that $\mathsf{MeCo}$-based NAS can select the architecture with the highest accuracy and a low search cost. For instance, the best network searched by $\mathsf{MeCo}$-based NAS achieves 97.31% on CIFAR-10, which is 0.04% higher than the baselines under the same settings. Our code is available at https://github.com/HamsterMimi/MeCo

MeCo: Zero-Shot NAS with One Data and Single Forward Pass via Minimum Eigenvalue of Correlation

Abstract

Neural Architecture Search (NAS) is a promising paradigm in automatic architecture engineering. Zero-shot NAS can evaluate the network without training via some specific metrics called zero-cost proxies. Though effective, the existing zero-cost proxies either invoke at least one backpropagation or depend highly on the data and labels. To alleviate the above issues, in this paper, we first reveal how the Pearson correlation matrix of the feature maps impacts the convergence rate and the generalization capacity of an over-parameterized neural network. Enlightened by the theoretical analysis, we propose a novel zero-cost proxy called MeCo , which requires only one random data for a single forward pass. We further propose an optimization approach MeCo opt to improve the performance of our method. We design comprehensive experiments and extensively evaluate MeCo on multiple popular benchmarks. MeCo achieves the highest correlation with the ground truth (e.g., 0.89 on NATS-Bench-TSS with CIFAR-10) among all the state-of-the-art proxies, which is also fully independent of the data and labels. Moreover, we integrate MeCo with the existing generation method to comprise a complete NAS. The experimental results illustrate that MeCo -based NAS can select the architecture with the highest accuracy and a low search cost. For instance, the best network searched by MeCo -based NAS achieves 97.31% on CIFAR-10, which is 0.04% higher than the baselines under the same settings. Our code is available at https://github.com/HamsterMimi/MeCo.

1 Introduction

Deep Neural Networks (DNNs) have been ubiquitously adopted in various fields due to their ability in hierarchical feature extraction [1-5]. It is a common consensus that the architecture of DNN is essential in model performance. Unfortunately, the manual trial-and-error method is non-scalable and inefficient, making it infeasible to find the globally optimal structure. Thanks to the development of the Neural Architecture Search (NAS), it is possible to automatically select the network architecture with the best performance.

There are abundant works proposed to improve the performance of NAS. The most fundamental observation is to fully or partially train the candidate networks with specific data and select the network with the highest accuracy. Though effective, these multi-shot NAS methods are inevitably resource-consuming, thus, are non-trivial to be adopted in real-world applications. To address this issue, the one-shot NAS has been proposed in which a supernet is designed and only requires one-time full training [6]. In 2020, Mellor et al. [7] proposed the zero-shot NAS and realized an efficient approach in which a zero-cost proxy is proposed to evaluate the networks without training.

Despite the effectiveness of the existing zero-cost proxies, there are some critical issues that remain to be solved. (i) Almost all current proxies are established from the network gradients. As a consequence, computing these proxies requires at least one backpropagation, which is resource-consuming. (ii) The majority of the currently used proxies rely highly on the input data and labels. The reliance on the input samples underestimates the evaluation of the network's intrinsic characteristics, and the incorrect labels (which are commonly occurred in real-world datasets) may obfuscate the ranking results. (iii) Although various zero-cost proxies have been proposed, the performance of those proxies can still be promoted in network evaluation.

Objectives. The main objective of this paper is to design a novel zero-cost proxy with better performance than the current methods. The proposed proxy should be easily computed by a small number of data without backpropagation. Furthermore, to estimate the intrinsic trait of the architectures and avoid the label influence, our proxy should be label-independent.

Our Contributions. To this end, in this paper, we craft a novel zero-cost proxy for zero-shot NAS from a new perspective. The proposed proxy, called MeCo , outperforms the State-Of-The-Art (SOTA) zero-cost proxies in NAS and only requires one data sample for a single forward pass. Specifically, we harness a novel observation on the multi-channel convolutional layers and view them as the 'multi-sample fully-connected layers with constraints'. This insight permits MeCo to be computed by only one data. Moreover, we rigorously analyze how the Pearson correlation matrix of the feature maps impacts the training convergence rate and the generalization capacity of the networks, which theoretically proves the effectiveness of MeCo . Unlike the existing methods [7-9], MeCo is established upon each layer of the feature maps and hence requires only a single forward pass. Based on our design, we further propose an optimization approach MeCo opt to promote the performance of our method. The experimental results show that both MeCo and MeCo opt achieve a higher correlation with the network ground truth than the SOTA methods. For example, the Spearman correlation coefficients of MeCo on NATS-Bench-TSS with CIFAR-10 and CIFAR-100 are 0.89 and 0.88, respectively, which are 0.09 and 0.08 higher than the best baseline method. Our MeCo -based zero-shot NAS also acquires the highest accuracy among all the existing approaches, e.g., MeCo -based NAS selects the network with 97.31% accuracy, which is 0.04% higher than the SOTA work. In all, our major contributions are as follows.

We theoretically analyze how the Pearson correlation matrix of the feature maps impact the training convergence rate and the generalization capacity of the networks, based on the characteristics of the multi-channel convolutional layers.

We propose MeCo , a novel zero-cost proxy for zero-shot NAS that achieves better performance than SOTA methods. Through systematic analysis, we further proposed MeCo opt as an optimization approach. Our proxy is fully training-free and data-independent, which requires only one data sample with a single forward pass.

We rigorously implement our MeCo and extensively design the experiments to evaluate its performance on several popular benchmarks (e.g., NAS-Bench-101, NATS-Bench-TSS, NATS-Bench-SSS, NAS-Bench-301, and Transbench-101) with diverse datasets and random data samples. The experimental results show that our zero-cost proxy outperforms the existing methods, and MeCo -based NAS can select the network with the highest accuracy.

2 Related work

2.1 Zero-shot NAS and zero-cost proxy

Zero-shot NAS uses specific metrics called zero-cost proxies to evaluate the networks. Unlike the traditional multi-shot and one-shot strategies [10, 6, 11-19], zero-shot NAS eliminates the training procedure thus significantly improves the efficiency. Various studies have been brought forward to

promote the quality of the zero-cost proxies, and most of them are based on the gradients. Mellor et al. [7] firstly evaluated the performance of the initialized networks using the activation overlaps between data points. Abdelfattah et al. [8] proposed a series of parameter-pruning based proxies including snip [20], grasp [21], synflow [22], and fisher [23]. Recently, Li et al. [9] proposed ZiCo , which firstly works better than the parameter amounts. These approaches need at least one backpropagation and rely on the data labels to compute the proxies. Another line of work [24-26] is based on the theory of deep learning such as NTK, which is independent of the labels yet still needs backpropagation. For example, Chen et al. [24] utilized the condition number of NTK to evaluate the trainability of the networks. Finally, there are a few works that do not require backpropagation or data labels. For instance, Lin et al. [27] proposed Zen-Score, in which they design an efficient zero-cost proxy with Gaussian random inputs. However, Zen-Score requires multiple forward passes while our MeCo only needs a single forward pass.

2.2 Over-parameterized networks

Over-parameterized networks have received a lot of attention due to their outstanding effect and ease of optimization. Jacot et al. [28] demonstrated that the training dynamic of an infinite-width network follows a kernel called NTK [29, 30]. For finite-width networks, the training dynamic can be illustrated by a gram matrix [31, 32]. Further, Du et al. [31] proved that the loss of an overparameterized network could converge to a global minimum [33-35], and the training convergence rate could be characterized by the minimum eigenvalue of the gram matrix. Some works also discuss the optimization of the training convergence rate [36] and the topological properties of DNNs [37].

3 MeCo : minimum eigenvalue of correlation on feature maps

3.1 Preliminaries

Notations. We define [ n ] = { 1 , 2 , . . . , n } . The lower and uppercase bold font represent vectors and matrices, respectively, e.g., x is a vector with entry x i , and M is a matrix with entry [ M ] ij . The minimum eigenvalue of M is denoted as λ min ( M ) . We define A ◦ B as Hadamard product between two same-sized matrices A and B . 1 m × n represents a m × n matrix filled by ones and ∥ · ∥ 2 is used to represent the l 2 norm of a vector. N ( 0 , I ) and U { S } represent the standard Gaussian distribution and uniform distribution over a set S , respectively. We denote by X = { ( x i , y i ) | x i ∈ R d × 1 , y i ∈ R , i ∈ [ n ] } the training set, where x i and y i represent the i -th data and label. I {·} is defined as the indicator function that demonstrates the event occurrence, such that for event A , I {A} = 1 if and only if A happened, otherwise it equals to 0.

For input x ∈ R d × 1 , weight vector w ∈ R d × 1 in the weight matrix W ∈ R d × m , and output weight a ∈ R m × 1 , we denote f ( W , a , x ) as a neural network with a single hidden layer such that where σ is the activation function. In this paper, we mainly consider the ReLU function due to its effectiveness, i.e., σ ( z ) = z I { z > 0 } . Given a training set X , the optimization goal is to minimize the empirical risk loss function

We follow the definitions in [31] and define the matrices H ( t ) and H ∞ as follows.

Definition 1 (Gram Matrix) . For a neural network with a single hidden layer, the gram matrix H ( t ) ∈ R n × n induced by the ReLU activation function on a training set X := { ( x i , y i ) } n i =1 with entry [ H ( t )] ij is defined as:

where w r ( t ) is a vector that depends on t . We further construct H ∞ with entry [ H ∞ ] ij such that

We denote ∞

Input

Filters

Output

Inputs

Hidden nodes

Output

Zero Operation

Full Connection

Sum operation

Feature map

̸

Remark 1. Gram matrix H ( t ) reflects the prediction dynamic at the t -th iteration of the network training. Du et al. [31] proved that for all t ≥ 0 , ∥ H ( t ) -H ∞ ∥ 2 → 0 as m →∞ . Moreover, the gram matrix H ∞ has the following key property: if x i ∦ x j , ∀ i = j , then λ 0 > 0 . The proof of this property can be found in Theorem 3.1 in [31].

Definition 2 (Pearson Correlation Matrix) . P ( X ) is a Pearson correlation matrix with the ( i, j ) -th entry [ P ( X )] ij as

where µ x and σ x are the mean and standard deviation of x , respectively.

Remark 2. Pearson correlation matrix P ( X ) is closely related to H ( t ) . For instance, suppose x 1 , x 2 ∼ N ( 0 , I ) , x i ∈ R d × 1 and w T r ( t ) x i ≥ 0 , for r ∈ [ m ] and i ∈ [2] , then [ H ( t )] 12 = ( d -1)[ P ( X )] 12 . We will use this characteristic to approximate the training convergence rate and generation capacity of the networks when designing our proxy.

3.2 The construction of MeCo

To present the theoretical analysis, we first approximate the Convolutional Neural Network (CNN) to an over-parameterized Neural Network (NN) and then craft our proxy.

3.2.1 Back to convolution

The convolutional layers are the fundamental parts of CNN. Suppose we have the input x in ∈ R c in × w × h and the filter w ∈ R c in × k × k . We set the stride size to one and retain the size of the feature maps via zero padding, then the convolutional layer can be formally expressed as:

where x c in ( resp. w c ) represents the c -th channel of x in ( resp. w ), p = ( k -1) / 2 , i ∈ [ w ] , j ∈ [ h ] . In this paper, we observe that x out and x c in can be flattened to one-dimensional vectors ˜ x out ∈ R d × 1 and [˜ x c in ] ∈ R d × 1 , where c ∈ [ c in ] , d = w × h . More concretely, we have the following transformations:

B c ◦ W

where d h = c in × d , W ∈ R d × d h is the weight matrix. Hence, the convolutional layer in CNN is equivalent to a fully-connected layer with constraints, i.e., shown in Equation 8. To further simplify the problem, we relax the second constraint to B c = 1 d h × d and ignore the last constraint. As a consequence, we obtain a multi-sample fully-connected network in which each flattened channel ˜ x c in is regarded as an independent data sample, and the total number of samples is c in . To better illustrate our insight, we present an example of the conversion procedure in Figure 1.

3.2.2 Over-parameterized neural networks

The over-parameterized NNs are competitive in hierarchical feature extraction due to a large number of parameters. One of the typical architectures is the networks with wide hidden layers, which is proved to be tractable in training [38]. In the previous subsection, we convert a multi-channel convolutional layer to a multi-sample fully-connected layer with constraints. We further argue that if the number of hidden nodes in the transformed fully-connected layer is large enough, then it can be viewed as an over-parameterized NN layer. Therefore, the characteristics of the overparameterized NN can be transferred to CNN. To this end, we present the following theorem [31] about the convergence rate of the fully-connected NN with a single hidden layer as follows.

Theorem 1. If gram matrix H ∞ ≻ 0 , ∥ x i ∥ 2 = 1 , | y i | < C for some constant C and i ∈ [ n ] , hidden nodes m = Ω ( n 6 λ 4 0 δ 3 ) , and i.i.d. initialize w r ∼ N ( 0 , I ) , a r ∼ U { [ -1 , 1] } for r ∈ [ m ] , then with probability at least 1 -δ over the initialization, the following inequality holds:

Remark 3. This inequality shows that λ 0 positively affects the training convergence rate of the network. We further point out that the convergence rate of the network is independent of the label y , which makes it possible to measure the network performance with only a single forward pass. Based on this observation, we establish our proxy on the intermediate feature maps to measure the performance of the network.

Based on Theorem 1 and Remark 2, we further analyze the relationship between the minimum eigenvalue of the matrix P ( X ) and the training convergence rate of the over-parameterized network. We summarize our results in the following theorem:

Theorem 2. Suppose f is an NN with a single hidden layer and ReLU activation function. Assume X ∈ R d × n , w (0) ∼ N ( 0 , I ) , P ( X ) ≻ 0 , p 0 := λ min ( P ( X )) , and hidden nodes m = Ω ( n 6 d 2 λ 4 0 δ 3 ) , then the following formula holds with probability at least 1 -δ over the initialization where c is a constant depending on m , and d .

Proof sketch. The key of the proof is to find the relationship between λ 0 and p 0 by Hoeffding's inequality and Weyl inequalities [39]. We provide the full proof in Appendix A.1. Theorem 2 shows that p 0 can reflect the convergence rate of the network to a certain extent. The advantage of P ( X ) over H ( t ) is that its computation only depends on the data. Therefore, it is feasible to estimate the training convergence rate of the network only through the feature maps.

Other than the convergence rate, we also analyze the relationship between p 0 and the generalization capacity of the over-parameterized NN. We present the results in the following theorem:

Theorem 3. For an over-parameterized neural network with the loss on the testing set as L ( W ) . Let y = ( y 1 , ..., y N ) T , and γ be the step of SGD, γ = κC 1 √ y T ( H ∞ ) -1 y / ( m √ N ) for some small enough absolute constant κ . Under the assumption of Theorem 2, for any δ ∈ (0 , e -1 ] , there exists m ∗ ( δ, N, λ 0 ) , such that if m ≥ m ∗ , then with probability at least 1 -δ , we have

where C, C ′ , δ are constants.

Proof sketch. The proof of the above theorem derives from the Corollary 3.10 of [40] and Section D.2 of [26]. We present the detailed proof in Appendix A.2.

3.2.3 New zero-cost proxy: MeCo

The above theorem and analysis indicate the following key insights:

A multi-channel CNN layer can be viewed as a multi-sample over-parameterized NN layer.

The minimum eigenvalue of P ( X ) positively affects the training convergence rate and the generalization capacity of an NN, which is independent of the labels.

Enlightened by these observations, we harness the Minimum eigenvalue of the Pearson Correlation matrix upon each layer of the feature maps to craft our novel zero-cost proxy called MeCo , such that

Definition 3 ( MeCo ) . Assume the NN f ( · ; θ ) has a total of D layers, then MeCo is defined as

where f i ( X ; θ ) represents the i -th feature map with the initialized parameters θ on dataset X .

Remark 4. Note that the Pearson correlation matrix contains the process of data normalization, thus our MeCo eliminates the deviation caused by the data, which is more conducive to discovering the essence of the network architectures.

4 Experiments and evaluations

We evaluate MeCo from three aspects: (i) The correlation with the ground truth and comparison with the baseline proxies; (ii) The dependency on data/labels; (iii) Performance of MeCo -based NAS scheme. We first describe the experimental configurations and then give the detailed performance.

4.1 Experimental configurations

Hardware. We fully implement our MeCo in Python. We evaluate MeCo on a desktop with an Inter Core i7-12700F CPU and GeForce RTX 3090.

Benchmarks. We use several popular benchmarks in NAS. (i) NATS-Bench-TSS [41, 42] contains 15,625 CNN architectures with different topologies. We demonstrate the performance of the zero-cost proxied on NATS-Bench-TSS with CIFAR-10 [43], CIFAR-100 [43], ImageNet16-120[44], and three extra datasets [45] in Appendix E.1. (ii) NATS-Bench-SSS [42] contains 32,768 CNN architectures with the same topology but with different numbers of channels. We show the results on NATS-BenchSSS with CIFAR-10, CIFAR-100, and ImageNet16-120. (iii) NAS-Bench-301 [46] is a surrogate NAS benchmark for the DARTS search space. We illustrate the results with CIFAR-10 and the extra three datasets in Appendix E.1. (iv) TransBench-101 [47] is a benchmark composed of two subsets, i.e., TransBench-101-Micro and TransBench-101-Macro. The former is evaluated with 10 datasets and the latter with 7 datasets. We present these results in Appendix E.3. (v) NAS-Bench-101 [48] includes 423,624 cell-based CNN architectures, which are trained and evaluated on the CIFAR-10 dataset and shown in Appendix E.2. (vi) AutoFormer [49] and MobileNet OFA [50]. The description and results are shown in Appendix E.4.

Baselines. The baselines for zero-cost proxies include the number of parameters # Param , grasp [21], fisher [23], snip [20], synflow [22], jacov [7], grad _ norm [8], NTK [24], zen [27], NASWOT [7], KNAS [51], NASI [52], GradSign [53], and ZiCo [9]. Specific forms of these proxies are provided in Appendix B. We adopt the Spearman rank correlation coefficient ρ between zero-cost proxies and the dataset ground truth to measure their performance.

4.2 Performance of our MeCo and comparisons

We evaluate ρ between MeCo and the ground truth, and compare it with the baseline zero-cost proxies on various benchmarks. We present the results on NATS-Bench-TSS and NATS-Bench-SSS in Table 1, and the results on NAS-Bench-301 are provided in Table 2. We run MeCo for ten times with different inputs and present the mean and standard deviation.

NATS-Bench-TSS. We run our MeCo and all baseline proxies on 15,625 networks of NATS-BenchTSS with three public datasets. As we can see in Table 1, MeCo achieves 0.894 ± 0.003, 0.883 ± 0.005,

±

±

±

±

±

±

and 0.845 ± 0.004 correlation on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively. The variation shows that our method is stable and independent of the input samples. The Spearman correlation coefficients are significantly higher than the baselines, e.g., the SOTA proxy ZiCo [9] only achieves 0.80, 0.81, 0.79 on three datasets, which is 0.06 to 0.09 lower than ours. Note that ZiCo is the first proxy that is proved to be consistently better than the naive proxy # Param and requires a batch of input data. On the other hand, we surpass all the previous works and only use one data sample. The high correlation coefficients demonstrate the effectiveness of MeCo , which is consistent with the theoretical analysis. To make our results more intuitive, we sample 2,000 networks randomly and present the relationship between MeCo and the ground truth on three public datasets. The results are shown in Figure 4(a) in the Appendix E, where the trend in the scatter charts effectively proves that our zero-cost proxy has a highly strong correlation with the architecture quality.

NATS-Bench-SSS. The results of MeCo on 32,768 networks of NATS-Bench-SSS are summarized in Table 1. Among all the datasets, MeCo shows the highest correlation with the ground truth on CIFAR100, i.e., ρ = -0 . 87 ± 0 . 01 , which is 0.07 higher than the best baseline synflow . MeCo also achieves stable and competitive results on CIFAR-10 and ImageNet16-120 with one data as input. We find that MeCo is highly negatively correlated with the test accuracy, which is somehow counterintuitive. In fact, MeCo is sensitive to the number of channels (#channels) that cause this phenomenon. Specifically, the channels are viewed as the data samples as described in Section 3.2.1, This leads to the fact that the more channels contained, the larger the Pearson matrix, which will lead to smaller p 0 . On the other hand, larger #channels will normally lead to higher accuracy of the architectures (yet lower

0

200

400

0

.

2

0

.

4

0

.

6

0

.

8

1

Number of Channels

Minimum Eigenvalue

Channels v.s. Eigenvalue

convergence rate). Thus, the variation of #channels in NATS-Bench-SSS(chosen from 8 to 64) leads to a negative correlation on this benchmark. In NATS-Bench-TSS, however, all the networks share the same #channels at each stage, thus MeCo does not show a negative correlation. To verify this, we generate random data with different numbers of channels and calculated p 0 respectively, and demonstrate the results in Figure 2. We also sample 2,000 networks randomly and present the correlation of MeCo v.s. test accuracy in Figure 4(b) in the Appendix E.

NATS-Bench-301 . We demonstrate the results of MeCo and the baseline proxies on NATS-Bench301 with CIFAR-10 in Table 2. It can be shown from the results that MeCo and MeCo opt also achieve the highest correlation with the testing accuracy on NATS-Bench-301, which is 0.04-0.05 higher than the SOTA approach ZiCo . More experimental results of MeCo can be found in Appendix E

±

±

4.3 Dependency on data

MeCo is designed to be fully data-independent due to the characteristics of Pearson correlation matrix (Theorem 2). To further illustrate this attribute, we generate a random dataset and use it as the input of MeCo and NTK [24], which is also independent of the data and labels. We compute the Spearman correlation coefficients between MeCo and the test accuracy on three public datasets, and compare them with the baseline approach. All the results for MeCo is ran ten times with random inputs.

More concretely, the random data is generated following a Gaussian distribution and fed into the initialized network. Note that the baseline method requires a batch of random data as input, while MeCo only uses one. We present the results for NATS-Bench-TSS and NATS-Bench-SSS in Table 3.

The results show that even if we use random data as the inputs, MeCo is still able to maintain a high correlation with the corresponding ground truth. For example, ρ = 0 . 87 on NATS-BenchTSS with CIFAR-10 and ρ = -0 . 93 on NATSBench-SSS with ImageNet16-120, which are significantly higher than the data-independent proxy NTK . Note that most existing proxies rely on the gradients and backpropagation. Hence they cannot utilize the random input data for architecture evaluation, such as ZiCo and jacov .

4.4 NAS with MeCo

We integrate our MeCo with the existing search space and generation strategy to comprise a complete NAS method. Specifically, we use NATSBench-TSS, DARTS-CNN [6], and MobileNet V3 [54] as the search spaces to demonstrate the performance. We use the Zero-Cost-PT [55] as the generation strategy. The algorithm of MeCo -based NAS and the results on MobileNet V3 are presented in Appendix C and F.3, respectively.

NATS-Bench-TSS. We directly evaluate all networks in NATS-Bench-TSS using MeCo and baseline proxies. The average test accuracy of the Top-10 networks on three datasets is summarized in Table 12 in Appendix F.1. As shown in the table, the average test accuracy on CIFAR-10 of MeCo is 0 . 08% higher than the best baseline

±

±

16

32

64

128

256

0

.

83

0

.

84

0

.

85

0

.

86

0

.

87

0

.

88

0

.

89

data size

ρ

NATS-Bench-TSS

16

32

64

128

256

-

0

.

9

-

0

.

8

-

0

.

7

-

0

.

6

-

0

.

5

data size

ρ

NATS-Bench-SSS

CIFAR-10;

CIFAR-100;

ImageNet16-120

ZiCo . Moreover, we further search the networks using Zero-Cost-PT with different proxies and run the search algorithm three times. The networks we searched have the highest average test accuracy on three public datasets, that is, 93.76%, 71.11%, and 41.44% on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively. The results are provided in Table 13 in Appendix F.1.

DARTS-CNN. We compare our MeCo -based NAS with multi-shot, one-shot, and zero-shot NAS schemes in DARTS-CNN spaces. All the NAS schemes are executed on CIFAR-10 and CIFAR-100

datasets, respectively, to compare the Top-1 errors and the search cost. For zero-shot NAS, we use Zero-Cost-PT as the generation strategy due to its effectiveness. The results on CIFAR-10 are summarized in Table 4, where we denote multi-shot, one-shot, and zero-shot as MS, OS, and ZS, respectively. We present the CIFAR-100 results in Appendix F.2. The results demonstrate that our MeCo -based NAS outperforms the existing baseline methods. More concretely, for zero-shot methods, MeCo outperforms all the baselines for 0.04% to 0.43% on Top-1 error with a similar level of search cost under the same generation strategy. For training-based methods, MeCo -based NAS achieves competitive accuracy but with much less search cost. For instance, our approach requires only 0.08 GPU days and obtains 97.31% accuracy, while DARTS-PT costs 10 × GPU days than ours with 0.08% higher accuracy.

We remark that the experimental results in this section demonstrate the following properties of MeCo : (i) High correlation with the ground truth; (ii) Requies only one data without labels for a single forward pass. That is because MeCo is established upon the observations over the multi-channel CNN layers. Moreover, the minimum eigenvalue of the Pearson correlation matrix is independent of the data labels, and thus we exert MeCo on the feature maps instead of the gradients.

5 Discussion and Optimization

5.1 Optimization method MeCo opt

In the previous sections, we illustrated the theoretical basis of MeCo and evaluated our proposed proxy on various benchmarks. However, as described in Section 4.2, the negative correlation on NATS-Bench-SSS and some other tasks (e.g., Transbench-101 in the appendix) reflect that MeCo is highly sensitive to the number of channels. Although this phenomenon does not contradict the theoretical results or undermine the effectiveness of MeCo , it might be problematic when p 0 = 0 . To this end, we present an optimization method on top of MeCo to alleviate the channel-sensitive trait.

̸

Specifically, for convolutional layers, p 0 is strictly greater than zero if for ∀ i = j , x i ∦ x j , and c < w × h , where c is the number of channels, w,h are the size of the inputs. Thus, in real-world applications, MeCo might lose efficacy during the down-sampling procedure. To address this issue, instead of flattening all the channels of the feature map as described in Section 3.2.1, we randomly sample a fixed number of channels and flatten them as matrix P ′ . We then compute the final result by multiplying the minimum eigenvalue of P ′ with a channel weight, such that

where c ( i ) is the number of channels in the i -th layer, and n is the fixed sampling numbers. The high-level idea of this optimization is to limit the dimension of the Pearson correlation matrix by constraining #channels. Instead of computing upon a large matrix, we calculate the minimum eigenvalue upon a fixed-sized matrix and then enlarge it with corresponding constants. Note that the bonus of this optimization is that the time cost can be controlled because the matrix dimension is significantly reduced compared to the original one.

To show the effectiveness of MeCo opt , we calculate the Spearman correlation coefficients of MeCo opt on NATS-Bench-TSS, NATS-Bench-SSS (Table 1), NATS-Bench-301 (Table 2 and Table 8), and Transbench-101 (Table 10 and Table 11). All the experiments are conducted under n = 8 . The results demonstrate that MeCo opt effectively solves the negative correlation issue on multiple benchmarks, e.g., NATS-Bench-SSS and TransBench-101. Meanwhile, the majority of the positive correlations are promoted up to 0.23, and the remaining results are not impacted.

5.2 Ablation study

Sample size . We consider the effect of the sample size on the performance of MeCo . We select the size of data from { 16 , 32 , 64 , 128 , 256 } , respectively, and calculate the Spearman rank correlation coefficient ρ between MeCo and the test accuracy of the networks on three public datasets. The results are summarized in Figure 3. For NATS-Bench-TSS, ρ increases rapidly before size 64 and grows slowly after that. The maximum value is acquired at size 128, which is slightly higher than size 64. On the other hand, the overall fluctuation is relatively large on NATS-Bench-SSS. The Spearman correlation coefficients are low when the size is smaller than 32, and reach higher at 32. Then ρ reduces as the sample size grows larger. The ablation results show that the most appropriate size of random data is around 32 and 64, which is also adopted in our settings.

Sample numbers . We explore the performance of MeCo regarding the sample numbers that are used as the inputs. We randomly choose a different number of samples to evaluate the architectures in the entire NATS-Bench-TSS and NATS-Bench-SSS, respectively. The results are shown in Figure 5. It can be demonstrated from the figure that MeCo is relatively stable when varying the number of samples. The performance of MeCo fluctuates around 0.01 throughout the experiments. These results support that MeCo only requires one data sample.

Fixed channel numbers n . We conduct an ablation study on n v.s. MeCo opt with NATS-BenchTSS and NATS-Bench-SSS to illustrate the best choice of n . The results are shown in Table 5.

The results show that with n growing larger, MeCo opt increases by 0.1 to 0.5. We set n = 8 in our experiments to obtain the best performance.

6 Conclusion

In this work, we propose a novel feature-based zero-cost proxy called MeCo and its optimization method MeCo opt . Unlike the existing methods, our zero-cost proxies require only one data for a single forward pass. Specifically, we theoretically approximate a multi-channel convolution layer to an over-parameterized NN layer. We then harness the relationship between the theoretical properties and the minimum eigenvalue of the Pearson correlation matrix to craft our new proxies. We rigorously implement our proxies and extensively design the experiments. The experimental results show that MeCo and MeCo opt significantly outperforms the SOTA zero-cost proxies, which is also independent of the data and labels. Moreover, MeCo can be integrated into a complete NAS and enables us to efficiently find the architecture with the highest accuracy.

Acknowledgment

We sincerely thank all the reviewers of NeurIPS'23 for their constructive suggestions which significantly improve the quality of our paper. The work was supported by National Key R&D Program of China [Grant No. 2022ZD0115901], in part by the National Natural Science Foundation of China (Grant No. 62177007), and the China-Central Eastern European Countries High Education Joint Education Project (Grant No. 202012).

References

K. Simonyan and A. Zisserman, 'Very deep convolutional networks for large-scale image recognition,' in International Conference on Learning Representations , 2015.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770-778.

J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, 'You only look once: Unified, real-time object detection,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 779-788.

S. Hochreiter and J. Schmidhuber, 'Long short-term memory,' Neural computation , vol. 9, no. 8, pp. 1735-1780, 1997.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

H. Liu, K. Simonyan, and Y. Yang, 'DARTS: Differentiable architecture search,' in International Conference on Learning Representations , 2019.

J. Mellor, J. Turner, A. Storkey, and E. J. Crowley, 'Neural architecture search without training,' in International Conference on Machine Learning . PMLR, 2021, pp. 7588-7598.

M. S. Abdelfattah, A. Mehrotra, Ł. Dudziak, and N. D. Lane, 'Zero-cost proxies for lightweight nas,' in International Conference on Learning Representations , 2021.

G. Li, Y. Yang, K. Bhardwaj, and R. Marculescu, 'Zico: Zero-shot NAS via inverse coefficient of variation on gradients,' in The Eleventh International Conference on Learning Representations , 2023.

B. Zoph and Q. Le, 'Neural architecture search with reinforcement learning,' in International Conference on Learning Representations , 2017.

L. Xie and A. Yuille, 'Genetic cnn,' in 2017 IEEE International Conference on Computer Vision (ICCV) , 2017, pp. 1388-1397.

B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, 'Learning transferable architectures for scalable image recognition,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 8697-8710.

E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, 'Large-scale evolution of image classifiers,' in International Conference on Machine Learning . PMLR, 2017, pp. 2902-2911.

C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, 'Auto-weka: Combined selection and hyperparameter optimization of classification algorithms,' in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , 2013, pp. 847-855.

Y. Geifman and R. El-Yaniv, 'Deep active learning with a neural architecture search,' Advances in Neural Information Processing Systems , vol. 32, 2019.

T. Elsken, J. H. Metzen, and F. Hutter, 'Efficient multi-objective neural architecture search via lamarckian evolution,' in International Conference on Learning Representations , 2019.

M. Javaheripi, G. de Rosa, S. Mukherjee, S. Shah, T. Religa, C. C. Teodoro Mendes, S. Bubeck, F. Koushanfar, and D. Dey, 'Litetransformersearch: Training-free neural architecture search for efficient language models,' Advances in Neural Information Processing Systems , vol. 35, pp. 24 254-24 267, 2022.

I. Fedorov, R. Matas, H. Tann, C. Zhou, M. Mattina, and P. Whatmough, 'Udc: Unified dnas for compressible tinyml models for neural processing units,' Advances in Neural Information Processing Systems , vol. 35, pp. 18 456-18 471, 2022.

X. Wang, W. Guo, J. Su, X. Yang, and J. Yan, 'Zarts: On zero-order optimization for neural architecture search,' in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 12 868-12 880.

N. Lee, T. Ajanthan, and P. Torr, 'Snip: Single-shot network pruning based on connection sensitivity,' in International Conference on Learning Representations , 2019.

C. Wang, G. Zhang, and R. Grosse, 'Picking winning tickets before training by preserving gradient flow,' in International Conference on Learning Representations , 2020.

H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli, 'Pruning neural networks without any data by iteratively conserving synaptic flow,' Advances in Neural Information Processing Systems , vol. 33, pp. 6377-6389, 2020.

J. Turner, E. J. Crowley, M. O'Boyle, A. Storkey, and G. Gray, 'Blockswap: Fisher-guided block substitution for network compression on a budget,' in International Conference on Learning Representations , 2020.

W. Chen, X. Gong, and Z. Wang, 'Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective,' in International Conference on Learning Representations (ICLR) , 2021.

Y. Shu, Z. Dai, Z. Wu, and B. K. H. Low, 'Unifying and boosting gradient-based training-free neural architecture search,' in Advances in Neural Information Processing Systems , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022.

Z. Zhu, F. Liu, G. Chrysos, and V. Cevher, 'Generalization properties of nas under activation and skip connection search,' Advances in Neural Information Processing Systems , vol. 35, pp. 23 551-23 565, 2022.

M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin, 'Zen-nas: A zero-shot nas for high-performance image recognition,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 347-356.

A. Jacot, F. Gabriel, and C. Hongler, 'Neural tangent kernel: Convergence and generalization in neural networks,' Advances in neural information processing systems , vol. 31, 2018.

S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang, 'On exact computation with an infinitely wide neural net,' Advances in neural information processing systems , vol. 32, 2019.

J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, 'Wide neural networks of any depth evolve as linear models under gradient descent,' Advances in neural information processing systems , vol. 32, 2019.

S. S. Du, X. Zhai, B. Poczos, and A. Singh, 'Gradient descent provably optimizes overparameterized neural networks,' in International Conference on Learning Representations , 2019.

B. Hanin and M. Nica, 'Finite depth and width corrections to the neural tangent kernel,' in International Conference on Learning Representations , 2020.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai, 'Gradient descent finds global minima of deep neural networks,' in International conference on machine learning . PMLR, 2019, pp. 1675-1685.

D. Zou and Q. Gu, 'An improved analysis of training over-parameterized deep neural networks,' Advances in neural information processing systems , vol. 32, 2019.

Q. N. Nguyen and M. Mondelli, 'Global convergence of deep networks with one wide layer followed by pyramidal topology,' Advances in Neural Information Processing Systems , vol. 33, pp. 11 961-11 972, 2020.

N. Lee, T. Ajanthan, S. Gould, and P. H. S. Torr, 'A signal propagation perspective for pruning neural networks at initialization,' in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=HJeTo2VFwH

K. Bhardwaj, G. Li, and R. Marculescu, 'How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections?' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 13 49813 507.

H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, 'Visualizing the loss landscape of neural nets,' Advances in neural information processing systems , vol. 31, 2018.

R. A. Horn and C. R. Johnson, Matrix analysis . Cambridge university press, 2012.

Y. Cao and Q. Gu, 'Generalization bounds of stochastic gradient descent for wide and deep neural networks,' Advances in neural information processing systems , vol. 32, 2019.

X. Dong and Y. Yang, 'Nas-bench-201: Extending the scope of reproducible neural architecture search,' in International Conference on Learning Representations , 2020.

X. Dong, L. Liu, K. Musial, and B. Gabrys, 'Nats-bench: Benchmarking nas algorithms for architecture topology and size,' IEEE transactions on pattern analysis and machine intelligence , 2021.

A. Krizhevsky, G. Hinton et al. , 'Learning multiple layers of features from tiny images.' Toronto, ON, Canada, 2009.

P. Chrabaszcz, I. Loshchilov, and F. Hutter, 'A downsampled variant of imagenet as an alternative to the cifar datasets,' arXiv preprint arXiv:1707.08819 , 2017.

A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, 'Taskonomy: Disentangling task transfer learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3712-3722.

A. Zela, J. N. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, 'Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks,' in International Conference on Learning Representations , 2022.

Y. Duan, X. Chen, H. Xu, Z. Chen, X. Liang, T. Zhang, and Z. Li, 'Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5251-5260.

C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, 'Nas-bench-101: Towards reproducible neural architecture search,' in International Conference on Machine Learning . PMLR, 2019, pp. 7105-7114.

M. Chen, H. Peng, J. Fu, and H. Ling, 'Autoformer: Searching transformers for visual recognition,' in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 12 270-12 280.

H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, 'Once-for-all: Train one network and specialize it for efficient deployment,' in International Conference on Learning Representations , 2020. [Online]. Available: https://openreview.net/forum?id=HylxE1HKwS

J. Xu, L. Zhao, J. Lin, R. Gao, X. Sun, and H. Yang, 'Knas: green neural architecture search,' in International Conference on Machine Learning . PMLR, 2021, pp. 11 613-11 625.

Y. Shu, S. Cai, Z. Dai, B. C. Ooi, and B. K. H. Low, 'NASI: Label- and data-agnostic neural architecture search at initialization,' in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=v-v1cpNNK_v

Z. Zhang and Z. Jia, 'Gradsign: Model performance inference with theoretical insights,' in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=HObMhrCeAAF

A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al. , 'Searching for mobilenetv3,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1314-1324.

L. Xiang, Ł. Dudziak, M. S. Abdelfattah, T. Chau, N. D. Lane, and H. Wen, 'Zero-cost proxies meet differentiable architecture search,' arXiv preprint arXiv:2106.06799 , 2021.

E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, 'Regularized evolution for image classifier architecture search,' in Proceedings of the aaai conference on artificial intelligence , vol. 33, no. 01, 2019, pp. 4780-4789.

C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, 'Progressive neural architecture search,' in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 19-34.

R. Wang, M. Cheng, X. Chen, X. Tang, and C.-J. Hsieh, 'Rethinking architecture selection in differentiable nas,' in International Conference on Learning Representation , 2021.

X. Chen and C.-J. Hsieh, 'Stabilizing differentiable architecture search via perturbation-based regularization,' in International conference on machine learning . PMLR, 2020, pp. 1554-1565.

G. Li, G. Qian, I. C. Delgadillo, M. Muller, A. Thabet, and B. Ghanem, 'Sgas: Sequential greedy architecture search,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

W. Hoeffding, 'Probability inequalities for sums of bounded random variables,' The collected works of Wassily Hoeffding , pp. 409-426, 1994.

G. H. Golub and C. F. Van Loan, Matrix computations . JHU press, 2013.

Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, 'Pc-darts: Partial channel connections for memory-efficient architecture search,' in International Conference on Learning Representations , 2020.

X. Chen, L. Xie, J. Wu, and Q. Tian, 'Progressive differentiable architecture search: Bridging the depth gap between search and evaluation,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1294-1303.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, 'Densely connected convolutional networks,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4700-4708.

S. Xie, H. Zheng, C. Liu, and L. Lin, 'SNAS: stochastic neural architecture search,' in International Conference on Learning Representations , 2019.

A. Zela, T. Elsken, T. Saikia, Y . Marrakchi, T. Brox, and F. Hutter, 'Understanding and robustifying differentiable architecture search,' in International Conference on Learning Representations , 2020.

P. Ye, B. Li, Y. Li, T. Chen, J. Fan, and W. Ouyang, ' β -darts: Beta-decay regularization for differentiable architecture search,' in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2022, pp. 10 864-10 873.

A Detailed proofs

A.1 Proof of Theorem 2

In this section, we present the proof for our main theorem (Theorem 2 in Section 3.4). We first introduce the following definitions and lemmas to better illustrate our proof.

Lemma 1 (Lemma 3.1 in [31]) . If m = Ω ( n 2 λ 2 0 log ( n δ ) ) , then with probability at least 1 -δ , ∥ H (0) -H ∞ ∥ 2 ≤ λ 0 4 and 3 4 λ 0 ≤ λ min ( H (0)) ≤ 5 4 λ 0 .

Definition 4. We denote the empirical indicator matrix as W (0) with entry [ W (0)] ij such that

and indicator matrix as ¯ W with entry [ ¯ W ] ij

The indicator matrix ¯ W has the following property.

Proposition 1. If x i ∦ x j , then [ ¯ W ] ij > 0 .

Proof. If Proposition 1 does not hold, then ∃ i 0 , j 0 ∈ [ n ] × [ n ] , satisfying

Thus x i 0 = -x j 0 , which contradicts the hypothesis x i 0 ∦ x j 0 .

■

In real-world datasets, the possibility of two different data being parallel is slight. Thus, proposition 1 holds in general. We further introduce Weyl inequality as follows:

Lemma 2 (Weyl inequality [39]) . Let A , B ∈ R n × n be Hermitian matrices, and let the eigenvalues of A , B , and A + B be { λ i ( A ) } n i =1 , { λ i ( B ) } n i =1 and { λ i ( A + B ) } n i =1 , respectively. The eigenvalues of each matrix are arranged in ascending order. Then we have

for each i = 1 , . . . , n , with equality for some pair i, j if and only if there is a nonzero vector x such that Ax = λ i + j ( A ) x , Bx = λ n -j ( B ) x , and ( A + B ) x = λ i ( A + B ) x . Also,

for each i = 1 , . . . , n , with equality for some pair i, j if and only if there is a nonzero vector x such that Ax = λ i -j +1 ( A ) x , Bx = λ j ( B ) x , and ( A + B ) x = λ i ( A + B ) x . If A and B have no common eigenvector, then inequality (14) and (15) are strict inequality.

Now, we provide the full proof of Theorem 2.

Theorem 2. Suppose f is an NN with a single hidden layer and ReLU activation function. Assume X ∈ R d × n , w (0) ∼ N ( 0 , I ) , P ( X ) ≻ 0 , p 0 := λ min ( P ( X )) , and hidden nodes m = Ω ( n 6 d 2 λ 4 0 δ 3 ) , then the following formula holds with probability at least 1 -δ over the initialization

where c is a constant depending on m and d .

Proof. To simplify our proof, we assume µ x = 0 , σ x = 1 , and ∥ x ∥ 2 ≤ C for all x ∈ X . Recall that

Due to I { w T r (0) x i ≥ 0 , w T r (0) x j ≥ 0 } is an independent random variable between 0 and 1, then by Hoeffding's inequality [61], the following inequality holds with probability 1 -δ :

Let µ 0 = min ( i,j ) ∈ [ n ] × [ n ] [ ¯ W ] ij , and choose m> 16 log (1 /δ ) µ 2 0 , then we have

Define the matrix M with entry [ M ] ij as

where c ( m,d ) = c ( m )( d -1) . We claim that if m is large enough, then M is positive definite. To clarify this statement, we consider the gap between H (0) and M :

If we choose m> 64 log (1 /δ )( d -1) 2 n 4 C 4 λ 2 0 , we have

Then the following formula holds by matrix perturbation theory (Corollary 6.3.8 in [39])

which indicates that M is positive definite. We next exert Lemma 2 on M and get

That means

Therefore, combined with Theorem 1, we have

where c ( m,d ) is a constant depending on m and d . This completes the proof of Theorem 2 . ■

A.2 Proof of Theorem 3

Proof. By Corollary 3.10 and Remark 3.11 in [40], we have

According to the courant minimax principle[62], D.2 in [26], and inequality 16, we get

Thus, we have

B Zero-cost proxies

In this section, we provide the details of the baseline zero-cost proxies. Suppose L is the loss function and θ is the parameters of an initialized network. We denote ◦ as the Hadamard product. The concrete formulations of the existing zero-cost proxies are as follows.

snip . Lee et al. [20] use the changes in loss caused by the parameter perturbations to measure the importance of the parameters in an initialized network, such that

grasp . Wang et al. [21] replace the loss change in snip with the change of the gradient norm to establish the proxy, such that

where H is the Hessian.

synflow . To avoid layer collapse, Tanaka et al. [22] utilize the product of all parameters in the network during the parameter perturbation to represent the loss, such that

grad _ norm . Abdelfattah et al. [8] adopt the l 2 norm of the gradients in an initialized work as a proxy, such that

jacov / NWOT . Mellor et al. [7] use the correlation of activations within a network as a proxy to evaluate the performance of the network, such that

■

where N A is the number of rectified linear units, d H ( c i , c j ) represents the Hamming distance between two binary codes c i and c j .

NTK . Chen et al. [24] propose to use the condition number of NTK to measure the trainability of the networks, such that

where ˆ Θ train stands for NTK of the networks. In our paper, we calculate the Spearman correlation coefficient between 1 /κ N and the test accuracy of the networks.

zen . Lin et al. [27] propose Zen-Score, in which they design an efficient zero-cost proxy with Gaussian random inputs, such that

where σ i,j is the mini-batch standard deviation statistic of the j -th channel in BN.

NASI . Shu et al. [52] propose NASI to evaluate the networks by approximating the trace of the NTK, such that

where X j is a mini-batch of data with size |X j | = b .

KNAS . Xu et al. [51] propose to use gradient kernel to evaluate the networks, such that

where y L is the output of L -th layer.

GradSign . Zhang and Jia [53] propose GradSign , in which they analyze the sample-wise optimization landscape of the networks, such that

ZiCo . Li et al. [9] explore the effect of gradient properties on network performance. They use absolute mean and standard deviation values of gradients to evaluate network performance:

where N and D represent the number of batches and network layers, respectively. θ l represents the parameters of the l -th layers.

C Algorithm of MeCo -based NAS

We slightly abuse the notation and denote MeCo as our proxy function for the network. We adopt the Zero-Cost-PT [55] to integrate MeCo to a zero-shot NAS. The algorithm of our MeCo -based NAS is summarized in Algorithm 1. We denote A 0 as an untrained supernet, e t represents the t -th edge, which stands for mixed operation in the search cells, and e t,k is the k -th operation of t -th edge. We denote E , N , and O as the set of edges, nodes, and candidate operations in the search cells, respectively. For any node n ∈ N , we use E ( n ) as the set of its input edges, and e k n is the k -th element of E ( n ) . Note that we can use other zero-cost proxies as described in section B to replace MeCo in Algorithm 1.

Algorithm 1 MeCo -based NAS

D Experimental configurations

We use the same settings for the experiments as in [55]. We summarized the configurations of the searching phase and training phase on CIFAR-10 and CIFAR-100 in Table 6 and Table 7, respectively. For ZiCo , we use two mini-batch of data, which has a size of 64. For the other baseline proxies, we use one mini-batch of data. On the other hand, our MeCo only uses one random data x ∈ R 1 × 3 × 32 × 32 .

E More experimental results of ρ

E.1 MeCo on NASBench-201 and NASBench-301 with three extra datasets

We evaluate MeCo and MeCo opt on NASBench-201 and NASBench-301 with Spherical-CIFAR-100, NinaPro, and SVHN, respectively. We only use one random data x ∈ R 3 × 32 × 32 for MeCo and MeCo opt . The results are summarized in Table 8. Our MeCo achieves competitive results on most benchmarks and achieves the best results on NASBench-201-SVHN, where the Spearman correlation coefficient is ρ = 0 . 88 .

0

20

40

60

80

20

40

60

80

Test acc

CIFAR-10 (0.89)

0

20

40

60

80

0

20

40

60

80

Test acc

CIFAR-100 (0.88)

0

20

40

0

20

40

Test acc

ImageNet16-120 (0.85)

(a) Performance on NATS-Bench-TSS

10

20

30

85

90

Test acc

CIFAR-10 (-0.80)

0

10

20

40

50

60

70

Test acc

CIFAR-100 (-0.89)

10

20

30

20

30

40

50

Test acc

ImageNet16-120 (-0.88)

(b) Performance on NATS-Bench-SSS

E.2 MeCo on NAS-Bench-101

We present the Spearman correlation coefficient between all zero-cost proxies and the test accuracy of the networks in NAS-Bench-101. Our MeCo uses one data of CIFAR-10, i.e., x ∈ R 1 × 3 × 32 × 32 while the baselines adopt a batch of samples as input. The results are summarized in Table 9.

MeCo achieves competitive results on NAS-Bench-101 with CIFAR-10, i.e., the Spearman correlation coefficient obtains 0.44, which is higher than the majority of the baselines. However, zen and ZiCo perform better than MeCo .

E.3 MeCo on TransBench-101

We further evaluate our MeCo on diverse tasks. Specifically, we compare MeCo and the other proxies on Transbench-101-Micro and Transbench-101-Macro [47], respectively.

TransBench-101-Micro. We evaluate MeCo and MeCo opt on TransBench-101-Micro with ten tasks. We calculate the Spearman correlation between our proxies and the accuracy of the networks. The results are summarized in Table 10. MeCo achieves the best performance on the three tasks, i.e., Class Objection, Spherical-Cifar100, and NinaPro, which are 0.58, 0.85, and 0.47, respectively. It can be seen that MeCo opt effectively improves the performance of MeCo . For example, MeCo opt improves MeCo from 0.62 to 0.77 on segmentation tasks.

TransBench-101-Macro. We compare MeCo , MeCo opt and the baselines on TransBench-101-Macro with seven tasks. The results are summarized in Table 11. Our proxies achieve the best performance on Autoencoding, Jigsaw, and Surface Normal, which are 0.74, 0.48, and 0.80, respectively. Our proxies also achieve competitive results on the remaining tasks. Experimental results show that

our proxies can be used for diverse tasks. It can be demonstrated that MeCo and MeCo opt become ineffective on a few tasks, such as Room Layout. We would like to note that the existing proxies do not achieve a high correlation on all tasks consistently.

E.4 MeCo on AutoFormer and MobileNet OFA

AutoFormer. Chen et al. [49] proposed a novel one-shot architecture search framework for transformer-based models. We load the trained supernets and regenerate the candidate subnets. We then re-evaluate the subnets to obtain the accuracy on ImageNet and compute MeCo . The correlation of MeCo and model accuracy on AutoFormer is 0.45.

OFA. To solve the problem of efficient inference across devices and resource constraints, Cai et al. [50] proposed to train a once-for-all (OFA) network, which supports diverse architectural settings. We randomly sample 1,000 subnets from the OFA network and use the accuracy of the predictor predictions as the test accuracy. The Spearman correlation between MeCo and test accuracy is 0.86.

F More experimental results of NAS with MeCo

In this section, we provide more results and comparisons of our zero-shot NAS on NATS-Bench-TSS and DARTS-CNN. In the following descriptions, we denote multi-shot, one-shot, and zero-shot as MS, OS, and ZS, respectively.

F.1 Results on NATS-Bench-TSS

In the NATS-Bench-TSS search space, we evaluate all networks using zero-cost proxies and choose the Top-10 networks with the highest scores. Then we calculate the average test accuracy and standard deviation. The results are summarized in Table 12. As shown in the table, the networks searched by MeCo have the highest average precision on CIFAR-10, which is 0.08 higher than the best baseline proxy ZiCo . For CIFAR-100 and ImageNet16-120, MeCo achieves competitive results,

e.g., 70 . 86% ± 0 . 96% with CIFAR-100 and 42 . 59% ± 1 . 77% with ImageNet16-120. In all, the results demonstrate that our MeCo has a great advantage in evaluating network performance, considering MeCo only requires one data sample as input.

We further combine MeCo with Zero-Cost-PT [55] and search for the best architecture three times with different seeds. The accuracy of the selected architectures as well as the comparisons with the baseline methods are presented in Table 13. It can be shown from the results that our MeCo -based NAS achieves competitive results with the SOTA baselines, e.g., synflow , zen , and ZiCo . However, MeCo -based NAS invokes one data for a single forward pass, thus being more resource-saving.

±

±

±

±

±

±

±

±

±

±

±

±

±

F.2 Results on DARTS-CNN

For DARTS-CNN search space, we search the architectures by Zero-Cost-PT with different zero-cost proxies on CIFAR-100. Each searched network is trained five times, and the results are summarized in Table 14. There are two settings in our experiments on DARTS-CNN: networks initialized with 16 channels and trained as in Table 7, and networks initialized with 36 channels and trained as in [63]. It can be shown from Table 14 that MeCo achieves competitive results compared with MS, OS, and ZS baselines. More concretely, MeCo -based NAS obtains 19 . 33% test error with 0.08 GPU days, which outperforms all the ZS methods under the same settings. On the other hand, compared with the manual, MS, and OS methods, MeCo is also competitive. For example, MeCo -based NAS achieves 83.14% accuracy, which is only 0.34% lower than the baseline method β -DARTS [64], but five times more efficient in computation.

We further visualize the networks searched by Zero-Cost-PT with different proxies on DARTS-CNN. The results on CIFAR-10 and CIFAR-100 are presented in Figure 6 and Figure 7, respectively.

F.3 Results on MobileNet V3

We search the architectures on MobileNet space using Algorithm 1. We retrain the searched network using ImageNet-1K for 480 epochs with a batch size of 256 and input resolution 224 × 224 . The results are summarized in Table 15. MeCo achieves 77 . 8% Top-1 accuracy, which is 0 . 7% higher than the SOTA baseline method ZiCo . Moreover, our method only takes 0.04 GPU days, which outperforms the SOTA methods.

G Limitations and future works

±

±

±

±

±

±

±

±

3

4

10

0

20

(b) Performance on NATS-Bench-SSS

20

Figure 2: Relationship between our zero-cost proxies and test accuracy for NATS-Bench-TSS and

NATS-Bench-SSS on three datasets.

1

2

3

4

5

-

0

.

9

-

0

.

88

-

0

.

86

-

0

.

84

-

0

.

82

-

0

.

8

-

0

.

78

ρ

NATS-Bench-SSS

CIFAR-100;

ImageNet16-120

MeCo with different #Samples on TSS and SSS,

Conclusion

ResultsinRebuttal

Table 4: The test accuracy of Top-10 architectures obtained by various zero-cost proxies on NATS-

Baselines

]

synflow

93

.

27

±

0

.

74

Bench-TSS using CIFAR-10

Dataset

[

?

CIFAR-10

93

.

64

±

0.31

89

.

±

2

34

.

16

fisher

89

.

27

±

2

.

10

]

1 2 3 4 5 0 . 83 0 . 84 0 . 85 0 . 86 0 . 87 0 . 88 0 . 89 0 . 9 # Samples ρ NATS-Bench-TSS CIFAR-10; Figure 3: Spearman correlation coefficients of respectively. 2 Ablation experiment 6 Finally, we demonstrate the limitations of this work and the possible directions of our future work. In Section 5, We propose an optimization method to alleviate the channel-sensitive issue of MeCo . This weight-sampling approach can be further improved in future work. Moreover, we would like to point out that the evaluation of a zero-cost proxy is often tied to the availability of benchmarks. Hench though the proxies are 'zero cost', the evaluation of the proxies is strongly dependent on a benchmark, which in the first place is very expensive to create. Finally, although MeCo achieves the highest correlation with the test accuracy on multiple benchmarks, it shows near zero correlations on a few tasks (e.g., TransBench-101-Micro with Autoencoding). We will leave these issues as our future work.

7

8

10

30

[

?

]

grad

_

89

.

±

2

27

.

10

norm

[

?

]

snip

[

?

89

.

±

2

2

27

.

10

[

?

]

jacov

[

91

.

23

±

1

.

01

?

]

NTK

[

92

.

67

±

0

.

46

?

10

]

zen

[

?

58

.

99

±

3

.

44

20

]

ZiCo

[

93

.

56

±

0

.

23

30

?

]

c_{k-2}

0

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_5x5

c_{k-1}

sep_conv_5x5

1

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

c_{k-2}

0

sep_conv_5x5

1

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_5x5

c_{k-1}

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

c_{k-2}

0

avg_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

sep_conv_5x5

c_{k-1}

avg_pool_3x3

dil_conv_3x3

skip_connect

c_{k}

dil_conv_3x3

c_{k-2}

0

sep_conv_3x3

1

max_pool_3x3

2

sep_conv_3x3

3

sep_conv_5x5

c_{k-1}

sep_conv_5x5

max_pool_3x3

sep_conv_3x3

c_{k}

dil_conv_5x5

c_{k-2}

0

sep_conv_3x3

1

dil_conv_5x5

c_{k-1}

avg_pool_3x3

sep_conv_5x5

2

dil_conv_3x3

3

sep_conv_3x3

c_{k}

sep_conv_3x3

sep_conv_5x5

c_{k-2}

0

sep_conv_5x5

1

avg_pool_3x3

2

sep_conv_3x3

3

dil_conv_3x3

c_{k-1}

sep_conv_5x5

sep_conv_5x5

skip_connect

c_{k}

sep_conv_3x3

c_{k-2}

0

avg_pool_3x3

1

skip_connect

c_{k-1}

avg_pool_3x3

sep_conv_5x5

2

dil_conv_3x3

sep_conv_3x3

3

sep_conv_5x5

c_{k}

sep_conv_3x3

c_{k-2}

0

sep_conv_3x3

1

avg_pool_3x3

2

sep_conv_5x5

c_{k-1}

skip_connect

skip_connect

skip_connect

3

max_pool_3x3

c_{k}

sep_conv_3x3

c_{k-2}

0

max_pool_3x3

1

dil_conv_3x3

2

dil_conv_3x3

3

sep_conv_5x5

c_{k-1}

dil_conv_3x3

sep_conv_3x3

c_{k}

sep_conv_3x3

dil_conv_3x3

c_{k-2}

0

sep_conv_3x3

1

max_pool_3x3

c_{k-1}

max_pool_3x3

avg_pool_3x3

2

sep_conv_3x3

3

dil_conv_3x3

c_{k}

sep_conv_5x5

sep_conv_3x3

c_{k-2}

0

dil_conv_5x5

1

sep_conv_3x3

2

sep_conv_3x3

3

dil_conv_5x5

c_{k-1}

dil_conv_3x3

sep_conv_5x5

sep_conv_5x5

dil_conv_5x5

c_{k}

c_{k-2}

0

sep_conv_3x3

3

sep_conv_5x5

c_{k-1}

sep_conv_3x3

1

sep_conv_5x5

dil_conv_3x3

sep_conv_5x5

2

dil_conv_5x5

c_{k}

sep_conv_5x5

c_{k-2}

0

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_5x5

c_{k-1}

sep_conv_3x3

1

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

c_{k-2}

0

sep_conv_3x3

1

sep_conv_5x5

2

sep_conv_5x5

3

skip_connect

c_{k-1}

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

(g)

ZiCo

c_{k-2}

0

dil_conv_5x5

1

sep_conv_5x5

2

dil_conv_5x5

c_{k-1}

sep_conv_5x5

skip_connect

sep_conv_3x3

3

sep_conv_3x3

c_{k}

sep_conv_5x5

Normal Cell

c_{k-2}

0

dil_conv_5x5

1

sep_conv_5x5

2

dil_conv_5x5

c_{k-1}

sep_conv_5x5

sep_conv_5x5

dil_conv_5x5

3

dil_conv_5x5

c_{k}

sep_conv_3x3

Reduction Cell

c_{k-2}

0

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_5x5

c_{k-1}

sep_conv_5x5

1

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

Normal Cell

c_{k-2}

0

sep_conv_5x5

1

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_5x5

c_{k-1}

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

sep_conv_5x5

c_{k}

Reduction Cell

(a)

synflow

c_{k-2}

0

sep_conv_3x3

1

sep_conv_5x5

3

dil_conv_5x5

c_{k-1}

max_pool_3x3

max_pool_3x3

2

dil_conv_3x3

skip_connect

c_{k}

skip_connect

Normal Cell

c_{k-2}

0

sep_conv_5x5

2

sep_conv_5x5

c_{k-1}

sep_conv_3x3

1

sep_conv_3x3

sep_conv_3x3

3

max_pool_3x3

dil_conv_3x3

c_{k}

sep_conv_3x3

Reduction Cell

(b)

fisher

c_{k-2}

0

sep_conv_5x5

2

skip_connect

3

dil_conv_3x3

c_{k-1}

skip_connect

1

sep_conv_3x3

sep_conv_5x5

sep_conv_3x3

c_{k}

max_pool_3x3

c_{k-2}

0

dil_conv_3x3

1

sep_conv_3x3

2

skip_connect

c_{k-1}

sep_conv_5x5

3

dil_conv_5x5

skip_connect

skip_connect

sep_conv_5x5

c_{k}

Reduction Cell

(c)

snip

c_{k-2}

0

max_pool_3x3

1

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

avg_pool_3x3

sep_conv_3x3

2

sep_conv_5x5

c_{k}

sep_conv_3x3

sep_conv_3x3

Normal Cell

c_{k-2}

0

max_pool_3x3

1

dil_conv_3x3

2

skip_connect

c_{k-1}

sep_conv_3x3

dil_conv_3x3

3

dil_conv_3x3

dil_conv_3x3

sep_conv_3x3

c_{k}

Reduction Cell

(d)

grasp

c_{k-2}

0

max_pool_3x3

1

dil_conv_3x3

2

avg_pool_3x3

c_{k-1}

avg_pool_3x3

dil_conv_5x5

3

dil_conv_5x5

c_{k}

avg_pool_3x3

dil_conv_5x5

Normal Cell

c_{k-2}

0

skip_connect

1

avg_pool_3x3

2

sep_conv_5x5

3

dil_conv_3x3

c_{k-1}

sep_conv_5x5

sep_conv_3x3

dil_conv_5x5

skip_connect

c_{k}

Reduction Cell

jacov

(e)

c_{k-2}

0

dil_conv_5x5

1

dil_conv_5x5

2

dil_conv_3x3

3

sep_conv_3x3

c_{k-1}

dil_conv_5x5

sep_conv_5x5

c_{k}

dil_conv_5x5

sep_conv_5x5

Normal Cell

c_{k-2}

0

sep_conv_5x5

1

sep_conv_3x3

3

dil_conv_5x5

c_{k-1}

sep_conv_5x5

sep_conv_5x5

2

sep_conv_3x3

sep_conv_5x5

sep_conv_3x3

c_{k}

Reduction Cell

(f)

NTK

c_{k-2}

0

sep_conv_5x5

1

sep_conv_3x3

2

sep_conv_5x5

3

sep_conv_3x3

c_{k-1}

sep_conv_5x5

sep_conv_3x3

sep_conv_5x5

c_{k}

sep_conv_3x3

Normal Cell

c_{k-2}

0

sep_conv_5x5

1

sep_conv_5x5

2

sep_conv_5x5

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

dil_conv_5x5

sep_conv_3x3

sep_conv_3x3

c_{k}

Reduction Cell

(g)

ZiCo

c_{k-2}

0

dil_conv_3x3

1

sep_conv_5x5

3

dil_conv_5x5

c_{k-1}

sep_conv_3x3

dil_conv_5x5

2

sep_conv_5x5

sep_conv_3x3

c_{k}

sep_conv_3x3

Normal Cell

c_{k-2}

0

avg_pool_3x3

1

sep_conv_5x5

2

sep_conv_3x3

3

dil_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

dil_conv_5x5

c_{k}

sep_conv_3x3

Reduction Cell

(h)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel zero-cost proxy called MeCo. The proxy is data-independent, which requires only one random data with a single forward pass. It analyzed that the minimum eigenvalue of P(X) is correlated to the training convergence rate of the network. The experiments on several NAS benchmarks evaluate the effectiveness of MeCo.\n\nSTRENGTHS:\n1. The paper proposes a novel zero-cost proxy MeCo, which is theoretically guaranteed.\n2. The ranking correlation results demonstrate the superiority of MeCo compared with other proxies.\n3. The writing is good and easy to follow.\n\nWEAKNESSES:\n1. On Table 5, the performance gain of Zero-Cost-PT_MeCo is small compared to other Zero-Cost-PT methods. Besides,  the performance gain of Zero-Cost-PT_MeCo is lower than TE-NAS.\n2. More experimental results should be provided to demonstrate the effectiveness of the proposed method, such as the ranking results on NAS-Bench-301 and the search results on ImageNet in DARTS search space.\n\nQUESTIONS:\n1. In the theoretical proof section, the paper mainly focuses on the network convergence rate and convolutional layer. I wonder how can the theory be connected with network generalization? And can the MeCo metric be applicable to the self-attention layer?\n2. The paper proposes to use only one random data. But the ablation experiments in Figure 4 show that the optimal result is obtained when the data size is not equal to 1. How to explain this?\n3. In Table 5, why is the search cost higher than the other gradient-based zero-cost methods? For example, Zero-Cost-PT_MeCo v.s. Zero-Cost-PT_ZiCo and Zero-Cost-PT_Snip, because the paper requires only one random data with a single forward pass which should be very low search cost. \n4. The results on ImageNet in DARTS-CNN search space should be provided.\n5. To show the effectiveness of MeCo, more ranking consistency experiments should be conducted in different search spaces, such as NAS-Bench-301.\n6. Minor Error. In line 220, \"0.6 to 0.9\" -> \"0.06 to 0.09\".\n\nLIMITATIONS:\nSee the Weaknesses and Questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MeCo, a new zero-shot NAS method that can rank model architectures without training, without any backward pass, and just using 1 data sample (instead of a batch of data like existing methods). The method is evaluated on multiple NAS Benches and the authors have also run NAS experiments on CIFAR-10 and CIFAR-100 datasets. Rest of the NAS runs are done directly on NAS-Benches. The authors have also presented a simple theoretical result relating Pearson’s correlation to gram matrix which is sometimes used to prove effectiveness of zero-cost proxies.\n\nSTRENGTHS:\nThe paper has following strengths: \n\n1. The main strength of this idea is that it uses only one data sample and no backward pass. So, the proposed zero-shot method is quick.\n\n2. On NATS-Bench-SSS and NATS-Bench-TSS, the proposed method achieves higher correlation than existing methods.\n\n3. The paper is very well-written.\n\nWEAKNESSES:\nThe paper has following weaknesses: \n\n1. No results on zero-shot NAS runs on generic search spaces on ImageNet. Due to low search cost, many new proxies like ZiCo, ZenNAS, etc., have shown results on ImageNet and often beat multi-shot and one-shot methods by a large margin (in search time as well as accuracy). The authors should conduct this experiment with MeCo and present detailed results and comparisons to existing methods. In the current state, the total accuracy improvements compared to existing baselines like Zen, ZiCo, etc., are not too significant (e.g., see tables 4, 5, 6 in supplementary and table 4 in main paper). Significant improvements on ImageNet can make the paper's claims much more convincing.\n\n2. Between NATS-Bench-TSS to NATS-Bench-SSS, the correlation changes from highly positive correlation to highly negative correlation (see Tables 1 and 2 in main paper). Authors show some results stating that this is related to large channel widths of networks which impacts the minimum eigenvalue. However, this may also mean there may be some channel widths where the MeCo correlations may become close to zero (since it changed from highly positive to highly negative when changing channel widths or other architecture properties). In case that happens, it may be problematic since zero correlations cannot really say anything about model quality. Can the authors conduct a detailed (theoretical/empirical) analysis towards this important issue? Under what conditions would MeCo work optimally?\n\n3. It was not clear what kind of data size the authors changed in Section 5 ablation study. Did they change input feature dimensions or number of samples? If they did not ablate over number of samples used to calculate MeCo, they should conduct such a study (e.g., how does correlation vary if you use 16 samples instead of 1 sample, etc.). \n\n4. Also, if they use only one data sample to calculate MeCo, what is the variance of MeCo across multiple samples from same or different classes?\n\n5. Few other NAS-Benches are available and has been utilized in literature (e.g., TransNASBench and others used in ZiCo paper). More results can be shown on those benchmarks.\n\n6. A couple other papers authors should also discuss in related work: (1) “Layerwise Dynamical Isometry” [ICLR 2020, https://arxiv.org/abs/1906.06307] that talks about signal propagation properties via layer Jacobians to improve training convergence of highly sparse networks. (2) NN-Mass [CVPR 2021, https://arxiv.org/abs/1910.00780] which explicitly relates topological properties of deep networks to gradient flow. NN-Mass does not even require any forward pass and ranks networks purely using their architecture's structural properties (although it works in very limited settings). In fact, these two works form the earliest precursors to the zero-shot NAS domain (and came before all other training free metrics) and must be discussed.\n\n\n==== UPDATE AFTER REBUTTAL ====\nI have read the author response and other reviews. The new data and analyses provided by the authors have improved the paper compared to the initial submission. However, I still have concerns regarding some of the new data: \n\n(1) ImageNet results: Looking at table 9 in zico paper, we see that they achieve 77.1% imagenet accuracy with 600M FLOPS. However, in a comment to another reviewer, authors reveal that their MeCo found model requires nearly 884M FLOPS. This is a major increase in FLOPS with only 0.2% improvement in accuracy. Hence, this result is not convincing. Note that, there may be major differences between ZiCo and authors' training recipe and search space. If this is the case, I would encourage the authors to provide more details. \n\n(2) The negative correlation thing and the channel sensitivity indeed seems to be a major weakness of this work. I appreciate authors' analysis that if c < wxh, MeCo would work better. However, this is violated for many image classification networks (e.g., on imagenet 224x224 input resolution with final layers getting 7x7 feature map size and much higher channel counts). The new results in Table 3 of the attached pdf also confirm that on some other use cases the MeCo does start losing correlations (although, on a lot of datasets, it performs very well). Hence, this can limit the applicability of this work. Can authors explore techniques to fix this channel sensitivity issue? Given this new information about c<wxh, it may actually be better if authors can mainly focus on vision tasks with much higher input-output resolutions, e.g., image super resolution and semantic segmentation. Specifically, for such vision applications, resolutions generally go up to 1080p or even 4K, so these applications may show the most benefits of authors' approach (and thus show that under applications with c<wxh, MeCo is always well behaved and does not lead to zero values, etc.). Of note, super resolution with smaller models (e.g., SESR (MLSys 2022), https://arxiv.org/abs/2103.09404) does not take long to train.\n\nAnyways, given the improvement in clarity, new results, and new analysis, I am raising the rating to 4. I can raise the rating to 5 (or even 6 depending on how strong the results are) provided the authors (a) clearly identify the settings under which MeCo must be used (i.e., this dependence on c<wxh) right from the beginning of the paper (introduction, motivation, etc.), and (b) clearly demonstrate at least one vision application with high resolution input-output (e.g., super resolution...there have been NAS papers on this topic before: https://arxiv.org/pdf/2007.04356.pdf (ECCV 2020)). But without seeing major improvements on the above issues, I cannot increase the rating further at this time. I do think the authors are on to some interesting research. I would encourage them to continue this work and improve it further.\n\n==== FURTHER UPDATE ====\nAuthors have fixed the main weakness with the introduction of their sensitivity correction method MeCo-Opt. All negative correlations are now positive (I do recommend to let other reviewers with lower rating know about this if they have similar concerns...but I leave that decision to the authors). A few Nas benches show near zero correlations but that is to be expected since no zero shot proxy is perfect. Moreover, the new zero shot proxy correction MeCo-Opt either maintains accuracy results or significantly improves them (e.g., for Imagenet-16-120 Nas bench). I hope the authors will use the remaining time until final notification to make sure their presentation is perfect in light of these new contributions: Specifically, motivate the need for metrics that work for higher resolution input-output vision tasks like segmentation, properly introduce the sensitivity correction in the main approach, update all the results with both MeCo and MeCo-Opt, conduct a few more real world experiments showing the significant improvements under MAC constraints that go beyond Nas benches (e.g., Imagenet and other Segmentation or Detection tasks) to complete the high-resolution input output picture and also provide conclusively better results over existing proxies under MAC constraints, and cover all the related work pointed out by all reviewers. The paper (particularly after rebuttal) has made strong contributions so please ensure that they are reflected in the presentation in the next version. I believe that with good presentation, this work should be of interest to the research community.\n\nGiven that my main concern is now resolved, I raise the rating to 7. Good luck!\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nYes, in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new proxy ie MeCo for zero-cost neural architecture search. The proxy design aims at overcoming two important disadvantages of zero-cost proxies proposed earlier. The first one is that these proxies invoke at least one back prop and hence are still expensive. The second disadvantage is that the proxies are very much dependent on the data and labels considered.  The designed proxy is evaluated on NATS-Bench-TSS  and NATS Bench-SSS benchmarks and achieves a very high correlation with the ground truth in comparison to other SOTA proxies. Further integration of MeCO can be easily integrated with existing generation method for complete NAS and shows performance gains in comparison to other zero cost proxies. The code of the paper is open-sourced making the approach reproducible.\n\nSTRENGTHS:\nThe paper is very well motivated, well-written and clear. The theoretical motivation behind the MeCO is very clearly established. The idea behind the designed proxy is fairly novel to NAS and its effectiveness is established across different benchmarks and also for complete NAS. The authors adequately discuss their hyperparameter settings and release their code for reproducibility. The evaluation is fairly thorough on the NATS-Bench and the method is well-ablated.\n\nWEAKNESSES:\nMain weakness of the paper in my opinion is the search spaces the proxy is studied on.  [NAS-Bench Suite Zero](https://arxiv.org/pdf/2210.03230.pdf) provides easy access and evaluation on about 13 proxies and 28 tasks. Since the performance of a given proxy can vary widely depending upon the search space, task and datasets I am not convinced of the effectiveness of the proxy based on only the results on NATS-Bench and DARTS spaces. Furthermore how would one apply proxies to transformer spaces (eg [AutoFormer](https://openaccess.thecvf.com/content/ICCV2021/papers/Chen_AutoFormer_Searching_Transformers_for_Visual_Recognition_ICCV_2021_paper.pdf) , [HAT](https://arxiv.org/abs/2005.14187)) and MobileNets(eg: [OFA](https://arxiv.org/pdf/1908.09791.pdf),) which provide queryable validation accuracy based on a surrogate. Can this proxy be applied in these spaces? Since modern one-shot NAS methods are applied on transformer and mobilenet spaces too, it is important to design a proxy that does indeed generalize well. I encourage the authors to evaluate their proxy on these search spaces too.\n\nQUESTIONS:\nIn addition to comments in the weakness section I have the following questions:\n1. Could you evaluate the proxy on all tasks from NAS-Bench suite zero and transformer spaces?\n2. Why is the correlation coefficient negative on NATS-Bench-SSS? Could you please elaborate the explanation from line 228-246\n3. Could you compare the cost in computing MeCo v/s other zero-cost proxies?\n\nLIMITATIONS:\nI encourage to discuss the limitations of the designed proxy adequately, for example address the following points:\n1. Are there any important assumptions made in designing the proxy, which may not hold in practice?\n2. Evaluation of how good a zero cost proxy is, is often tied to availability of benchmarks it can be evaluated on. Hence though the proxies 3. are indeed “zero-cost”, the evaluation of the proxy quality is strongly tied to a benchmark which in the first place is very expensive to create. It would be great if the authors could comment on this in the limitations too.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce a new zero-cost proxy for NAS, called MeCo, which is fully independent of data and labels and requires only one data sample for a single forward pass. The paper provides a theoretical analysis of the relationship between the Pearson correlation matrix of the feature maps and the training convergence rate of the network. Meco shows high ranking correlation in NATS-Bench.\n\nSTRENGTHS:\nThe authors theoretically reveal that the minimum eigenvalue of the correlation on the feature map is correlated with the training convergence rate. Based on this analysis, they propose Meco as a novel training-free proxy. Experiments conducted on NATS-Bench demonstrate the superiority of Meco for three small datasets.\n\nWEAKNESSES:\n1. As acknowledged by the authors, Meco is sensitive to variations in the number of channels, implying that Meco exhibits a similar preference as #Param when selecting an architecture from the search space. However, the paper lacks the ranking correlation of Param in Tab1 and Tab2, making it difficult to assess the correlation between Meco and Param. Besides the marginal improvements in ranking correlation, it would be beneficial to understand the advantages of Meco over Param.\n\n2. It is recommended to address the variation of correlation in Tab1 and Tab2. Many zero-cost proxies can experience fluctuations in ranking correlation when different batches with the same batch size are used. To showcase the robustness of Meco, providing the variation would be valuable.\n\n3. I have a question regarding the NAS-Benchmarks. The abstract states that NAS-Bench-101 is used in this paper; however, there are no experiments conducted on NAS-Bench-101 in the main body, only in the supplementary material. It is inappropriate to claim that experiments were performed on NB101 without presenting the results in the main body. Additionally, NAS-Bench-201 is a widely-used benchmark in the NAS community. I noticed that there are some files related to NB201 in the provided code. Why was NAS-Bench-201 not utilized?\n\n4. I noticed an inconsistency in the ranking correlation of NTK between Table3 (0.78) and Table1 (0.76). Please provide an explanation for this discrepancy.\n\n5. It should be noted that in recent years, there have been advancements in zero-shot proxies, including NASWOT, KNAS, NASI, and GradSign. However, this paper only focuses on older methods and does not consider these newer counterparts.\n\n6. I am curious about the computational efficiency of Meco when using `torch.corrcoef(fea)`. In practical neural networks, especially in later layers, the channel number can be quite large. Therefore, I am questioning whether Meco can be extended to larger networks with a large number of channels while maintaining computational efficiency.\n\nQUESTIONS:\nWhat are the advantages of Meco over #Param, besides the marginal improvements in ranking correlation? Is there a high correlation between Meco and #Param?\n\nHow does the correlation variation of Meco address the issue of fluctuation in ranking correlation seen in other zero-cost proxies? Can the robustness of Meco be demonstrated through providing the correlation variation?\n\nWhy does the paper claim to use NAS-Bench-101 but not provide the experimental results in the main body? Why was NAS-Bench-201, a widely-used benchmark in the NAS community, not utilized?\n\nCan Meco maintain computational efficiency when applied to larger networks with a large number of channels, considering the practical implications for neural networks?\n\nLIMITATIONS:\nI have concerns regarding the experimental aspects of the paper. During the rebuttal period, I hope the authors can address these concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new zero-cost proxy (called MeCo) for neural architecture search motivated by theoretical insights about the relationship of the statistics of Pearson Matrix and a network's convergence rate. The proposed proxy is evaluated by investigating ranking correlation on NATSBench-TSS, NATSBench-SSS, and NASBench101, as well as CIFAR1 dataset.\n\nSTRENGTHS:\n- A novel zero-cost proxy for NAS grounded in theory is proposed\n- Good performance of the proposed method (although see below)\n\nWEAKNESSES:\nMy concern primarily comes from the evaluation sections.\n- The author didn't show the experiments on large-scale datasets, such as ImageNet\n- The author didn't show the results on MobileNet-v2 like search space.\n- The author didn't show the performance on other types of NAS benchmarks that are beyond the image classification, e.g., TransNASBench101. \n\nI am also concerned about the theory part. \n- The author only discusses the convergence speed of the network. The generalization capacity of the network is not involved in the discussion.\n- The author shows contrasting results on TSS and SSS. \n\n\n\n\n\nMinor:\n- Typos: In line 220, the author claims that MeCo is 0.6-0.9 higher than ZiCo.\n\nQUESTIONS:\nMy questions are aligned with the weakness sections\n- How could Meco be connected with the generalization capacity of the network?\n- Can the author show more results on diverse tasks beyond the image classification?\n- How does the number of input samples impact the performance of the MeCo?\n- Does MeCo generalize to other search spaces and large-scale datasets?\n- Can the author show the performance of the searched optimal network via MeCo on these benchmarks?\n- Why on SSS the correlation is negative while on TSS the correlation is positive? Based on the theoretical analysis (line 183, 184), it should be always positive.\n\nLIMITATIONS:\nCurrently, the limitations are primarily the insufficient evaluation of different tasks and different search spaces. Also, some results are not clear in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}