SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| f (\mathbf {W} (t), \mathbf {a}, \mathbf {X}) - \mathbf {y} \right\| _ {2} ^ {2} \leq \exp \left(- c p _ {0} t\right) \left\| f (\mathbf {W} (0), \mathbf {a}, \mathbf {X}) - \mathbf {y} \right\| _ {2} ^ {2}
$$
where  $c$  is a constant depending on  $m$  and  $d$ .
Proof. To simplify our proof, we assume  $\mu_{\mathbf{x}} = 0$ ,  $\sigma_{\mathbf{x}} = 1$ , and  $\| \mathbf{x}\| _2\leq C$  for all  $\mathbf{x}\in \mathbf{X}$ . Recall that
$$
[ \mathbf {H} (0) ] _ {i j} = \mathbf {x} _ {i} ^ {T} \mathbf {x} _ {j} \frac {1}{m} \sum_ {r = 1} ^ {m} \mathbb {I} \left\{\mathbf {w} _ {r} ^ {T} (0) \mathbf {x} _ {i} \geq 0, \mathbf {w} _ {r} ^ {T} (0) \mathbf {x} _ {j} \geq 0 \right\} = \mathbf {x} _ {i} ^ {T} \mathbf {x} _ {j} [ \mathbf {W} (0) ] _ {i j}
$$
Due to  $\mathbb{I}\{\mathbf{w}_r^T (0)\mathbf{x}_i\geq 0,\mathbf{w}_r^T (0)\mathbf{x}_j\geq 0\}$  is an independent random variable between 0 and 1, then by Hoeffding's inequality [61], the following inequality holds with probability  $1 - \delta$  ..
$$
[ \mathbf {W} (0) ] _ {i j} \geq [ \bar {\mathbf {W}} ] _ {i j} - \frac {2 \sqrt {\log {(1 / \delta)}}}{\sqrt {m}}
$$
Let  $\mu_0 = \min_{(i,j)\in [n]\times [n]}[\bar{\mathbf{W}} ]_{ij}$ , and choose  $m > \frac{16\log(1 / \delta)}{\mu_0^2}$ , then we have
$$
[ \mathbf {W} (0) ] _ {i j} \geq \mu_ {0} - \frac {2 \sqrt {\log {(1 / \delta)}}}{\sqrt {m}} \geq \frac {2 \sqrt {\log {(1 / \delta)}}}{\sqrt {m}} =: c (m)
$$
Define the matrix  $\mathbf{M}$  with entry  $[\mathbf{M}]_{ij}$  as
$$
[ \mathbf {M} ] _ {i j} := [ \mathbf {H} (0) ] _ {i j} - c (m, d) [ P (\mathbf {X}) ] _ {i j}
$$
where  $c(m, d) = c(m)(d - 1)$ . We claim that if  $m$  is large enough, then  $\mathbf{M}$  is positive definite. To clarify this statement, we consider the gap between  $\mathbf{H}(0)$  and  $\mathbf{M}$ :
$$
\left\| \mathbf {H} (0) - \mathbf {M} \right\| _ {2} = \left\| c (m, d) P (\mathbf {X}) \right\| _ {2} \leq c (m, d) \left\| P (X) \right\| _ {F} \leq c (m, d) n ^ {2} C ^ {2}
$$
If we choose  $m > \frac{64\log(1 / \delta)(d - 1)^2n^4C^4}{\lambda_0^2}$ , we have
$$
\| \mathbf {H} (0) - \mathbf {M} \| _ {2} \leq \frac {\lambda_ {0}}{4}
$$
Then the following formula holds by matrix perturbation theory (Corollary 6.3.8 in [39])
$$
0 <   \frac {\lambda_ {0}}{2} \leq \lambda_ {\min } (\mathbf {H} (0)) - \frac {\lambda_ {0}}{4} \leq \lambda_ {\min } (\mathbf {M})
$$
which indicates that  $\mathbf{M}$  is positive definite. We next exert Lemma 2 on  $\mathbf{M}$  and get
$$
0 <   \lambda_ {\min } (\mathbf {H} (0) - c (m, d) P (\mathbf {X})) <   \lambda_ {\min } (\mathbf {H} (0)) - \lambda_ {\min } (c (m, d) \mathbf {P} (\mathbf {X}))
$$
$$
0 <   c (m, d) p _ {0} \leq \lambda_ {\min } (\mathbf {H} (0)) \leq \frac {5}{4} \lambda_ {0} \tag {16}
$$
Therefore, combined with Theorem 1, we have
$$
\begin{array}{l} \left\| f (\mathbf {W} (t), \mathbf {a}, \mathbf {X}) - \mathbf {y} \right\| _ {2} ^ {2} \\ \leq \exp (- \lambda_ {0} t) \| f (\mathbf {W} (0), \mathbf {a}, \mathbf {X}) - \mathbf {y} \| _ {2} ^ {2} \\ \leq \exp (- c (m, d) p _ {0} t) \| f (\mathbf {W} (0), \mathbf {a}, \mathbf {X}) - \mathbf {y} \| _ {2} ^ {2} \\ \end{array}
$$
where  $c(m, d)$  is a constant depending on  $m$  and  $d$ . This completes the proof of Theorem 2.
Proof. By Corollary 3.10 and Remark 3.11 in [40], we have
$$
\mathbb {E} [ L (W) ] \leq O \left(c \cdot \sqrt {\frac {\mathbf {y} ^ {T} (\mathbf {H} ^ {\infty}) ^ {- 1} \mathbf {y}}{N}}\right) + O \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right)
$$
According to the courant minimax principle[62], D.2 in [26], and inequality 16, we get
$$
\mathbf {y} ^ {T} (\mathbf {H} ^ {\infty}) ^ {- 1} \mathbf {y} \leq \frac {\mathbf {y} ^ {T} \mathbf {y}}{\lambda_ {\min} (\mathbf {H} ^ {\infty})} \leq c \cdot \frac {\mathbf {y} ^ {T} \mathbf {y}}{p _ {0}}.
$$
$$
\begin{array}{l} \mathbb {E} [ L (W) ] \leq O \left(c \cdot \sqrt {\frac {\mathbf {y} ^ {T} (\mathbf {H} ^ {\infty}) ^ {- 1} \mathbf {y}}{N}}\right) + O \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right) \\ \leq O \left(c \cdot \sqrt {\frac {\mathbf {y} ^ {T} \mathbf {y}}{N p _ {0}}}\right) + O \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right) \\ \end{array}
$$
B Zero-cost proxies
In this section, we provide the details of the baseline zero-cost proxies. Suppose  $L$  is the loss function and  $\theta$  is the parameters of an initialized network. We denote  $\circ$  as the Hadamard product. The concrete formulations of the existing zero-cost proxies are as follows.
- snip. Lee et al. [20] use the changes in loss caused by the parameter perturbations to measure the importance of the parameters in an initialized network, such that
$$
\mathcal {S} _ {\mathrm {s n i p}} (\theta) = \left| \frac {\partial L}{\partial \theta} \circ \theta \right|
$$
- grasp. Wang et al. [21] replace the loss change in snip with the change of the gradient norm to establish the proxy, such that
$$
\mathcal {S} _ {\mathrm {g r a s p}} (\theta) = - \left(H \frac {\partial L}{\partial \theta}\right) \circ \theta
$$
where  $H$  is the Hessian.
- synflow. To avoid layer collapse, Tanaka et al. [22] utilize the product of all parameters in the network during the parameter perturbation to represent the loss, such that
$$
\mathcal {S} _ {\text {s y n f l o w}} (\theta) = \frac {\partial L}{\partial \theta} \circ \theta
$$
- grad_norm. Abdelfattah et al. [8] adopt the  $l_{2}$  norm of the gradients in an initialized work as a proxy, such that
$$
\mathcal {S} _ {\text {g r a d \_ n o r m}} = \left\| \frac {\partial L}{\partial \theta} \right\| _ {2}
$$
- jacov/NWOT. Mellor et al. [7] use the correlation of activations within a network as a proxy to evaluate the performance of the network, such that
$$
\mathcal {S} _ {\mathrm {j a c o v}} = \log | \mathbf {K} _ {H} |, \quad \mathbf {K} _ {H} = \left( \begin{array}{c c c} N _ {A} - d _ {H} (\mathbf {c} _ {1}, \mathbf {c} _ {1}) & \dots & N _ {A} - d _ {H} (\mathbf {c} _ {1}, \mathbf {c} _ {N}) \\ \vdots & \ddots & \vdots \\ N _ {A} - d _ {H} (\mathbf {c} _ {N}, \mathbf {c} _ {1}) & \dots & N _ {A} - d _ {H} (\mathbf {c} _ {N}, \mathbf {c} _ {N}) \end{array} \right)
$$
where  $N_{A}$  is the number of rectified linear units,  $d_{H}(\mathbf{c}_{i},\mathbf{c}_{j})$  represents the Hamming distance between two binary codes  $\mathbf{c}_i$  and  $\mathbf{c}_j$ .
- NTK. Chen et al. [24] propose to use the condition number of NTK to measure the trainability of the networks, such that
$$
\kappa_ {\mathcal {N}} = \frac {\lambda_ {\mathrm {m a x}} (\hat {\Theta} _ {t r a i n})}{\lambda_ {\mathrm {m i n}} (\hat {\Theta} _ {t r a i n})}
$$
where  $\hat{\Theta}_{train}$  stands for NTK of the networks. In our paper, we calculate the Spearman correlation coefficient between  $1 / \kappa_{\mathcal{N}}$  and the test accuracy of the networks.
- zen. Lin et al. [27] propose Zen-Score, in which they design an efficient zero-cost proxy with Gaussian random inputs, such that
$$
\mathcal {S} _ {\mathrm {z e n}} = \log \left(\mathbb {E} _ {\mathbf {x}, \epsilon} \| f (\mathbf {x}; \theta) - f (\mathbf {x} + \alpha \epsilon ; \theta) \| _ {F}\right) + \sum_ {i} \log \left(\sqrt {\sum_ {j} \sigma_ {i , j} ^ {2} / m}\right)
$$
where  $\sigma_{i,j}$  is the mini-batch standard deviation statistic of the  $j$ -th channel in BN.
- NASI. Shu et al. [52] propose NASI to evaluate the networks by approximating the trace of the NTK, such that
$$
\mathrm {N A S I} = m \gamma^ {- 1} \left\| b ^ {- 1} \sum_ {x \in \mathcal {X} _ {j}} \nabla_ {\theta_ {0} (\mathcal {A})} \mathcal {L} _ {x} \right\| _ {2} ^ {2}
$$
where  $\mathcal{X}_j$  is a mini-batch of data with size  $|\mathcal{X}_j| = b$ .
- KNAS. Xu et al. [51] propose to use gradient kernel to evaluate the networks, such that
$$
g = \frac {1}{n ^ {2}} \sum_ {i, j} \left(\frac {\partial y _ {j} ^ {L} (t)}{\partial \mathbf {w} (t)}\right) \left(\frac {\partial y _ {i} ^ {L} (t)}{\partial \mathbf {w} (t)}\right) ^ {T}
$$
where  $y^{L}$  is the output of  $L$ -th layer.
- GradSign. Zhang and Jia [53] propose GradSign, in which they analyze the sample-wise optimization landscape of the networks, such that
$$
\operatorname {G r a d S i g n} = \sum_ {k} | \sum_ {i} s i g n ([ \nabla_ {\theta} l (f _ {\theta} (x _ {i}), y _ {i}) | _ {\theta_ {0}} ] _ {k}) |
$$
- ZiCo. Li et al. [9] explore the effect of gradient properties on network performance. They use absolute mean and standard deviation values of gradients to evaluate network performance:
$$
Z i C o = \sum_ {l = 1} ^ {D} \log \left(\sum_ {\omega \in \theta_ {l}} \frac {\mathbb {E} [ | \partial L (\mathbf {X} _ {i} , \mathbf {y} _ {i} ; \theta) / \partial \omega | ]}{\sqrt {V a r (| \partial L (\mathbf {X} _ {i} , \mathbf {y} _ {i} ; \theta) / \partial \omega |)}}\right), \quad i \in [ N ]
$$
where  $N$  and  $D$  represent the number of batches and network layers, respectively.  $\theta_{l}$  represents the parameters of the  $l$ -th layers.
C Algorithm of MeCo-based NAS
We slightly abuse the notation and denote MeCo as our proxy function for the network. We adopt the Zero-Cost-PT [55] to integrate MeCo to a zero-shot NAS. The algorithm of our MeCo-based NAS is summarized in Algorithm 1. We denote  $A_0$  as an untrained supernet,  $e_t$  represents the  $t$ -th edge, which stands for mixed operation in the search cells, and  $e_{t,k}$  is the  $k$ -th operation of  $t$ -th edge. We denote  $\mathcal{E}, \mathcal{N}$ , and  $\mathcal{O}$  as the set of edges, nodes, and candidate operations in the search cells, respectively. For any node  $n \in \mathcal{N}$ , we use  $\mathcal{E}(n)$  as the set of its input edges, and  $e_n^k$  is the  $k$ -th element of  $\mathcal{E}(n)$ . Note that we can use other zero-cost proxies as described in section B to replace MeCo in Algorithm 1.
Require:  $A_0$ : An untrained supernet;  $\mathcal{E}$ : The set of edges in search cells;  $\mathcal{N}$ : The set of nodes in search cells;  $\mathcal{O}$ : The set of candidate operations;  $N$ : the number of candidate networks.
Algorithm 1 MeCo-based NAS
Ensure: The best network  $A_{best}$ .
1: // Stage 1: Architecture Proposal  
2:  $C = \emptyset$ ;  
3: for  $i = 1$ ;  $i \leq N$ ;  $i++$  do  
4: for  $j = 1$ ;  $j \leq |\mathcal{E}|$ ;  $j++$  do  
5: Randomly choose an un-discretized edge  $e_t$   
6: Choose the best edge from the supernet, s.t.  
 $e_{t,\text{best}} = \arg \min_{1 \leq k \leq |\mathcal{O}|} \operatorname{MeCo}(A_0 / e_{t,k})$   
7: Use operation  $e_{t,\text{best}}$  to substitute  $e_t$   
8: end for  
9:  $A_{|\mathcal{E}|}$  consists of  $\{e_{t,\text{best}} | 1 \leq t \leq |\mathcal{E}|\}$   
10: for  $j = 1$ ;  $j < |\mathcal{N}|$ ,  $j++$  do  
11: Randomly select an unselected node  $n \in \mathcal{N}$   
12: for  $k = 1$ ;  $k < |\mathcal{E}(n)|$ ;  $k++$  do  
13: Calculate MeCo of the architecture  $A_{|\mathcal{E}|} / e_n^k$   
14: end for  
15: Retain edges  $e_n^1$ ,  $e_n^2$  with the 1st and 2nd best MeCo value, and remove the other edges  
16: end for  
17: Get the candidate networks  $A_i$  that consist of  $\{e_{t,\text{best}} | 1 \leq t \leq \mathcal{E}\}$ , and append it to the set  $C$   
18: end for  
19: // Stage 2: Architecture Validation  
20: Get the best network:  
 $A_{best} = \arg \max_{1 \leq i \leq N} \operatorname{MeCo}(A_i)$ , s.t.  $A_i \in C$
D Experimental configurations
We use the same settings for the experiments as in [55]. We summarized the configurations of the searching phase and training phase on CIFAR-10 and CIFAR-100 in Table 6 and Table 7, respectively. For ZiCo, we use two mini-batch of data, which has a size of 64. For the other baseline proxies, we use one mini-batch of data. On the other hand, our MeCo only uses one random data  $\mathbf{x} \in \mathbb{R}^{1 \times 3 \times 32 \times 32}$ .
E More experimental results of  $\rho$
E.1 MeCo on NASBench-201 and NASBench-301 with three extra datasets
We evaluate MeCo and  $\mathrm{MeCo}_{\mathrm{opt}}$  on NASBench-201 and NASBench-301 with Spherical-CIFAR-100, NinaPro, and SVHN, respectively. We only use one random data  $x \in \mathbb{R}^{3 \times 32 \times 32}$  for MeCo and  $\mathrm{MeCo}_{\mathrm{opt}}$ . The results are summarized in Table 8. Our MeCo achieves competitive results on most benchmarks and achieves the best results on NASBench-201-SVHN, where the Spearman correlation coefficient is  $\rho = 0.88$ .
Table 6: The settings of Zero-Cost-PT with all proxies in DARTS-CNN for CIFAR-10
Settings	Searching phase	Training phase
Baselines	MeCo (Ours)	Baselines	MeCo (Ours)
batch size	64	1	96	96
cutout	True	False	True	True
cutout length	16	-	16	16
learning rate	0.025	0.025	0.025	0.025
learning rate min	0.001	0.001	-	-
momentum	0.9	0.9	0.9	0.9
weight decay	3e-4	3e-4	3e-4	3e-4
grad clip	5	5	5	5
init channels	16	16	36	36
layers	8	8	20	20
drop path prob	-	-	0.2	0.2
Table 7: The settings of Zero-Cost-PT with all proxies in DARTS-CNN for CIFAR-100
Settings	Searching phase	Training phase
Baselines	MeCo (Ours)	Baselines	MeCo (Ours)
batch size	64	1	96	96
cutout	True	False	True	True
cutout length	16	-	16	16
learning rate	0.025	0.025	0.025	0.025
learning rate min	0.001	0.001	-	-
momentum	0.9	0.9	0.9	0.9
weight decay	3e-4	3e-4	3e-4	3e-4
grad clip	5	5	5	5
init channels	16	16	16	16
layers	8	8	20	20
drop path prob	-	-	0.2	0.2
Table 8: Comparisons of  $\rho$  with baselines using NASBench-301 and NASBench-201 on three extra datasets
Method	NASBench-301	NASBench-201
Sph-Cifar100	NinaPro	SVHN	Sph-Cifar100	NinaPro	SVHN
grasp	0.13	0.04	0.18	-0.01	-0.01	0.62
fisher	0.00	-0.11	0.05	0.07	-0.38	0.71
grad_norm	-0.00	-0.20	0.42	-0.08	-0.23	0.77
snip	-0.01	-0.10	0.38	-0.09	-0.28	0.76
synflow	0.05	-0.07	0.50	0.13	0.02	0.71
l2_norm	0.12	-0.07	0.70	-0.00	0.02	0.67
#params	0.07	-0.07	0.70	-0.14	-0.11	0.72
zen	0.07	-0.09	0.68	0.23	0.15	0.18
jacov	0.08	0.13	-0.36	-0.41	0.29	0.67
nwot	0.05	0.02	0.64	-0.02	0.06	0.76
MeCo (Ours)	-0.05	-0.11	0.68	-0.23	0.02	0.88
MeCoopt (Ours)	0.03	0.12	0.68	-	-	-
(a) Performance on NATS-Bench-TSS
(b) Performance on NATS-Bench-SSS
Figure 4: Relationship between our zero-cost proxies and test accuracy for NATS-Bench-TSS and NATS-Bench-SSS on three datasets.
E.2 MeCo on NAS-Bench-101
We present the Spearman correlation coefficient between all zero-cost proxies and the test accuracy of the networks in NAS-Bench-101. Our MeCo uses one data of CIFAR-10, i.e.,  $\mathbf{x} \in \mathbb{R}^{1 \times 3 \times 32 \times 32}$  while the baselines adopt a batch of samples as input. The results are summarized in Table 9.
Table 9: Spearman correlation coefficients  $\rho$  between zero-cost proxies and test accuracy on NAS-Bench-101
Dataset	MeCo	Baselines
grasp [21]	fisher[23]	grad_norm[8]	snip[20]	synflow [22]	jacov[7]	zen[27]	ZiCo [9]
CIFAR-10	0.44	-0.33	-0.27	-0.32	-0.25	0.36	-0.35	0.63	0.63
MeCo achieves competitive results on NAS-Bench-101 with CIFAR-10, i.e., the Spearman correlation coefficient obtains 0.44, which is higher than the majority of the baselines. However, zen and ZiCo perform better than MeCo.
E.3 MeCo on TransBench-101
We further evaluate our MeCo on diverse tasks. Specifically, we compare MeCo and the other proxies on Transbench-101-Micro and Transbench-101-Macro [47], respectively.
TransBench-101-Micro. We evaluate MeCo and  $\mathrm{MeCo}_{\mathrm{opt}}$  on TransBench-101-Micro with ten tasks. We calculate the Spearman correlation between our proxies and the accuracy of the networks. The results are summarized in Table 10. MeCo achieves the best performance on the three tasks, i.e., Class Objection, Spherical-Cifar100, and NinaPro, which are 0.58, 0.85, and 0.47, respectively. It can be seen that  $\mathrm{MeCo}_{\mathrm{opt}}$  effectively improves the performance of MeCo. For example,  $\mathrm{MeCo}_{\mathrm{opt}}$  improves MeCo from 0.62 to 0.77 on segmentation tasks.
TransBench-101-Macro. We compare MeCo,  $\mathrm{MeCo}_{\mathrm{opt}}$  and the baselines on TransBench-101-Macro with seven tasks. The results are summarized in Table 11. Our proxies achieve the best performance on Autoencoding, Jigsaw, and Surface Normal, which are 0.74, 0.48, and 0.80, respectively. Our proxies also achieve competitive results on the remaining tasks. Experimental results show that
our proxies can be used for diverse tasks. It can be demonstrated that MeCo and  $\mathrm{MeCo}_{\mathrm{opt}}$  become ineffective on a few tasks, such as Room Layout. We would like to note that the existing proxies do not achieve a high correlation on all tasks consistently.
Table 10: Comparisons of  $\rho$  with baselines using Transbench-101-Micro on Ten Tasks
Approach	Autoencoding	Class
Objection	Scene
Classification	Jigsaw	Surface
Normal	Segmentation	Room
Layout	Spherical
-Cifar100	NinaPro	SVHN
grasp	-0.12	-0.22	-0.43	-0.12	0.01	0.00	-0.29	-0.03	-0.20	-0.24
fisher	-0.58	0.44	-0.13	0.30	0.16	0.12	0.30	0.72	0.42	0.81
grad_norm	-0.32	0.39	-0.33	0.36	0.36	0.60	0.25	0.72	0.40	0.78
snip	-0.27	0.45	-0.14	0.41	0.49	0.68	0.32	0.76	0.42	0.83
synflow	0.00	0.48	0.27	0.47	0.00	0.00	0.30	0.79	0.45	0.92
l2_norm	0.04	0.32	0.28	0.35	0.50	0.48	0.18	0.53	0.36	0.52
#params	-0.01	0.45	0.32	0.44	0.62	0.68	0.30	0.79	0.36	0.76
zen	0.14	0.54	0.27	0.51	0.71	0.67	0.38	0.67	0.42	0.74
jacov	0.18	0.51	0.19	0.56	0.75	0.80	0.40	0.71	0.40	0.77
nwot	0.03	0.39	0.89	0.42	0.57	0.53	0.25	0.64	0.38	0.63
zico	0.35	-	0.71	0.52	0.68	-	-	-	-	-
MeCo (Ours)	0.03	0.58	0.62	0.45	0.65	0.62	-0.25	0.85	0.47	0.88
MeCoopt (Ours)	0.03	0.59	0.64	0.47	0.67	0.77	0.26	0.85	0.47	0.88
Table 11: Comparisons of  $\rho$  with baselines using Transbench-101-Macro on Seven Tasks
Approach	Autoencoding	Class
Objection	Scene
Classification	Jigsaw	Surface
Normal	Segmentation	Room
Layout
grasp	-0.02	-0.64	-0.43	-0.26	-0.05	-0.02	-0.26
fisher	-0.19	-0.30	-0.13	-0.26	0.15	0.03	-0.26
grad_norm	0.31	-0.56	-0.33	-0.27	0.35	0.21	-0.27
snip	0.20	-0.38	-0.14	-0.19	0.45	0.27	-0.19
synflow	0.00	0.12	0.27	0.34	0.00	0.00	0.34
l2_norm	-0.20	0.08	0.28	0.15	0.30	0.18	0.15
#params	-0.18	0.16	0.32	0.15	0.30	0.06	0.15
zen	-0.01	0.10	0.27	0.24	0.38	0.27	0.24
jacov	0.45	0.07	0.19	0.19	0.50	0.57	0.19
nwot	0.67	0.83	0.89	0.48	0.78	0.80	0.76
MeCo (Ours)	0.51	0.59	0.81	0.17	0.80	0.62	0.23
MeCoopt (Ours)	0.74	0.73	0.76	0.48	0.76	0.63	0.33
E.4 MeCo on AutoFormer and MobileNet OFA
AutoFormer. Chen et al. [49] proposed a novel one-shot architecture search framework for transformer-based models. We load the trained supernets and regenerate the candidate subnets. We then re-evaluate the subnets to obtain the accuracy on ImageNet and compute MeCo. The correlation of MeCo and model accuracy on AutoFormer is 0.45.
OFA. To solve the problem of efficient inference across devices and resource constraints, Cai et al. [50] proposed to train a once-for-all (OFA) network, which supports diverse architectural settings. We randomly sample 1,000 subnets from the OFA network and use the accuracy of the predictor predictions as the test accuracy. The Spearman correlation between MeCo and test accuracy is 0.86.
F More experimental results of NAS with MeCo
In this section, we provide more results and comparisons of our zero-shot NAS on NATS-Bench-TSS and DARTS-CNN. In the following descriptions, we denote multi-shot, one-shot, and zero-shot as MS, OS, and ZS, respectively.
F.1 Results on NATS-Bench-TSS
In the NATS-Bench-TSS search space, we evaluate all networks using zero-cost proxies and choose the Top-10 networks with the highest scores. Then we calculate the average test accuracy and standard deviation. The results are summarized in Table 12. As shown in the table, the networks searched by MeCo have the highest average precision on CIFAR-10, which is 0.08 higher than the best baseline proxy ZiCo. For CIFAR-100 and ImageNet16-120, MeCo achieves competitive results,
e.g.,  $70.86\% \pm 0.96\%$  with CIFAR-100 and  $42.59\% \pm 1.77\%$  with ImageNet16-120. In all, the results demonstrate that our MeCo has a great advantage in evaluating network performance, considering MeCo only requires one data sample as input.
We further combine MeCo with Zero-Cost-PT [55] and search for the best architecture three times with different seeds. The accuracy of the selected architectures as well as the comparisons with the baseline methods are presented in Table 13. It can be shown from the results that our MeCo-based NAS achieves competitive results with the SOTA baselines, e.g., synflow, zen, and ZiCo. However, MeCo-based NAS invokes one data for a single forward pass, thus being more resource-saving.
Table 12: The average test accuracy of Top-10 architectures obtained by various zero-cost proxies on NATS-Bench-TSS using CIAFR-10, CIAFR-100, and ImageNet16-120, respectively
Dataset	MeCo	Baselines
grasp[21]	fisher [23]	grad_norm[8]	snip [20]	synflow[22]	jacov[7]	NTK[24]	zen[27]	ZiCo[9]
CIFAR-10	93.64	89.34	89.27	89.27	89.27	93.27	91.23	92.67	58.99	93.56
±0.31	±2.16	±2.10	±2.10	±2.10	±0.74	±1.01	±0.46	±3.44	±0.23
CIFAR-100	70.86	60.89	61.06	60.89	60.89	71.12	68.50	69.31	12.73	70.64
±0.96	±3.88	±4.02	±3.88	±3.88	±1.59	±1.21	±1.17	±1.33	±0.28
ImageNet16-120	42.59	22.99	24.10	23.35	23.35	42.65	40.59	39.98	15.10	42.74
±1.77	±11.01	±11.36	±11.44	±11.44	±3.59	±1.86	±1.73	±0.51	±1.78
Table 13: The test accuracy of optimal architectures obtained by Zero-Cost-PT with various zero-cost proxies on NATS-Bench-TSS using CIAFR-10, CIFAR-100, and ImageNet16-120, respectively
Dataset	MeCo	Baselines
grasp[21]	fisher [23]	grad_norm[8]	snip [20]	synflow[22]	jacov[7]	NTK[24]	zen[27]	ZiCo[9]
CIFAR-10	93.76	92.59	88.39	91.64	90.11	93.76	91.92	92.61	93.76	93.76
±0	±1.12	±2.55	±0.68	±2.85	±0	±2.40	±0.65	±0	±0
CIFAR-100	71.11	68.98	65.77	65.20	65.29	71.11	69.67	68.27	71.11	71.11
±0	±2.69	±0.93	±0.56	±0.96	±0	±2.39	±2.34	±0	±0
ImageNet16-120	41.44	35.29	28.91	35.82	37.38	41.44	40.35	41.25	41.44	41.44
±0	±8.03	±5.18	±3.99	±4.41	±0	±6.56	±2.37	±0	±0
F.2 Results on DARTS-CNN
For DARTS-CNN search space, we search the architectures by Zero-Cost-PT with different zero-cost proxies on CIFAR-100. Each searched network is trained five times, and the results are summarized in Table 14. There are two settings in our experiments on DARTS-CNN: networks initialized with 16 channels and trained as in Table 7, and networks initialized with 36 channels and trained as in [63]. It can be shown from Table 14 that MeCo achieves competitive results compared with MS, OS, and ZS baselines. More concretely, MeCo-based NAS obtains  $19.33\%$  test error with 0.08 GPU days, which outperforms all the ZS methods under the same settings. On the other hand, compared with the manual, MS, and OS methods, MeCo is also competitive. For example, MeCo-based NAS achieves  $83.14\%$  accuracy, which is only  $0.34\%$  lower than the baseline method  $\beta$ -DARTS [64], but five times more efficient in computation.
We further visualize the networks searched by Zero-Cost-PT with different proxies on DARTS-CNN. The results on CIFAR-10 and CIFAR-100 are presented in Figure 6 and Figure 7, respectively.
F.3 Results on MobileNet V3
We search the architectures on MobileNet space using Algorithm 1. We retrain the searched network using ImageNet-1K for 480 epochs with a batch size of 256 and input resolution  $224 \times 224$ . The results are summarized in Table 15. MeCo achieves  $77.8\%$  Top-1 accuracy, which is  $0.7\%$  higher than the SOTA baseline method ZiCo. Moreover, our method only takes 0.04 GPU days, which outperforms the SOTA methods.
G Limitations and future works
Table 14: Comparison of our method with SOTA NAS methods using DARTS-CNN and CIFAR-100.
Approach	Test Error (%)	Search Cost (GPU Days)	Params (MB)	Method
DenseNet-BC[65]	17.18	-	25.6	Manual
NASNet-A[12]	16.82	2000	3.3	MS
DARTS(1st)[6]	17.76	1.5	3.3	OS
SNAS [66]	17.55	1.5	2.8	OS
P-DARTS[64]	15.92 ± 0.18	0.4	3.7	OS
R-DARTS[67]	18.01 ± 0.26	-	-	OS
PC-DARTS[63]	16.9	0.1	3.6	OS
β-DARTS[68]	16.52 ± 0.03	0.4	3.83 ± 0.08	OS
Zero-Cost-PT†synflow [22]	19.82 ± 0.35	0.04	1.2	ZS
Zero-Cost-PT†fisher [23]	21.14 ± 0.24	0.06	0.7	ZS
Zero-Cost-PT†grasp [21]	22.65 ± 0.30	0.13	0.7	ZS
Zero-Cost-PT†jakov [7]	22.90 ± 0.35	0.04	0.6	ZS
Zero-Cost-PT†snip [20]	19.95 ± 0.28	0.04	0.8	ZS
Zero-Cost-PT†NTK [24]	20.30 ± 0.33	0.19	0.9	ZS
Zero-Cost-PTZiCo† [9]	19.54 ± 0.28	0.06	1.1	ZS
Zero-Cost-PT†MeCo	19.33 ± 0.25	0.08	0.8	ZS
Zero-Cost-PT‡MeCo	16.86 ± 0.30	0.08	3.7	ZS
†: networks initialized with 16 channels and trained as settings in 7.
: networks initialized with 36 channels and trained as settings in [63].