SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tangyu Jiang†
jty@mail.bnu.edu.cn
Haodi Wang†
whd@mail.bnu.edu.cn
Rongfang Bie*
rfbie@bnu.edu.cn
Neural Architecture Search (NAS) is a promising paradigm in automatic architecture engineering. Zero-shot NAS can evaluate the network without training via some specific metrics called zero-cost proxies. Though effective, the existing zero-cost proxies either invoke at least one backpropagation or depend highly on the data and labels. To alleviate the above issues, in this paper, we first reveal how the Pearson correlation matrix of the feature maps impacts the convergence rate and the generalization capacity of an over-parameterized neural network. Enlightened by the theoretical analysis, we propose a novel zero-cost proxy called MeCo, which requires only one random data for a single forward pass. We further propose an optimization approach  $\mathrm{MeCo}_{\mathrm{opt}}$  to improve the performance of our method. We design comprehensive experiments and extensively evaluate MeCo on multiple popular benchmarks. MeCo achieves the highest correlation with the ground truth (e.g., 0.89 on NATS-Bench-TSS with CIFAR-10) among all the state-of-the-art proxies, which is also fully independent of the data and labels. Moreover, we integrate MeCo with the existing generation method to comprise a complete NAS. The experimental results illustrate that MeCo-based NAS can select the architecture with the highest accuracy and a low search cost. For instance, the best network searched by MeCo-based NAS achieves  $97.31\%$  on CIFAR-10, which is  $0.04\%$  higher than the baselines under the same settings. Our code is available at https://github.com/HamsterMimi/MeCo.
Deep Neural Networks (DNNs) have been ubiquitously adopted in various fields due to their ability in hierarchical feature extraction [1-5]. It is a common consensus that the architecture of DNN is essential in model performance. Unfortunately, the manual trial-and-error method is non-scalable and inefficient, making it infeasible to find the globally optimal structure. Thanks to the development of the Neural Architecture Search (NAS), it is possible to automatically select the network architecture with the best performance.
There are abundant works proposed to improve the performance of NAS. The most fundamental observation is to fully or partially train the candidate networks with specific data and select the network with the highest accuracy. Though effective, these multi-shot NAS methods are inevitably resource-consuming, thus, are non-trivial to be adopted in real-world applications. To address this issue, the one-shot NAS has been proposed in which a supernet is designed and only requires one-time full training [6]. In 2020, Mellor et al. [7] proposed the zero-shot NAS and realized an efficient approach in which a zero-cost proxy is proposed to evaluate the networks without training.
Despite the effectiveness of the existing zero-cost proxies, there are some critical issues that remain to be solved. (i) Almost all current proxies are established from the network gradients. As a consequence, computing these proxies requires at least one backpropagation, which is resource-consuming. (ii) The majority of the currently used proxies rely highly on the input data and labels. The reliance on the input samples underestimates the evaluation of the network's intrinsic characteristics, and the incorrect labels (which are commonly occurred in real-world datasets) may obfuscate the ranking results. (iii) Although various zero-cost proxies have been proposed, the performance of those proxies can still be promoted in network evaluation.
Objectives. The main objective of this paper is to design a novel zero-cost proxy with better performance than the current methods. The proposed proxy should be easily computed by a small number of data without backpropagation. Furthermore, to estimate the intrinsic trait of the architectures and avoid the label influence, our proxy should be label-independent.
Our Contributions. To this end, in this paper, we craft a novel zero-cost proxy for zero-shot NAS from a new perspective. The proposed proxy, called MeCo, outperforms the State-Of-The-Art (SOTA) zero-cost proxies in NAS and only requires one data sample for a single forward pass. Specifically, we harness a novel observation on the multi-channel convolutional layers and view them as the "multi-sample fully-connected layers with constraints". This insight permits MeCo to be computed by only one data. Moreover, we rigorously analyze how the Pearson correlation matrix of the feature maps impacts the training convergence rate and the generalization capacity of the networks, which theoretically proves the effectiveness of MeCo. Unlike the existing methods [7-9], MeCo is established upon each layer of the feature maps and hence requires only a single forward pass. Based on our design, we further propose an optimization approach  $\mathsf{MeCo}_{\mathrm{opt}}$  to promote the performance of our method. The experimental results show that both MeCo and  $\mathsf{MeCo}_{\mathrm{opt}}$  achieve a higher correlation with the network ground truth than the SOTA methods. For example, the Spearman correlation coefficients of MeCo on NATS-Bench-TSS with CIFAR-10 and CIFAR-100 are 0.89 and 0.88, respectively, which are 0.09 and 0.08 higher than the best baseline method. Our MeCo-based zero-shot NAS also acquires the highest accuracy among all the existing approaches, e.g., MeCo-based NAS selects the network with  $97.31\%$  accuracy, which is  $0.04\%$  higher than the SOTA work. In all, our major contributions are as follows.
- We theoretically analyze how the Pearson correlation matrix of the feature maps impact the training convergence rate and the generalization capacity of the networks, based on the characteristics of the multi-channel convolutional layers.
- We propose MeCo, a novel zero-cost proxy for zero-shot NAS that achieves better performance than SOTA methods. Through systematic analysis, we further proposed  $\mathrm{MeCo}_{\mathrm{opt}}$  as an optimization approach. Our proxy is fully training-free and data-independent, which requires only one data sample with a single forward pass.
- We rigorously implement our MeCo and extensively design the experiments to evaluate its performance on several popular benchmarks (e.g., NAS-Bench-101, NATS-Bench-TSS, NATS-Bench-SSS, NAS-Bench-301, and Transbench-101) with diverse datasets and random data samples. The experimental results show that our zero-cost proxy outperforms the existing methods, and MeCo-based NAS can select the network with the highest accuracy.
2.1 Zero-shot NAS and zero-cost proxy
Zero-shot NAS uses specific metrics called zero-cost proxies to evaluate the networks. Unlike the traditional multi-shot and one-shot strategies [10, 6, 11-19], zero-shot NAS eliminates the training procedure thus significantly improves the efficiency. Various studies have been brought forward to
promote the quality of the zero-cost proxies, and most of them are based on the gradients. Mellor et al. [7] firstly evaluated the performance of the initialized networks using the activation overlaps between data points. Abdelfattah et al. [8] proposed a series of parameter-pruning based proxies including snip [20], grasp [21], synflow [22], and fisher [23]. Recently, Li et al. [9] proposed ZiCo, which firstly works better than the parameter amounts. These approaches need at least one backpropagation and rely on the data labels to compute the proxies. Another line of work [24-26] is based on the theory of deep learning such as NTK, which is independent of the labels yet still needs backpropagation. For example, Chen et al. [24] utilized the condition number of NTK to evaluate the trainability of the networks. Finally, there are a few works that do not require backpropagation or data labels. For instance, Lin et al. [27] proposed Zen-Score, in which they design an efficient zero-cost proxy with Gaussian random inputs. However, Zen-Score requires multiple forward passes while our MeCo only needs a single forward pass.
2.2 Over-parameterized networks
Over-parameterized networks have received a lot of attention due to their outstanding effect and ease of optimization. Jacot et al. [28] demonstrated that the training dynamic of an infinite-width network follows a kernel called NTK [29, 30]. For finite-width networks, the training dynamic can be illustrated by a gram matrix [31, 32]. Further, Du et al. [31] proved that the loss of an overparameterized network could converge to a global minimum [33-35], and the training convergence rate could be characterized by the minimum eigenvalue of the gram matrix. Some works also discuss the optimization of the training convergence rate [36] and the topological properties of DNNs [37].
3 MeCo: minimum eigenvalue of correlation on feature maps
Notations. We define  $[n] = \{1,2,\dots ,n\}$ . The lower and uppercase bold font represent vectors and matrices, respectively, e.g.,  $\mathbf{x}$  is a vector with entry  $x_{i}$ , and  $\mathbf{M}$  is a matrix with entry  $[\mathbf{M}]_{ij}$ . The minimum eigenvalue of  $\mathbf{M}$  is denoted as  $\lambda_{\mathrm{min}}(\mathbf{M})$ . We define  $\mathbf{A} \circ \mathbf{B}$  as Hadamard product between two same-sized matrices  $\mathbf{A}$  and  $\mathbf{B}$ .  $\mathbf{1}_{m\times n}$  represents a  $m\times n$  matrix filled by ones and  $\| \cdot \| _2$  is used to represent the  $l_{2}$  norm of a vector.  $N(\mathbf{0},\mathbf{I})$  and  $U\{S\}$  represent the standard Gaussian distribution and uniform distribution over a set  $S$ , respectively. We denote by  $\mathbf{X} = \{(\mathbf{x}_i,y_i)|\mathbf{x}_i\in \mathbb{R}^{d\times 1},y_i\in \mathbb{R},i\in [n]\}$  the training set, where  $\mathbf{x}_i$  and  $y_{i}$  represent the  $i$ -th data and label.  $\mathbb{I}\{\cdot \}$  is defined as the indicator function that demonstrates the event occurrence, such that for event  $\mathcal{A}$ ,  $\mathbb{I}\{\mathcal{A}\} = 1$  if and only if  $\mathcal{A}$  happened, otherwise it equals to 0.
For input  $\mathbf{x} \in \mathbb{R}^{d \times 1}$ , weight vector  $\mathbf{w} \in \mathbb{R}^{d \times 1}$  in the weight matrix  $\mathbf{W} \in \mathbb{R}^{d \times m}$ , and output weight  $\mathbf{a} \in \mathbb{R}^{m \times 1}$ , we denote  $f(\mathbf{W}, \mathbf{a}, \mathbf{x})$  as a neural network with a single hidden layer such that
$$
f (\mathbf {W}, \mathbf {a}, \mathbf {x}) = \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} a _ {r} \sigma \left(\mathbf {w} _ {r} ^ {T} \mathbf {x}\right) \tag {1}
$$
where  $\sigma$  is the activation function. In this paper, we mainly consider the ReLU function due to its effectiveness, i.e.,  $\sigma (z) = z\mathbb{I}\{z > 0\}$ . Given a training set  $\mathbf{X}$ , the optimization goal is to minimize the empirical risk loss function
$$
L (\mathbf {W}, \mathbf {a}) = \sum_ {i = 1} ^ {n} \frac {1}{2} \left(f \left(\mathbf {W}, \mathbf {a}, \mathbf {x} _ {i}\right) - y _ {i}\right) ^ {2} \tag {2}
$$
We follow the definitions in [31] and define the matrices  $\mathbf{H}(t)$  and  $\mathbf{H}^{\infty}$  as follows.
Definition 1 (Gram Matrix). For a neural network with a single hidden layer, the gram matrix  $\mathbf{H}(t) \in \mathbb{R}^{n \times n}$  induced by the ReLU activation function on a training set  $\mathbf{X} := \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  with entry  $[\mathbf{H}(t)]_{ij}$  is defined as:
$$
[ \mathbf {H} (t) ] _ {i j} = \frac {1}{m} \sum_ {r = 1} ^ {m} \mathbf {x} _ {i} ^ {T} \mathbf {x} _ {j} \mathbb {I} \left\{\mathbf {w} _ {r} ^ {T} (t) \mathbf {x} _ {i} \geq 0, \mathbf {w} _ {r} ^ {T} (t) \mathbf {x} _ {j} \geq 0 \right\} \tag {3}
$$
where  $\mathbf{w}_r(t)$  is a vector that depends on  $t$ . We further construct  $\mathbf{H}^\infty$  with entry  $[\mathbf{H}^\infty]_{ij}$  such that
$$
\left[ \mathbf {H} ^ {\infty} \right] _ {i j} = \mathbb {E} _ {\mathbf {w} \sim N (\mathbf {0}, \mathbf {I})} \left[ \mathbf {x} _ {i} ^ {T} \mathbf {x} _ {j} \mathbb {I} \left\{\mathbf {w} ^ {T} \mathbf {x} _ {i} \geq 0, \mathbf {w} ^ {T} \mathbf {x} _ {j} \geq 0 \right\} \right] \tag {4}
$$
We denote  $\lambda_0\coloneqq \lambda_{\mathrm{min}}(\mathbf{H}^{\infty})$
Figure 1: Conversion between the multi-channel convolution and multi-sample fully-connected operation. The input size is  $3 \times w \times h$  and each filter size is  $3 \times 3 \times 3$ . Each input channel can be flattened with size  $d \times 1$ ,  $d = w \times h$ . We collect the three flattened samples to obtain the output with constrained fully-connected operations (dot lines for zero operations and solid lines for full connection). The final output is computed by a sum operation (dashed lines) from the hidden nodes.
Remark 1. Gram matrix  $\mathbf{H}(t)$  reflects the prediction dynamic at the  $t$ -th iteration of the network training. Du et al. [31] proved that for all  $t \geq 0$ ,  $\| \mathbf{H}(t) - \mathbf{H}^{\infty}\|_2 \to 0$  as  $m \to \infty$ . Moreover, the gram matrix  $\mathbf{H}^{\infty}$  has the following key property: if  $\mathbf{x}_i \nVdash \mathbf{x}_j$ ,  $\forall i \neq j$ , then  $\lambda_0 > 0$ . The proof of this property can be found in Theorem 3.1 in [31].
Definition 2 (Pearson Correlation Matrix).  $P(\mathbf{X})$  is a Pearson correlation matrix with the  $(i,j)$ -th entry  $[P(\mathbf{X})]_{ij}$  as
$$
[ P (\mathbf {X}) ] _ {i j} = \rho \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right) = \frac {\mathbb {E} \left[ \left(\mathbf {x} _ {i} - \mu_ {\mathbf {x} _ {i}}\right) \left(\mathbf {x} _ {j} - \mu_ {\mathbf {x} _ {j}}\right) \right]}{\sigma_ {\mathbf {x} _ {i}} \sigma_ {\mathbf {x} _ {j}}} \tag {5}
$$
where  $\mu_{\mathbf{x}}$  and  $\sigma_{\mathbf{x}}$  are the mean and standard deviation of  $\mathbf{x}$ , respectively.
Remark 2. Pearson correlation matrix  $P(\mathbf{X})$  is closely related to  $\mathbf{H}(t)$ . For instance, suppose  $\mathbf{x}_1, \mathbf{x}_2 \sim N(\mathbf{0}, \mathbf{I})$ ,  $\mathbf{x}_i \in \mathbb{R}^{d \times 1}$  and  $\mathbf{w}_r^T(t)\mathbf{x}_i \geq 0$ , for  $r \in [m]$  and  $i \in [2]$ , then  $[\mathbf{H}(t)]_{12} = (d - 1)[P(\mathbf{X})]_{12}$ . We will use this characteristic to approximate the training convergence rate and generation capacity of the networks when designing our proxy.
3.2 The construction of MeCo
To present the theoretical analysis, we first approximate the Convolutional Neural Network (CNN) to an over-parameterized Neural Network (NN) and then craft our proxy.
3.2.1 Back to convolution
The convolutional layers are the fundamental parts of CNN. Suppose we have the input  $\mathbf{x}_{\mathrm{in}} \in \mathbb{R}^{c_{\mathrm{in}} \times w \times h}$  and the filter  $\mathbf{w} \in \mathbb{R}^{c_{\mathrm{in}} \times k \times k}$ . We set the stride size to one and retain the size of the feature maps via zero padding, then the convolutional layer can be formally expressed as:
$$
[ \mathbf {x} _ {\text {o u t}} ] _ {i, j} = \sum_ {c = 1} ^ {c _ {\mathrm {i n}}} \sum_ {a = - p} ^ {p} \sum_ {b = - p} ^ {p} [ \mathbf {w} ^ {c} ] _ {a + p + 1, b + p + 1} \cdot [ \mathbf {x} _ {\mathrm {i n}} ^ {c} ] _ {a + i, b + j} \tag {6}
$$
where  $\mathbf{x}_{\mathrm{in}}^c$  (resp.  $\mathbf{w}^c$ ) represents the  $c$ -th channel of  $\mathbf{x}_{\mathrm{in}}$  (resp.  $\mathbf{w}$ ),  $p = (k - 1) / 2$ ,  $i \in [w]$ ,  $j \in [h]$ . In this paper, we observe that  $\mathbf{x}_{\mathrm{out}}$  and  $\mathbf{x}_{\mathrm{in}}^c$  can be flattened to one-dimensional vectors  $\tilde{\mathbf{x}}_{\mathrm{out}} \in \mathbb{R}^{d \times 1}$  and  $[\tilde{\mathbf{x}}_{\mathrm{in}}^c] \in \mathbb{R}^{d \times 1}$ , where  $c \in [c_{\mathrm{in}}]$ ,  $d = w \times h$ . More concretely, we have the following transformations:
$$
\tilde {\mathbf {x}} _ {\text {o u t}} = \sum_ {c = 1} ^ {c _ {\mathrm {i n}}} \mathbf {A} _ {c} \sigma \left(\left(\mathbf {B} _ {c} \circ \mathbf {W}\right) ^ {T} \tilde {\mathbf {x}} _ {\mathrm {i n}} ^ {c}\right) \tag {7}
$$
$$
s. t. \quad \mathbf {A} _ {c} \in \mathbb {R} ^ {d \times d _ {h}}, [ \mathbf {A} _ {c} ] _ {i j} = \mathbb {I} \{j = (c - 1) d + i \}
$$
$$
\mathbf {B} _ {c} \in \mathbb {R} ^ {d _ {h} \times d}, [ \mathbf {B} _ {c} ] _ {i j} = \mathbb {I} \{(c - 1) d <   i \leq c d \}, \tag {8}
$$
$\mathbf{B}_c \circ \mathbf{W}$  satisfies weight sharing constraints
where  $d_h = c_{\mathrm{in}} \times d$ ,  $\mathbf{W} \in \mathbb{R}^{d \times d_h}$  is the weight matrix. Hence, the convolutional layer in CNN is equivalent to a fully-connected layer with constraints, i.e., shown in Equation 8. To further simplify the problem, we relax the second constraint to  $\mathbf{B}_c = \mathbf{1}_{d_h \times d}$  and ignore the last constraint. As a consequence, we obtain a multi-sample fully-connected network in which each flattened channel  $\tilde{\mathbf{x}}_{\mathrm{in}}^c$  is regarded as an independent data sample, and the total number of samples is  $c_{\mathrm{in}}$ . To better illustrate our insight, we present an example of the conversion procedure in Figure 1.
3.2.2 Over-parameterized neural networks
The over-parameterized NNs are competitive in hierarchical feature extraction due to a large number of parameters. One of the typical architectures is the networks with wide hidden layers, which is proved to be tractable in training [38]. In the previous subsection, we convert a multi-channel convolutional layer to a multi-sample fully-connected layer with constraints. We further argue that if the number of hidden nodes in the transformed fully-connected layer is large enough, then it can be viewed as an over-parameterized NN layer. Therefore, the characteristics of the overparameterized NN can be transferred to CNN. To this end, we present the following theorem [31] about the convergence rate of the fully-connected NN with a single hidden layer as follows.
Theorem 1. If gram matrix  $\mathbf{H}^{\infty} \succ 0$ ,  $\| \mathbf{x}_i\|_2 = 1$ ,  $|y_i| < C$  for some constant  $C$  and  $i \in [n]$ , hidden nodes  $m = \Omega \left(\frac{n^6}{\lambda_0^4\delta^3}\right)$ , and i.i.d. initialize  $\mathbf{w}_r \sim N(\mathbf{0}, \mathbf{I})$ ,  $a_r \sim U\{-[-1,1]\}$  for  $r \in [m]$ , then with probability at least  $1 - \delta$  over the initialization, the following inequality holds:
$$
\left\| f (\mathbf {W} (t), \mathbf {a}, \mathbf {X}) - \mathbf {y} \right\| _ {2} ^ {2} \leq \exp (- \lambda_ {0} t) \| f (\mathbf {W} (0), \mathbf {a}, \mathbf {X}) - \mathbf {y} \| _ {2} ^ {2} \tag {9}
$$
Remark 3. This inequality shows that  $\lambda_0$  positively affects the training convergence rate of the network. We further point out that the convergence rate of the network is independent of the label  $y$ , which makes it possible to measure the network performance with only a single forward pass. Based on this observation, we establish our proxy on the intermediate feature maps to measure the performance of the network.
Based on Theorem 1 and Remark 2, we further analyze the relationship between the minimum eigenvalue of the matrix  $P(\mathbf{X})$  and the training convergence rate of the over-parameterized network. We summarize our results in the following theorem:
Theorem 2. Suppose  $f$  is an NN with a single hidden layer and ReLU activation function. Assume  $\mathbf{X} \in \mathbb{R}^{d \times n}$ ,  $\mathbf{w}(0) \sim N(\mathbf{0}, \mathbf{I})$ ,  $P(\mathbf{X}) \succ 0$ ,  $p_0 \coloneqq \lambda_{\min}(P(\mathbf{X}))$ , and hidden nodes  $m = \Omega \left( \frac{n^6 d^2}{\lambda_0^4 \delta^3} \right)$ , then the following formula holds with probability at least  $1 - \delta$  over the initialization
$$
\left\| f (\mathbf {W} (t), \mathbf {a}, \mathbf {X}) - \mathbf {y} \right\| _ {2} ^ {2} \leq \exp (- c p _ {0} t) \| f (\mathbf {W} (0), \mathbf {a}, \mathbf {X}) - \mathbf {y} \| _ {2} ^ {2} \tag {10}
$$
where  $c$  is a constant depending on  $m$ , and  $d$ .
Proof sketch. The key of the proof is to find the relationship between  $\lambda_0$  and  $p_0$  by Hoeffding's inequality and Weyl inequalities [39]. We provide the full proof in Appendix A.1. Theorem 2 shows that  $p_0$  can reflect the convergence rate of the network to a certain extent. The advantage of  $P(\mathbf{X})$  over  $\mathbf{H}(t)$  is that its computation only depends on the data. Therefore, it is feasible to estimate the training convergence rate of the network only through the feature maps.
Other than the convergence rate, we also analyze the relationship between  $p_0$  and the generalization capacity of the over-parameterized NN. We present the results in the following theorem:
Theorem 3. For an over-parameterized neural network with the loss on the testing set as  $L(\mathbf{W})$ . Let  $\mathbf{y} = (y_1, \dots, y_N)^T$ , and  $\gamma$  be the step of SGD,  $\gamma = \kappa C_1 \sqrt{\mathbf{y}^T (\mathbf{H}^\infty)^{-1} \mathbf{y}} / (m \sqrt{N})$  for some small enough absolute constant  $\kappa$ . Under the assumption of Theorem 2, for any  $\delta \in (0, e^{-1}]$ , there exists  $m^*(\delta, N, \lambda_0)$ , such that if  $m \geq m^*$ , then with probability at least  $1 - \delta$ , we have
$$
\mathbb {E} [ L (\mathbf {W}) ] \leq \mathcal {O} \left(C ^ {\prime} \sqrt {\frac {\mathbf {y} ^ {T} \mathbf {y}}{p _ {0} N}}\right) + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right) \tag {11}
$$
where  $C,C^{\prime},\delta$  are constants.
Proof sketch. The proof of the above theorem derives from the Corollary 3.10 of [40] and Section D.2 of [26]. We present the detailed proof in Appendix A.2.
3.2.3 New zero-cost proxy: MeCo
The above theorem and analysis indicate the following key insights:
- A multi-channel CNN layer can be viewed as a multi-sample over-parameterized NN layer.
- The minimum eigenvalue of  $P(\mathbf{X})$  positively affects the training convergence rate and the generalization capacity of an NN, which is independent of the labels.
Enlightened by these observations, we harness the Minimum eigenvalue of the Pearson Correlation matrix upon each layer of the feature maps to craft our novel zero-cost proxy called MeCo, such that
Definition 3 (MeCo). Assume the  $NNf(\cdot ;\theta)$  has a total of  $D$  layers, then MeCo is defined as
$$
\operatorname {M e C o} := \sum_ {i = 1} ^ {D} \lambda_ {\min } \left(P \left(f ^ {i} (\mathbf {X}; \theta)\right)\right) \tag {12}
$$
where  $f^i(\mathbf{X};\theta)$  represents the  $i$ -th feature map with the initialized parameters  $\theta$  on dataset  $\mathbf{X}$ .
Remark 4. Note that the Pearson correlation matrix contains the process of data normalization, thus our MeCo eliminates the deviation caused by the data, which is more conducive to discovering the essence of the network architectures.
4 Experiments and evaluations
We evaluate MeCo from three aspects: (i) The correlation with the ground truth and comparison with the baseline proxies; (ii) The dependency on data/labels; (iii) Performance of MeCo-based NAS scheme. We first describe the experimental configurations and then give the detailed performance.
4.1 Experimental configurations
Hardware. We fully implement our MeCo in Python. We evaluate MeCo on a desktop with an Inter Core i7-12700F CPU and GeForce RTX 3090.