SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Benchmarks.** We use several popular benchmarks in NAS. (i) NATS-Bench-TSS [41, 42] contains 15,625 CNN architectures with different topologies. We demonstrate the performance of the zero-cost proxied on NATS-Bench-TSS with CIFAR-10 [43], CIFAR-100 [43], ImageNet16-120[44], and three extra datasets [45] in Appendix E.1. (ii) NATS-Bench-SSS [42] contains 32,768 CNN architectures with the same topology but with different numbers of channels. We show the results on NATS-BenchSSS with CIFAR-10, CIFAR-100, and ImageNet16-120. (iii) NAS-Bench-301 [46] is a surrogate NAS benchmark for the DARTS search space. We illustrate the results with CIFAR-10 and the extra three datasets in Appendix E.1. (iv) TransBench-101 [47] is a benchmark composed of two subsets, i.e., TransBench-101-Micro and TransBench-101-Macro. The former is evaluated with 10 datasets and the latter with 7 datasets. We present these results in Appendix E.3. (v) NAS-Bench-101 [48] includes 423,624 cell-based CNN architectures, which are trained and evaluated on the CIFAR-10 dataset and shown in Appendix E.2. (vi) AutoFormer [49] and MobileNet OFA [50]. The description and results are shown in Appendix E.4.
Baselines. The baselines for zero-cost proxies include the number of parameters #Param, grasp [21], fisher [23], snip [20], synflow [22], jacov [7], grad_norm [8], NTK [24], zen [27], NASWOT [7], KNAS [51], NASI [52], GradSign [53], and ZiCo [9]. Specific forms of these proxies are provided in Appendix B. We adopt the Spearman rank correlation coefficient  $\rho$  between zero-cost proxies and the dataset ground truth to measure their performance.
4.2 Performance of our MeCo and comparisons
We evaluate  $\rho$  between MeCo and the ground truth, and compare it with the baseline zero-cost proxies on various benchmarks. We present the results on NATS-Bench-TSS and NATS-Bench-SSS in Table 1, and the results on NAS-Bench-301 are provided in Table 2. We run MeCo for ten times with different inputs and present the mean and standard deviation.
NATS-Bench-TSS. We run our MeCo and all baseline proxies on 15,625 networks of NATS-Bench-TSS with three public datasets. As we can see in Table 1, MeCo achieves  $0.894 \pm 0.003$ ,  $0.883 \pm 0.005$ ,
Table 1: Spearman correlation coefficients  $\rho$  of proxies on NATS-Bench-TSS and NATS-Bench-SSS
Approach	NATS-Bench-TSS	NATS-Bench-SSS
CIFAT-10	CIFAR-100	ImageNet16	CIFAT-10	CIFAR-100	ImageNet16
grasp	0.39	0.46	0.45	-0.13	0.01	0.42
fisher	0.40	0.46	0.42	0.44	0.55	0.47
grad_norm	0.42	0.49	0.47	0.51	0.49	0.67
snip	0.43	0.49	0.48	0.59	0.62	0.76
synflow	0.74	0.76	0.75	0.81	0.80	0.57
#Param	0.72	0.73	0.69	0.72	0.73	0.84
NWOT	0.77	0.80	0.77	0.45	0.43	0.42
jacov	0.73	0.70	0.70	0.30	0.13	0.30
NTK	0.76	0.75	0.72	0.34	0.29	0.28
zen	0.38	0.36	0.40	0.69	0.71	0.87
KNAS	0.20	0.35	0.42	0.25	0.12	0.32
NASI	0.44	0.43	0.63	0.17	0.04	0.20
GradSign	0.77	0.79	0.78	0.21	0.16	0.04
ZiCo	0.80	0.81	0.79	0.73	0.75	0.88
MeCo (Ours)	0.894±0.003	0.883±0.005	0.845±0.004	-0.79±0.01	-0.87±0.01	-0.86±0.02
MeCoopt (Ours)	0.901±0.002	0.890±0.003	0.850±0.003	0.89±0.002	0.83±0.004	0.89±0.003
and  $0.845 \pm 0.004$  correlation on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively. The variation shows that our method is stable and independent of the input samples. The Spearman correlation coefficients are significantly higher than the baselines, e.g., the SOTA proxy ZiCo [9] only achieves 0.80, 0.81, 0.79 on three datasets, which is 0.06 to 0.09 lower than ours. Note that ZiCo is the first proxy that is proved to be consistently better than the naive proxy #Param and requires a batch of input data. On the other hand, we surpass all the previous works and only use one data sample. The high correlation coefficients demonstrate the effectiveness of MeCo, which is consistent with the theoretical analysis. To make our results more intuitive, we sample 2,000 networks randomly and present the relationship between MeCo and the ground truth on three public datasets. The results are shown in Figure 4(a) in the Appendix E, where the trend in the scatter charts effectively proves that our zero-cost proxy has a highly strong correlation with the architecture quality.
NATS-Bench-SSS. The results of MeCo on 32,768 networks of NATS-Bench-SSS are summarized in Table 1. Among all the datasets, MeCo shows the highest correlation with the ground truth on CIFAR-100, i.e.,  $\rho = -0.87 \pm 0.01$ , which is 0.07 higher than the best baseline synflow. MeCo also achieves stable and competitive results on CIFAR-10 and ImageNet16-120 with one data as input. We find that MeCo is highly negatively correlated with the test accuracy, which is somehow counterintuitive. In fact, MeCo is sensitive to the number of channels (#channels) that cause this phenomenon. Specifically, the channels are viewed as the data samples as described in Section 3.2.1. This leads to the fact that the more channels contained, the larger the Pearson matrix, which will lead to smaller  $p_0$ . On the other hand, larger #channels will normally lead to higher accuracy of the architectures (yet lower
convergence rate). Thus, the variation of #channels in NATS-Bench-SSS(chosen from 8 to 64) leads to a negative correlation on this benchmark. In NATS-Bench-TSS, however, all the networks share the same #channels at each stage, thus MeCo does not show a negative correlation. To verify this, we generate random data with different numbers of channels and calculated  $p_0$  respectively, and demonstrate the results in Figure 2. We also sample 2,000 networks randomly and present the correlation of MeCo v.s. test accuracy in Figure 4(b) in the Appendix E.
Figure 2: The influence of different channel numbers of data on the minimum eigenvalue.
NATS-Bench-301. We demonstrate the results of MeCo and the baseline proxies on NATS-Bench-301 with CIFAR-10 in Table 2. It can be shown from the results that MeCo and  $\mathrm{MeCo}_{\mathrm{opt}}$  also achieve the highest correlation with the testing accuracy on NATS-Bench-301, which is 0.04-0.05 higher than the SOTA approach ZiCo. More experimental results of MeCo can be found in Appendix E
Table 2:  $\rho$  between zero-cost proxies and test accuracy on NAS-Bench-301 with CIFAR-10
Dataset	MeCo	MeCo opt	Baselines
grasp	fisher	grad_norm	snip	synflow	l2_norm	#Param	zen	jacov	nwot	ZiCo
CIFAR-10	0.7±0.01	0.71±0.01	0.34	-0.28	-0.04	-0.05	0.18	0.45	0.46	0.43	-0.04	0.47	0.66
4.3 Dependency on data
MeCo is designed to be fully data-independent due to the characteristics of Pearson correlation matrix (Theorem 2). To further illustrate this attribute, we generate a random dataset and use it as the input of MeCo and NTK [24], which is also independent of the data and labels. We compute the Spearman correlation coefficients between MeCo and the test accuracy on three public datasets, and compare them with the baseline approach. All the results for MeCo is ran ten times with random inputs.
More concretely, the random data is generated following a Gaussian distribution and fed into the initialized network. Note that the baseline method requires a batch of random data as input, while MeCo only uses one. We present the results for NATS-Bench-TSS and NATS-Bench-SSS in Table 3.
The results show that even if we use random data as the inputs, MeCo is still able to maintain a high correlation with the corresponding ground truth. For example,  $\rho = 0.87$  on NATS-BenchTSS with CIFAR-10 and  $\rho = -0.93$  on NATS-Bench-SSS with ImageNet16-120, which are significantly higher than the data-independent proxy NTK. Note that most existing proxies rely on the gradients and backpropagation. Hence they cannot utilize the random input data for architecture evaluation, such as ZiCo and jacov.
Table 3: Spearman correlation coefficients of MeCo and NTK with random data on NATS-Bench-TSS and NATS-Bench-SSS
Dataset	NTAS-Bench-TSS	NATS-Bench-SSS
MeCo	NTK[24]	MeCo	NTK[24]
CIFAR-10	0.87±0.001	0.78	-0.87±0.000	0.23
CIFAR-100	0.85±0.001	0.79	-0.88±0.001	0.16
ImageNet16-120	0.83±0.001	0.76	-0.94±0.000	0.20
4.4 NAS with MeCo
We integrate our MeCo with the existing search space and generation strategy to comprise a complete NAS method. Specifically, we use NATSBench-TSS, DARTS-CNN [6], and MobileNet V3 [54] as the search spaces to demonstrate the performance. We use the Zero-Cost-PT [55] as the generation strategy. The algorithm of MeCo-based NAS and the results on MobileNet V3 are presented in Appendix C and F.3, respectively.
NATS-Bench-TSS. We directly evaluate all networks in NATS-Bench-TSS using MeCo and baseline proxies. The average test accuracy of the Top-10 networks on three datasets is summarized in Table 12 in Appendix F.1. As shown in the table, the average test accuracy on CIFAR-10 of MeCo is  $0.08\%$  higher than the best baseline
ZiCo. Moreover, we further search the networks using Zero-Cost-PT with different proxies and run the search algorithm three times. The networks we searched have the highest average test accuracy on three public datasets, that is,  $93.76\%$ ,  $71.11\%$ , and  $41.44\%$  on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively. The results are provided in Table 13 in Appendix F.1.
Figure 3: Spearman correlation coefficients of MeCo with different data size on NATS-BenchTSS and NATS-Bench-SSS, respectively.
DARTS-CNN. We compare our MeCo-based NAS with multi-shot, one-shot, and zero-shot NAS schemes in DARTS-CNN spaces. All the NAS schemes are executed on CIFAR-10 and CIFAR-100
datasets, respectively, to compare the Top-1 errors and the search cost. For zero-shot NAS, we use Zero-Cost-PT as the generation strategy due to its effectiveness. The results on CIFAR-10 are summarized in Table 4, where we denote multi-shot, one-shot, and zero-shot as MS, OS, and ZS, respectively. We present the CIFAR-100 results in Appendix F.2. The results demonstrate that our MeCo-based NAS outperforms the existing baseline methods. More concretely, for zero-shot methods, MeCo outperforms all the baselines for  $0.04\%$  to  $0.43\%$  on Top-1 error with a similar level of search cost under the same generation strategy. For training-based methods, MeCo-based NAS achieves competitive accuracy but with much less search cost. For instance, our approach requires only 0.08 GPU days and obtains  $97.31\%$  accuracy, while DARTS-PT costs  $10\times$  GPU days than ours with  $0.08\%$  higher accuracy.
Table 4: Comparisons of MeCo-based NAS with baselines using DARTS-CNN and CIFAR-10
Approach	Test Error (%)	Search Cost (GPU Days)	Params (M)	Method
AmoebaNet-A [56]	3.34 ± 0.06	3150	3.2	MS
PNAS [57]	3.41 ± 0.09	225	3.2	MS
DARTS (1st) [6]	3.00 ± 0.14	1.5	3.3	OS
DARTS-PT[58]	2.61 ± 0.08	0.8	3.0	OS
SDARTS-RS [59]	2.67 ± 0.03	0.4	3.4	OS
SGAS (Cri.1)[60]	2.66 ± 0.24	0.8	3.7	OS
Eigen-NAS[26]	7.4	-	-	ZS
TE-NAS [24]	2.63 ± 0.064	0.05	3.8	ZS
Zero-Cost-PTsynflow[22]	2.96 ± 0.11	0.03	5.1	ZS
Zero-Cost-PTfisher[23]	3.12 ± 0.16	0.05	2.5	ZS
Zero-Cost-PTgrasp[21]	2.73 ± 0.10	0.1	3.3	ZS
Zero-Cost-PTjakov[7]	2.88 ± 0.15	0.04	3.5	ZS
Zero-Cost-PTsnip[20]	2.90 ± 0.03	0.04	4.0	ZS
Zero-Cost-PTNTK[24]	2.89 ± 0.09	0.21	4.1	ZS
Zero-Cost-PTZiCo[9]	2.80 ± 0.03	0.04	5.1	ZS
Zero-Cost-PTMeCo(Ours)	2.69± 0.05	0.08	4.2	ZS
We remark that the experimental results in this section demonstrate the following properties of MeCo: (i) High correlation with the ground truth; (ii) Requires only one data without labels for a single forward pass. That is because MeCo is established upon the observations over the multi-channel CNN layers. Moreover, the minimum eigenvalue of the Pearson correlation matrix is independent of the data labels, and thus we exert MeCo on the feature maps instead of the gradients.
5 Discussion and Optimization
5.1 Optimization method  $\mathrm{MeCo}_{\mathrm{opt}}$
In the previous sections, we illustrated the theoretical basis of MeCo and evaluated our proposed proxy on various benchmarks. However, as described in Section 4.2, the negative correlation on NATS-Bench-SSS and some other tasks (e.g., Transbench-101 in the appendix) reflect that MeCo is highly sensitive to the number of channels. Although this phenomenon does not contradict the theoretical results or undermine the effectiveness of MeCo, it might be problematic when  $p_0 = 0$ . To this end, we present an optimization method on top of MeCo to alleviate the channel-sensitive trait.
Specifically, for convolutional layers,  $p_0$  is strictly greater than zero if for  $\forall i \neq j$ ,  $x_i \nmid x_j$ , and  $c < w \times h$ , where  $c$  is the number of channels,  $w, h$  are the size of the inputs. Thus, in real-world applications, MeCo might lose efficacy during the down-sampling procedure. To address this issue, instead of flattening all the channels of the feature map as described in Section 3.2.1, we randomly sample a fixed number of channels and flatten them as matrix  $P'$ . We then compute the final result by multiplying the minimum eigenvalue of  $\mathbf{P}'$  with a channel weight, such that
$$
\operatorname {M e C o} _ {\text {o p t}} := \sum_ {i = 1} ^ {D} \frac {c ^ {(i)}}{n} \cdot \lambda_ {\min } \left(P ^ {\prime}\right) \tag {13}
$$
where  $c^{(i)}$  is the number of channels in the  $i$ -th layer, and  $n$  is the fixed sampling numbers. The high-level idea of this optimization is to limit the dimension of the Pearson correlation matrix by constraining #channels. Instead of computing upon a large matrix, we calculate the minimum eigenvalue upon a fixed-sized matrix and then enlarge it with corresponding constants. Note that the bonus of this optimization is that the time cost can be controlled because the matrix dimension is significantly reduced compared to the original one.
To show the effectiveness of  $\mathrm{MeCo}_{\mathrm{opt}}$ , we calculate the Spearman correlation coefficients of  $\mathrm{MeCo}_{\mathrm{opt}}$  on NATS-Bench-TSS, NATS-Bench-SSS (Table 1), NATS-Bench-301 (Table 2 and Table 8), and Transbench-101 (Table 10 and Table 11). All the experiments are conducted under  $n = 8$ . The results demonstrate that  $\mathrm{MeCo}_{\mathrm{opt}}$  effectively solves the negative correlation issue on multiple benchmarks, e.g., NATS-Bench-SSS and TransBench-101. Meanwhile, the majority of the positive correlations are promoted up to 0.23, and the remaining results are not impacted.
Sample size. We consider the effect of the sample size on the performance of MeCo. We select the size of data from  $\{16, 32, 64, 128, 256\}$ , respectively, and calculate the Spearman rank correlation coefficient  $\rho$  between MeCo and the test accuracy of the networks on three public datasets. The results are summarized in Figure 3. For NATS-Bench-TSS,  $\rho$  increases rapidly before size 64 and grows slowly after that. The maximum value is acquired at size 128, which is slightly higher than size 64. On the other hand, the overall fluctuation is relatively large on NATS-Bench-SSS. The Spearman correlation coefficients are low when the size is smaller than 32, and reach higher at 32. Then  $\rho$  reduces as the sample size grows larger. The ablation results show that the most appropriate size of random data is around 32 and 64, which is also adopted in our settings.
Sample numbers. We explore the performance of MeCo regarding the sample numbers that are used as the inputs. We randomly choose a different number of samples to evaluate the architectures in the entire NATS-Bench-TSS and NATS-Bench-SSS, respectively. The results are shown in Figure 5. It can be demonstrated from the figure that MeCo is relatively stable when varying the number of samples. The performance of MeCo fluctuates around 0.01 throughout the experiments. These results support that MeCo only requires one data sample.
Fixed channel numbers  $n$ . We conduct an ablation study on  $n$  v.s.  $\mathrm{MeCo}_{\mathrm{opt}}$  with NATS-Bench-TSS and NATS-Bench-SSS to illustrate the best choice of  $n$ . The results are shown in Table 5.
Table 5: Spearman correlation coefficients  $\rho$  of proxies on NATS-Bench-TSS and NATS-Bench-SSS
Approach	NATS-Bench-TSS	NATS-Bench-SSS
CIFAT-10	CIFAR-100	ImageNet16	CIFAT-10	CIFAR-100	ImageNet16
n = 4	0.87±0.003	0.88±0.002	0.84±0.005	0.88±0.005	0.82±0.003	0.84±0.003
n = 6	0.88±0.004	0.88±0.001	0.84±0.004	0.88±0.007	0.83±0.002	0.85±0.006
n = 8	0.90±0.002	0.89±0.003	0.85±0.003	0.89±0.002	0.83±0.004	0.89±0.003
The results show that with  $n$  growing larger,  $\mathrm{MeCo}_{\mathrm{opt}}$  increases by 0.1 to 0.5. We set  $n = 8$  in our experiments to obtain the best performance.
In this work, we propose a novel feature-based zero-cost proxy called MeCo and its optimization method  $\mathrm{MeCo}_{\mathrm{opt}}$ . Unlike the existing methods, our zero-cost proxies require only one data for a single forward pass. Specifically, we theoretically approximate a multi-channel convolution layer to an over-parameterized NN layer. We then harness the relationship between the theoretical properties and the minimum eigenvalue of the Pearson correlation matrix to craft our new proxies. We rigorously implement our proxies and extensively design the experiments. The experimental results show that  $\mathrm{MeCo}$  and  $\mathrm{MeCo}_{\mathrm{opt}}$  significantly outperforms the SOTA zero-cost proxies, which is also independent of the data and labels. Moreover, MeCo can be integrated into a complete NAS and enables us to efficiently find the architecture with the highest accuracy.
We sincerely thank all the reviewers of NeurIPS'23 for their constructive suggestions which significantly improve the quality of our paper. The work was supported by National Key R&D Program of China [Grant No. 2022ZD0115901], in part by the National Natural Science Foundation of China (Grant No. 62177007), and the China-Central Eastern European Countries High Education Joint Education Project (Grant No. 202012).
[1] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference on Learning Representations, 2015.
[2] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[3] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779-788.
[4] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[6] H. Liu, K. Simonyan, and Y. Yang, "DARTS: Differentiable architecture search," in International Conference on Learning Representations, 2019.
[7] J. Mellor, J. Turner, A. Storkey, and E. J. Crowley, “Neural architecture search without training,” in International Conference on Machine Learning. PMLR, 2021, pp. 7588–7598.
[8] M. S. Abdelfattah, A. Mehrotra, L. Dudziak, and N. D. Lane, “Zero-cost proxies for lightweight nas,” in International Conference on Learning Representations, 2021.
[9] G. Li, Y. Yang, K. Bhardwaj, and R. Marculescu, “Zico: Zero-shot NAS via inverse coefficient of variation on gradients,” in The Eleventh International Conference on Learning Representations, 2023.
[10] B. Zoph and Q. Le, “Neural architecture search with reinforcement learning,” in International Conference on Learning Representations, 2017.
[11] L. Xie and A. Yuille, "Genetic cnn," in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1388-1397.
[12] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8697–8710.
[13] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, "Large-scale evolution of image classifiers," in International Conference on Machine Learning. PMLR, 2017, pp. 2902-2911.
[14] C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, "Auto-weka: Combined selection and hyperparameter optimization of classification algorithms," in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 2013, pp. 847-855.
[15] Y. Geifman and R. El-Yaniv, “Deep active learning with a neural architecture search,” Advances in Neural Information Processing Systems, vol. 32, 2019.
A.1 Proof of Theorem 2
In this section, we present the proof for our main theorem (Theorem 2 in Section 3.4). We first introduce the following definitions and lemmas to better illustrate our proof.
Lemma 1 (Lemma 3.1 in [31]). If  $m = \Omega\left(\frac{n^2}{\lambda_0^2}\log\left(\frac{n}{\delta}\right)\right)$ , then with probability at least  $1 - \delta$ ,  $\| \mathbf{H}(0) - \mathbf{H}^\infty \|_2 \leq \frac{\lambda_0}{4}$  and  $\frac{3}{4}\lambda_0 \leq \lambda_{\min}(\mathbf{H}(0)) \leq \frac{5}{4}\lambda_0$ .
Definition 4. We denote the empirical indicator matrix as  $\mathbf{W}(0)$  with entry  $[\mathbf{W}(0)]_{ij}$  such that
$$
[ \mathbf {W} (0) ] _ {i j} := \frac {1}{m} \sum_ {r = 1} ^ {m} \mathbb {I} \left\{\mathbf {w} _ {r} ^ {T} (0) \mathbf {x} _ {i} \geq 0, \mathbf {w} _ {r} ^ {T} (0) \mathbf {x} _ {j} \geq 0 \right\}
$$
and indicator matrix as  $\bar{\mathbf{W}}$  with entry  $[\bar{\mathbf{W}}]_{ij}$
$$
[ \bar {\mathbf {W}} ] _ {i j} := \mathbb {E} _ {\mathbf {w} \sim N (\mathbf {0}, \mathbf {I})} [ \mathbb {I} \{\mathbf {w} ^ {T} \mathbf {x} _ {i} \geq 0, \mathbf {w} ^ {T} \mathbf {x} _ {j} \geq 0 \} ]
$$
The indicator matrix  $\bar{\mathbf{W}}$  has the following property.
Proposition 1. If  $\mathbf{x}_i \nVdash \mathbf{x}_j$ , then  $[\bar{\mathbf{W}}]_{ij} > 0$ .
Proof. If Proposition 1 does not hold, then  $\exists i_0,j_0\in [n]\times [n]$ , satisfying
$$
\begin{array}{l} 0 = [ \bar {\mathbf {W}} ] _ {i _ {0} j _ {0}} \\ = \mathbb {E} _ {\mathbf {w} \sim N (\mathbf {0}, \mathbf {I})} [ \mathbb {I} \left\{\mathbf {w} ^ {T} \mathbf {x} _ {i _ {0}} \geq 0, \mathbf {w} ^ {T} \mathbf {x} _ {j _ {0}} \geq 0 \right\} ] \\ = P _ {\mathbf {w} \sim N (\mathbf {0}, \mathbf {I})} \left(\mathbf {w} ^ {T} \mathbf {x} _ {i _ {0}} \geq 0, \mathbf {w} ^ {T} \mathbf {x} _ {j _ {0}} \geq 0\right) \\ \end{array}
$$
Thus  $\mathbf{x}_{i_0} = -\mathbf{x}_{j_0}$ , which contradicts the hypothesis  $\mathbf{x}_{i_0} \nVdash \mathbf{x}_{j_0}$ .
In real-world datasets, the possibility of two different data being parallel is slight. Thus, proposition 1 holds in general. We further introduce Weyl inequality as follows:
Lemma 2 (Weyl inequality [39]). Let  $\mathbf{A}$ ,  $\mathbf{B} \in \mathbb{R}^{n \times n}$  be Hermitian matrices, and let the eigenvalues of  $\mathbf{A}$ ,  $\mathbf{B}$ , and  $\mathbf{A} + \mathbf{B}$  be  $\{\lambda_i(\mathbf{A})\}_{i=1}^n$ ,  $\{\lambda_i(\mathbf{B})\}_{i=1}^n$  and  $\{\lambda_i(\mathbf{A} + \mathbf{B})\}_{i=1}^n$ , respectively. The eigenvalues of each matrix are arranged in ascending order. Then we have
$$
\lambda_ {i} (\mathbf {A} + \mathbf {B}) \leq \lambda_ {i + j} (\mathbf {A}) + \lambda_ {n - j} (\mathbf {B}), \quad j = 0, 1, \dots , n - i \tag {14}
$$
for each  $i = 1,\dots ,n$  , with equality for some pair  $i,j$  if and only if there is a nonzero vector  $\mathbf{x}$  such that  $\mathbf{A}\mathbf{x} = \lambda_{i + j}(\mathbf{A})\mathbf{x},\mathbf{B}\mathbf{x} = \lambda_{n - j}(\mathbf{B})\mathbf{x},$  and  $(\mathbf{A} + \mathbf{B})\mathbf{x} = \lambda_i(\mathbf{A} + \mathbf{B})\mathbf{x}$  . Also,
$$
\lambda_ {i - j + 1} (\mathbf {A}) + \lambda_ {j} (\mathbf {B}) \leq \lambda_ {i} (\mathbf {A} + \mathbf {B}), \quad j = 1, \dots , i \tag {15}
$$
for each  $i = 1,\dots ,n$  , with equality for some pair  $i,j$  if and only if there is a nonzero vector  $\mathbf{x}$  such that  $\mathbf{A}\mathbf{x} = \lambda_{i - j + 1}(\mathbf{A})\mathbf{x},\mathbf{B}\mathbf{x} = \lambda_{j}(\mathbf{B})\mathbf{x},$  and  $(\mathbf{A} + \mathbf{B})\mathbf{x} = \lambda_{i}(\mathbf{A} + \mathbf{B})\mathbf{x}$  . If  $\mathbf{A}$  and  $\mathbf{B}$  have no common eigenvector, then inequality (14) and (15) are strict inequality.
Now, we provide the full proof of Theorem 2.