SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison with state-of-the-art NAS methods on CIFAR-10, and the transferred results on CIFAR-100 are also reported.  $\diamond$ : reported by Chen et al. [2019]. *: produced by rerunning the published code. ‡: transfer architectures discovered on CIFAR-10 to CIFAR-100.
NAS method	Test Err(%)	Params (M)	Search Cost (GPU-days)	Search Method
CIFAR-10	CIFAR-100
DenseNet-BC Huang et al. [2016]	3.46	-	25.6	-	Manual
ResNet + CutOutHe et al. [2015]	4.61	17.8	1.7	-	Manual
NASNet-A + CutOut Zoph et al. [2017]	2.65	-	3.3	1800	RL
AmoebaNet-A Real et al. [2018]	3.34 ± 0.06	-	3.2	3150	Evolution
AmoebaNet-B Real et al. [2018]	2.55 ± 0.05	-	2.8	3150	Evolution
PNAS Liu et al. [2017]	3.41 ± 0.09	-	3.2	225	MDL
ENAS + CutOut Pham et al. [2018]	2.89	-	4.6	0.5	RL
DARTS(1st) + CutOutLiu et al. [2018]	3.00 ± 0.14	17.76°	3.3	0.4	Gradient
DARTS(2nd) + CutOutLiu et al. [2018]	2.85 ± 0.08*	17.54°	3.4	0.4	Gradient
P-DARTS Chen et al. [2019]	2.50	16.55	3.4	0.3	Gradient
R-DARTS(L2) Zela et al. [2019]	2.95 ± 0.21	-	-	1.6	Gradient
FairDARTS Chu et al. [2019a]	2.54	-	2.8	0.4	Gradient
DARTS- Chu et al. [2020]	2.59 ± 0.08	-	3.5 ± 0.13	0.4	Gradient
SGAS(1st) Li et al. [2020]	2.39	-	3.8	0.25	Gradient
DARTS+PT Wang et al. [2021]	2.48(2.61 ± 0.08)	19.05‡	3.0	0.8	Gradient
DrNAS Chen et al. [2021b]	2.46 ± 0.03	-	4.1	0.6	Gradient
β-DARTS Ye et al. [2022]	2.53 ± 0.08	16.24 ± 0.22	3.75 ± 0.15	0.4	Gradient
GM+DARTS(1st) Hu et al. [2022]	2.35	16.45	3.7	1.1	Gradient
OLES	2.30(2.41 ± 0.11)	16.30(16.35 ± 0.05)	3.4	0.4	Gradient
Table 2: Comparison with NAS methods on CIFAR-100.  $\diamond$ : reported by Dong and Yang [2019].  $\dagger$ : reported by Zela et al. [2019]. *: produced by rerunning the published code.
NAS method	Test Err (%)	Params (M)	Search Cost (GPU-days)	Search Method
ResNet + CutOutHe et al. [2015]	22.10°	1.7	-	Manual
PNAS Liu et al. [2017]	19.53°	3.2	150	MDL
ENAS + CutOut Pham et al. [2018]	19.43°	4.6	0.45	RL
DARTS Liu et al. [2018]	20.58 ± 0.44†	3.4	0.4	Gradient
GDAS Dong and Yang [2019]	18.38	3.4	0.2	Gradient
P-DARTS Chen et al. [2019]	17.46*	3.6	0.3	Gradient
R-DARTS(L2)Zela et al. [2019]	18.24*	-	1.6	Gradient
DARTS- Chu et al. [2020]	17.51 ± 0.25	3.3	0.4	Gradient
PR-DARTS Zhou et al. [2020]	20.10*	3.6	0.17	Gradient
DARTS+PT Wang et al. [2021]	18.78*	3.4	0.8	Gradient
β-DARTSYe et al. [2022]	17.33*	3.83 ± 0.08	0.4	Gradient
GM+DARTSHu et al. [2022]	17.42*	3.6	1.3	Gradient
OLES	17.30	3.4	0.4	Gradient
architecture from scratch for 250 epochs, and the results are presented in Table 3. Our method achieves  $24.5\%$  top-1 test error. Compared to other NAS methods, the architectures derived from OLES still exhibit competitive performance. Notably, we do not introduce any additional training techniques such as SE modules and Swish, but simply follow the original DARTS setting to transfer the architectures. Therefore, the performance of OLES on ImageNet can be further enhanced by incorporating these training tricks.
4.3 Performance on NAS-Bench-201
NAS-Bench-201 Dong and Yang [2020] provides a unified benchmark for analyzing various up-to-date NAS algorithms. It contains 4 internal nodes with 5 operations (i.e., Zero, Skip Connection,  $1 \times 1$  Conv,  $3 \times 3$  Conv,  $3 \times 3$  AvgPol). NAS-Bench-201 offers a similar cell-based search space that comprised a total of 15625 unique architectures. The architectures are trained on three datasets (i.e., CIFAR-10, CIFAR-100, and ImageNet16-120). The comparison results are shown in Table 4, and the threshold  $\xi$  is set to 0.7. DARTS performs extremely poorly on NAS-Bench-201. We suppose that it is due to the fact that the search space of NAS-Bench-201 is extremely compact, contains few parametric operations, and the number of operation parameters varies widely, which makes it difficult for DARTS to adapt to NAS-Bench-201. OLES significantly outperforms DARTS on all datasets and is competitive among other NAS methods. It demonstrates the generality of OLES, which can adapt to different search spaces and datasets.
Table 3: Comparison with NAS methods on ImageNet.  $\diamond$ : reported by Chu et al. [2019a]. Other results are from their original papers. DARTS- and NoisyDARTS-A are directly searched on ImageNet and use SE modules and Swish to improve performance. Other methods in the bottom block are searched on CIFAR-10 and then transferred to ImageNet.
NAS method	Test Err (%)	Params (M)	Search Cost (GPU-days)	Search Method
Top-1	Top-5
Inception-v1 Szegedy et al. [2015]	30.2	10.1	6.6	-	Manual
MobileNet-V2 Sandler et al. [2018]	28.0	-	3.4	-	Manual
ShuffleNet Zhang et al. [2017]	26.3	-	~5	-	Manual
NASNet-A Zoph et al. [2017]	26.0	8.4	5.3	1800	RL
AmoebaNet-A Real et al. [2018]	25.5	8.0	5.1	3150	Evolution
AmoebaNet-B Real et al. [2018]	26.0	8.5	5.3	3150	Evolution
PNAS Liu et al. [2017]	25.8	8.1	5.1	225	SMBO
MnasNet-92 Tan et al. [2018]	25.2°	8.0°	4.4	1667	RL
DARTS(2nd) + CutOutLiu et al. [2018]	26.9	8.7	4.7	0.4	Gradient
SNAS Xie et al. [2018]	27.3	9.2	4.3	1.5	Gradient
GDAS Dong and Yang [2019]	25.0	8.5	5.3	0.2	Gradient
P-DARTS Chen et al. [2019]	24.4	7.4	5.1	0.3	Gradient
PC-DARTS Xu et al. [2019]	25.1	7.8	5.3	3.8	Gradient
FairDARTS-B Chu et al. [2019a]	24.9	7.5	4.8	0.4	Gradient
NoisyDARTS-A Chu and Zhang [2020]	22.1	6.0	5.5	12	Gradient
DARTS- Chu et al. [2020]	23.8	6.1	5.5	4.5	Gradient
DARTS+PT Wang et al. [2021]	25.5	8.0	4.6	0.8	Gradient
DrNAS Chen et al. [2021b]	23.7	7.1	5.7	4.6	Gradient
β-DARTSYe et al. [2022]	23.9	7.0	5.5	0.4	Gradient
GM + DARTS(2nd) Hu et al. [2022]	24.5	7.3	5.1	2.7	Gradient
OLES	24.5	7.4	4.7	0.4	Gradient
Table 4: Results on NAS-Bench201.
NAS method	CIFAR-10	CIFAR-100	ImageNet16-120
valid	test	valid	test	valid	test
DARTS(2nd) Liu et al. [2018]	39.77 ± 0.00	54.30 ± 0.00	15.03 ± 0.00	15.61 ± 0.00	16.43 ± 0.00	16.32 ± 0.00
SNAS Xie et al. [2018]	90.10 ± 1.04	92.77 ± 0.83	69.69 ± 2.39	69.34 ± 1.98	42.84 ± 1.79	43.16 ± 2.64
DSNAS Hu et al. [2020]	89.96 ± 0.29	93.08 ± 0.13	30.87 ± 16.40	31.01 ± 16.38	40.61 ± 0.09	41.07 ± 0.09
PC-DARTS Xu et al. [2019]	89.96 ± 0.15	93.41 ± 0.30	67.12 ± 0.39	67.48 ± 0.89	40.83 ± 0.08	41.31 ± 0.22
DARTS- Chu et al. [2020]	91.03 ± 0.44	93.48 ± 0.40	71.36 ± 1.51	71.53 ± 1.51	44.87 ± 1.46	45.12 ± 0.82
iDARTS Zhang et al. [2021a]	89.86 ± 0.60	93.58 ± 0.32	70.57 ± 0.24	70.83 ± 0.48	40.38 ± 0.59	40.89 ± 0.68
RLNAS(random label)Zhang et al. [2021b]	89.94 ± 0.00	93.35 ± 0.00	70.98 ± 0.00	70.71 ± 0.00	43.86 ± 0.00	43.70 ± 0.00
DrNASChen et al. [2021b]	91.55 ± 0.00	94.36 ± 0.00	73.49 ± 0.00	73.51 ± 0.00	46.37 ± 0.00	46.34 ± 0.00
β-DARTS Ye et al. [2022]	91.55 ± 0.00	94.36 ± 0.00	73.49 ± 0.00	73.51 ± 0.00	46.37 ± 0.00	46.34 ± 0.00
GM + DARTSHu et al. [2022]	91.03 ± 0.24	93.72 ± 0.12	71.61 ± 0.62	71.83 ± 0.97	42.19 ± 0.00	42.60 ± 0.00
OLES	90.88 ± 0.10	93.70 ± 0.15	70.56 ± 0.28	70.40 ± 0.22	44.17 ± 0.49	43.97 ± 0.38
4.4 Performance on MobileNet-like Search Space
As described in FairDARTS Chu et al. [2019a], the domination of skip connections also exists in the MobileNet-like search space. The supernet is built on MobileNetV2 Sandler et al. [2018] and comprises 21 choice blocks, each with 7 candidate operations. As the MobileNet search space is not naturally designed for DARTS, we make minor modifications to the MobileNet search space as FairDARTS and perform NAS on the ImageNet dataset. The threshold
Table 5: Results on ImageNet using the MobileNet-like search space.
NAS method	Top-1	Top-5	Params(M)
PloxylessNAS Cai et al. [2018]	24.9	7.5	7.1
FBNet-C Wu et al. [2018]	25.1	7.9	4.4
FairNAS-A Chu et al. [2019b]	24.7	7.6	4.6
FairDARTS-D Chu et al. [2019a]	24.4	7.4	4.3
RLNAS Zhang et al. [2021b]	24.4	7.4	5.3
GM+ProxylessNAS Hu et al. [2022]	23.4	7.0	4.9
OLES	24.7	7.6	4.7
$\xi$  is set to 0.4. The results are summarized in Table 5, and the corresponding searched architecture is visualized in Appendix D.1. We can observe that OLES can also achieve highly competitive performance in the MobileNet search space.
5 In-Depth Analysis
5.1 Different Gradient Direction Similarity Measures
Table 6: Test accuracy(%) with different gradient direction similarity measures on NAS-Bench-201.
Dataset	Cosine Similarity	L2 Distance	Direction Distance	Hard Threshold
CIFAR-10	93.77	92.35	90.59	93.65
CIFAR-100	70.11	69.39	67.40	70.19
ImageNet	44.23	42.38	38.28	43.86
As described in Section 3.2, we use cosine similarity to measure the difference in the operation's gradient directions between the training and validation data. To examine the effect of different gradient direction similarity measures for computing the gradient matching score, we compare cosine similarity with three other measures:  $L_{2}$  distance, direction distance, and hard threshold. The mathematical definitions of the other three measures can be found in Appendix B. The direction distance directly computes the fraction of elements that have different directions in two gradients. And the hard threshold does not use the averaged value of  $M$  iterations to compute the GM score but instead directly checks whether the direction distance can consistently be greater than a threshold in consecutive  $M$  iterations. As shown in Table 6, cosine similarity performs better than the other three methods. Therefore, we adopt cosine similarity as the gradient direction similarity measure of OLES.
5.2 Gradient Matching Score
(a) Operations' cosine similarity and test accuracy on CIFAR-10.
(b) Operations' cosine similarity and test accuracy on CIFAR-100.
(c) Operations' cosine similarity and test accuracy on CIFAR-10 (NAS-Bench-201).
Figure 3: Trajectories of the cosine similarity of parametric operations and test accuracy during the training of DARTS.
In Figure 3, we present the trajectories of consine similarity between gradients on training data and validation data during the DARTS training. With the training process of DARTS, the cosine similarities of parametric operations gradually decrease, which indicates that the cosine similarity can reflect the overfitting of operations. And different parametric operations show similar trajectories. Consequently, we set an unified GM score threshold for all parametric operations. Moreover, when the number of training iterations continuously increases (>50), DARTS exhibits a noticeable decline in test performance. Figure 3c further corroborates this finding, as the problem of performance collapse due to the skip connection domination issue more easily appears in NAS-Bench-201. As a result, it manifests the test performance declines at an earlier stage.
5.3 Training With Longer Epochs
As mentioned in Bi et al. [2019] and Chu et al. [2020], training DARTS with longer epochs enables a better convergence of the supernet and architecture parameters. However, as shown in Figure 4a and 4b, training standard DARTS with longer epochs exhibit serious performance collapse with the domination of skip connections. The skip connection rapidly dominates after 50 epochs on CIFAR-10 and takes longer epochs on CIFAR-100. And we also find that other methods (e.g., FairDARTS, and DARTS-PT) fail with longer training epochs as well. In contrast, as shown in Figure 4c and 4d, OLES survives even after 200 epochs. The number of skip connections in OLES stays stable after a specific number of epochs since most parametric operations are frozen. It demonstrates that OLES is robust and can actually address the domination of skip connections. More experimental results and searched architectures with longer training epochs can be found in Appendix C.2 and Appendix D.2.
(a) DARTS on CIFAR-10.
(b) DARTS on CIFAR-100.
Figure 4: Trajectories of the numbers of skip connections in the normal cell and the reduction cell on CIFAR-10 and CIFAR-100.
(c) OLES on CIFAR-10.
(d) OLES on CIFAR-100.
5.4 Comparison of Kendall Rank Correlation Coefficients
(a) DARTS (2nd Order)
Figure 5: Comparison of search-retraining Kendall rank correlation coefficients  $\tau$ . The results are derived from 10 randomly selected architectures from a single run of DARTS and OLES.
(b) OLES (ours)
Table 7: Kendall-  $\tau$  coefficients of architecture ranks based on searching and retraining metrics. \*: sample 4 excellent architectures in 4 rounds. Other methods randomly select 10 architectures from a single run.
RandomNAS*	DARTS	GDAS*	SGAS	OLES
0.0909	0.13	-0.1818	0.42	0.41
Figure 5 indicates that the early stopping mechanism does not hurt the ranking performance of DARTS. Additionally, we compare the Kendall rank correlation coefficients with other NAS methods, including RandomNAS, DARTS, GDAS, and SGAS, as shown in Table 7.
Notably, the Kendall coefficient of OLES aligns closely with that of SGAS Li et al. [2020], which aims to alleviate the effect of the degenerate search-retraining correlation problem. The results demonstrate that by mitigating operation parameter overfitting, differentiable NAS is able to focus on the potential of architectures themselves, thus enhancing the correlation between search metrics and the architectures discovered.
This paper attempts to find the fundamental reason for the domination of skip connections in DARTS from the new perspective of overfitting of operations in the supernet. We verified our conjecture through motivational experiments and proposed the operation-level early stopping (OLES) method to solve the domination of skip connections. OLES monitors the gradient direction of each operation and determines whether it is overfitting based on the difference between the gradient directions on the training data and the validation data. OLES can solve the domination of skip connections with negligible computational overheads without imposing any limitations. Moreover, OLES achieves state-of-the-art results of  $2.30\%$  test error on CIFAR-10 and  $17.30\%$  test error on CIFAR-100, meanwhile reporting superior performance when transferred to ImageNet and on search spaces of NAS-Bench-201 and MobileNet as well.
This work was supported by the National Natural Science Foundation of China (#62102177), the Natural Science Foundation of Jiangsu Province (#BK20210181), the Key R&D Program of Jiangsu Province (#BE2021729), and the Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.
Chakkrit Termrittthikun, Yeshi Jamtsho, Jirarat Ieamsaard, Paisarn Muneesawang, and Ivan Lee. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence, 104:104397, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search, 2018. URL https://arxiv.org/abs/1802.01548.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018. URL https://arxiv.org/abs/1802.03268.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 82–92, 2019.
Guanghui Zhu, Wenjie Wang, Zhuoer Xu, Feng Cheng, Mengchuan Qiu, Chunfeng Yuan, and Yihua Huang. Psp: Progressive space pruning for efficient graph neural architecture search. In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 2168-2181, 2022a. doi: 10.1109/ICDE53745.2022.00208.
Guanghui Zhu, Zhennan Zhu, Wenjie Wang, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. Autoac: Towards automated attribute completion for heterogeneous graph neural network. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2808-2821, 2023. doi: 10.1109/ICDE55515.2023.00215.
Jingfan Chen, Guanghui Zhu, Haojun Hou, Chunfeng Yuan, and Yihua Huang. Autogsr: Neural architecture search for graph-based session recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1694-1704, 2022.
Guanghui Zhu, Feng Cheng, Defu Lian, Chunfeng Yuan, and Yihua Huang. Nas-ctr: Efficient neural architecture search for click-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 332–342, 2022b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2018. URL https://arxiv.org/abs/1806.09055.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pages 367-377. PMLR, 2020.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudi Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv:1902.08142, 2019.
Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. Advances in neural information processing systems, 32, 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild, 2019. URL https://arxiv.org/abs/1912.10952.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. CoRR, abs/1907.01845, 2019b. URL http://arxiv.org/abs/1907.01845.
Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing darts with amended gradient estimation on architectural parameters, 2019. URL https://arxiv.org/abs/1910.11831.
A The Selection of Parametric-Free Operations on Various Search Spaces
Zela et al. [2019] designed four variants of the DARTS' search space (S1-S4) and pointed out skip connections tend to dominate compared with two other parametric-free operations, i.e., zero and noise. To investigate whether the robustness issue in DARTS arises from the domination of skip connections rather than other parametric-free operations, we extend the experimental settings in Zela et al. [2019] to seven search spaces, including four types of parametric-free operations (avg_pool_3×3, skip_connect, noise, zero). These search spaces only differ from DARTS' original space in the number and types of operations on each edge. We follow the same experimental setup in Liu et al. [2018] and execute DARTS on each search space. Figures 6-8 and 13-15 display the results on S1-S3. We observe that skip connections tend to dominate more severely than other parametric-free operations. By designing different combinations of parametric-free operations in S4-S7 search spaces, Figures 9-12 and 16-19 show that skip connections also dominate under fair competition with other parametric-free operations when the searched cells contain more than one type of parametric-free operations. The corresponding performance for each search space is shown in Table 8. As a result, the domination of skip connections is the primary cause of the shallow architecture of DARTS with poor generalization and robustness.
- S1 consists of two operations: skip_connect and sep_conv_3×3.
- S2 consists of two operations: avg_pool_3×3 and sep_conv_3×3.
- S3 consists of two operations: noise and sep_conv_3×3. The noise operation replaces each input value with a random Gaussian noise for  $N(0,1)$ . Differentiable NAS algorithms generally avoid this operation because it hurts the performance of discretized architectures.
- S4 consists of three operations: skip_connect, noise, and sep_conv_3 × 3.
- S5 consists of three operations: skip_connect, avg_pool_3×3, and sep_conv_3×3.
- S6 consists of three operations: skip_connect, zero, and sep_conv_3×3. The zero operation replaces each input value with zeros.
- S7 consists of five operations: skip_connect, avg_pool_3×3, noise, zero, and sep_conv_3×3.
Table 8: Performance of DARTS on search spaces S1-S7.
Search Space	CIFAR-10	CIFAR-100
S1	3.28	26.05
S2	2.55	25.40
S3	3.69	24.70
S4	3.05	21.35
S5	3.03	23.77
S6	2.84	27.90
S7	2.70	22.17
(a) Normal Cell
(b) Reduction Cell
Figure 6: Normal and reduction cells found by DARTS on CIFAR-10 in search space S1.
(b) Reduction Cell
Figure 7: Normal and reduction cells found by DARTS on CIFAR-10 in search space S2.
(b) Reduction Cell
Figure 8: Normal and reduction cells found by DARTS on CIFAR-10 in search space S3.
(a) Normal Cell
Figure 9: Normal and reduction cells found by DARTS on CIFAR-10 in search space S4.
(b) Reduction Cell
(b) Reduction Cell
Figure 10: Normal and reduction cells found by DARTS on CIFAR-10 in search space S5.
(b) Reduction Cell
Figure 11: Normal and reduction cells found by DARTS on CIFAR-10 in search space S6.
(b) Reduction Cell
Figure 12: Normal and reduction cells found by DARTS on CIFAR-10 in search space S7.
(a) Normal Cell
Figure 13: Normal and reduction cells found by DARTS on CIFAR-100 in search space S1.
(a) Normal Cell
Figure 14: Normal and reduction cells found by DARTS on CIFAR-100 in search space S2.
(b) Reduction Cell
Figure 15: Normal and reduction cells found by DARTS on CIFAR-100 in search space S3.
(b) Reduction Cell
Figure 16: Normal and reduction cells found by DARTS on CIFAR-100 in search space S4.
(a) Normal Cell
Figure 17: Normal and reduction cells found by DARTS on CIFAR-100 in search space S5.
(a) Normal Cell
Figure 18: Normal and reduction cells found by DARTS on CIFAR-100 in search space S6.
(b) Reduction Cell
Figure 19: Normal and reduction cells found by DARTS on CIFAR-100 in search space S7.
B Gradient Distance Similarity Measures
In Section 5.1, four gradient distance measures are present. Next, we provide a comprehensive mathematical definition for each measure. Let  $g_{train}^{o}$  and  $g_{val}^{o}$  denote the gradients of the operation parameters  $P_{o}$  on a training batch and a validation batch, respectively.
The GM score with  $L_{2}$  distance can be formulated as follows:
$$
G M _ {L _ {2}} \left(g _ {\text {t r a i n}} ^ {o}, g _ {\text {v a l}} ^ {o}\right) = \frac {\sum_ {m = 1} ^ {M} \left\| g _ {\text {t r a i n}} ^ {o} - g _ {\text {v a l}} ^ {o} \right\| _ {2}}{M}. \tag {2}
$$
And the GM score with direction distance can be formulated as follows:
$$
G M _ {d i r e c t i o n} \left(g _ {t r a i n} ^ {o}, g _ {v a l} ^ {o}\right) = \frac {\sum_ {m = 1} ^ {M} \frac {\| g _ {t r a i n} ^ {o} \odot g _ {v a l} ^ {o} <   0 \| _ {1}}{\| P _ {o} \| _ {0}}}{M}, \tag {3}
$$
where  $\odot$  denotes the element-wise product, and  $\| g_{train}^{o}\odot g_{val}^{o} < 0\|_{1}$  represents the number of elements whose directions are different.
However, the hard threshold method does not use the averaged gradient distances over  $M$  steps to calculate the GM score. Instead, it directly checks whether the direction distance can consistently exceed a threshold over the consecutive  $M$  steps. Let  $\xi$  denote the threshold. The hard threshold method checks whether  $\frac{\|g_{train}^o\odot g_{val}^o < 0\|_1}{\|P_o\|_0} >\xi$  can still be satisfied for  $M$  consecutive steps.
C.1 Quantitative Analysis on Negative Correlation between Validation Loss and Architecture Parameter
The results of our motivation experiments, as depicted in Figure 1, indicate a negative correlation between validation loss and architecture parameter. To further provide a more rigid analysis, we calculate the Kendall-correlations between validation loss and architecture parameter for each operation and present the results in the Table 9. It can be observed that there is a clear negative correlation between validation losses and architecture parameters, particularly for parametric operations.
Table 9: Kendall-τ correlations for architecutre parameters and validation losses of different operations in Figure 1a.
Skip-Connections	Sep-Conv-3x3	Sep-Conv-5x5	Dil-Conv-3x
-0.36	-0.59	-0.73	-0.72
C.2 Results with Longer Training Epoches
Figure 20: Test accuracy of searched architectures on CIFAR-10 with longer search epochs.
(b) Reduction Cell
Figure 21: The number of skip connections in searched architectures on CIFAR-10. In Figure 21b, there are some overlaps among the curves of DARTS and OLES.