SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Despite previous attempts to investigate the origins of the skip connection domination issue and its corresponding solutions, it remains an open problem. These methods have put forth their hypotheses but have not fully resolved the issue. To further exemplify the efficacy of addressing the skip connection domination issue, we increase the number of iterations of supernet training. As depicted in Figures 20 and 21, DARTS-PT Wang et al. [2021] fails to address the skip connection domination issue when extending to very long training epochs, whereas OLES remains stable.
C.3 Hyperparameter Analysis
In OLES, the gradient matching score is dynamically computed by averaging over every 20 iterations throughout the entire training process. The threshold  $\xi$  is determined based on the GM scores within the initial 20 iterations. We conduct experiments to assess various iterations along with their corresponding thresholds across different search spaces to explain the rationale behind selecting 20 iterations. The results, as shown in Table 10, demonstrate that computing the GM score using an average across every 20 iterations yields the highest accuracy.
C.4 Results on DARTS S1-S4
DARTS S1-S4 are simplified variations of the DARTS search space proposed by R-DARTS Zela et al. [2019] to investigate DARTS' failure mode. In Appendix A, we have further expanded these search spaces and presented DARTS' failure mode (both in terms of performance and the searched architectures) in these extended search spaces. In Table 11, we additionally provide a performance comparison between OLES and other methods on DARTS S1-S4. The results demonstrate that OLES also consistently outperforms DARTS, as well as R-DARTS Zela et al. [2019] and DARTS-PT Wang et al. [2021] which are also dedicated to addressing the skip connection domination issue.
Table 10: Performance on various search spaces using different numbers of iterations to calculate the GM scores.
Iterations	DARTS Search Space	NAS-BENCH-201 Search Space
CIFAR-10	CIFAR-100	CIFAR-10	CIFAR-100	ImageNet16-120
10	97.33	80.83	80.57	47.93	26.29
20	97.70	82.70	93.89	70.75	44.38
30	97.05	82.39	91.93	66.97	38.87
40	97.19	81.32	89.76	64.69	32.73
Table 11: Performance comparison on the DARTS S1-S4 search spaces.
Datasets	Spaces	DARTS	R-DARTS	DARTS-PT	OLES
CIFAR-10	S1	3.28	3.31	2.79	2.76
S2	2.55	2.44	2.47	2.43
S3	3.69	3.56	2.64	2.57
S4	3.05	3.05	2.92	2.73
CIFAR-100	S1	26.05	22.24	23.16	22.10
S2	25.40	23.34	22.10	22.18
S3	24.70	21.94	20.80	20.70
S4	21.35	20.70	19.98	19.18
C.5 Comparison with Train-Free NAS Methods
Table 12 compares OLES with two recently proposed train-free NAS methods, TE-NAS Zhao et al. [2021] and ZiCo Li et al. [2023]. It is evident that OLES still outperforms train-free methods in most cases. While train-free methods exhibit slightly superior performance on NAS-Bench-201 due to its compact search space, in larger search spaces such as the DARTS search space, OLES performs better than TE-NAS and ZiCo. Train-free NAS is a highly promising research direction, as it not only greatly enhances NAS efficiency but also promotes a deeper understanding of neural networks.
Table 12: Comparison with TE-NAS and ZiCo. The test errors are reported.
NAS-Bench-201	DARTS Search Space
CIFAR-10	CIFAR-100	ImageNet-16-120	CIFAR-10	ImageNet
top-1	top-5
TE-NAS	6.1±0.47	28.76±0.56	57.62±0.46	2.63±0.064	26.2	8.3
ZiCo	6.0±0.4	28.9±0.3	58.20±0.3	2.45±0.11	-	-
OLES	6.3±0.15	29.60±0.22	56.03 ± 0.38	2.41 ± 0.11	24.5	7.4
D Searched Architectures
D.1 Architecture on MobileNet-like Search Space
Following the experimental settings in Chu et al. [2019a], we modify the MobileNetV2 Sandler et al. [2018] search space and perform OLES on ImageNet in this search space. The searched architecture is illustrated in Figure 22.
Figure 22: The architecture searched on the MobileNet-like search space.
D.2 Training With Longer Epochs
We train OLES and several other differentiable architecture search methods with 200 epochs to verify whether they are robust enough in this case. According to our conjecture, as the number of training epochs increases, the operations in the supernet become more prone to overfitting, leading to a more serious issue of the domination of skip connections. In this section, we present some additional results of the searched architectures using different numbers of training epochs.
D.2.1 Results on OLES
As described in Section 5.3, OLES can survive with longer training epochs. Specifically, the number of skip connections in OLES keeps stable even when the number of training epochs reaches 150 or 200. The corresponding architecture genotypes on CIFAR-10 and CIFAR-100 are visualized in Figures 23-26.
(b) Reduction Cell
Figure 23: Normal and Reduction cells discovered by OLES on CIFAR-10 for 150 epochs in the standard DARTS' search space.
(b) Reduction Cell
Figure 24: Normal and Reduction cells discovered by OLES on CIFAR-10 for 200 epochs in the standard DARTS' search space.
(b) Reduction Cell
Figure 25: Normal and Reduction cells discovered by OLES on CIFAR-100 for 150 epochs in the standard DARTS' search space.
(b) Reduction Cell
Figure 26: Normal and Reduction cells discovered by OLES on CIFAR-100 for 200 epochs in the standard DARTS' search space.
D.2.2 Results on FairDARTS
FairDARTS Chu et al. [2019a] adopts the Sigmoid instead of the exclusive Softmax for architecture parameters and selects operations through a threshold value. However, our experiments show that no operation can achieve the pre-defined threshold value in the normal cell when FairDARTS is trained for 150 or 200 epochs. Following FairDARTS' settings, we train it for 50, 150, and 200 epochs. Table 13 shows the architecture genotypes, which can be visualized in Figure 27 and Figure 28.
Table 13: Architecture genotypes found by FairDARTS on CIFAR-10 for 50, 150 and 200 epochs, respectively.
Epochs	Architecture genotype
50	Genotype(normal=[('sep_conv_3x3', 2, 0), ('dil_conv_5x5', 2, 1), ('dil_conv_3x3', 3, 0), ('dil_conv_3x3', 3, 1), ('sep_conv_5x5', 4, 1)], normal_concat=range(2, 6), reduce=[('skip_connect', 2, 0), ('dil_conv_5x5', 2, 1), ('skip_connect', 3, 2), ('sep_conv_3x3', 3, 0), ('skip_connect', 4, 2), ('sep_conv_3x3', 4, 0), ('skip_connect', 5, 2), ('skip_connect', 5, 0)], reduce_concat=range(2, 6))
150	Genotype(normal=[ ], normal_concat=range(2, 6), reduce=[('max_pool_3x3', 2, 0), ('max_pool_3x3', 2, 1), ('skip_connect', 3, 2), ('max_pool_3x3', 3, 0), ('skip_connect', 4, 2), ('max_pool_3x3', 4, 0), ('skip_connect', 5, 3), ('skip_connect', 5, 2)], reduce_concat=range(2, 6))
200	Genotype(normal=[ ], normal_concat=range(2, 6), reduce=[('avg_pool_3x3', 2, 0), ('max_pool_3x3', 2, 1), ('skip_connect', 3, 2), ('max_pool_3x3', 3, 0), ('skip_connect', 4, 3), ('skip_connect', 4, 2), ('sep_conv_3x3', 5, 0), ('dil_conv_3x3', 5, 1)], reduce_concat=range(2, 6))
Figure 27: Normal and Reduction cells discovered by FairDARTS on CIFAR-10 for 50 epochs in the standard DARTS' search space.
(a) Epoch  $= 150$
Figure 28: Reduction cells discovered by FairDARTS on CIFAR-10 for 150 and 200 epochs in the standard DARTS' search space. Note that normal cells are empty when FairDARTS is trained for 150 and 200 epochs.
(b) Epoch=200
D.2.3 Results on DARTS+PT
DARTS+PT Wang et al. [2021] evaluates operation strength by measuring its contribution to the supernet's performance instead of the magnitude of the architecture parameter. Following DARTS+PT's settings, we train DARTS with the permutation-based architecture selection method for 50, 150, and 200 epochs, and the corresponding architecture genotypes are shown in Table 14. When extending to over 150 epochs, the resulting architectures are also full of skip connections. Furthermore, to avoid inaccurate evaluation of the operation strength during discretization, DARTS+PT needs to fine-tune the remaining supernet until it converges again. The architecture genotypes with tuned supernets are shown in Table 15, Figure 29, and Figure 30. It's worth noting that the test error of the final resulting architecture for 200 epochs on CIFAR-10 will drop to  $6.28\%$ .
Table 14: Architecture genotypes found by DARTS+PT on CIFAR-10 for 50, 150, and 200 epochs, respectively. (without supernet tuning)
Epochs	Architecture genotype
50	Genotype(normal=[('sep_conv_3x3', 0), ('dil_conv_3x3', 1), ('sep_conv_3x3', 0), ('dil_conv_3x3', 2), ('dil_conv_3x3', 3), ('skip_connect', 0), ('dil_conv_5x5', 4), ('dil_conv_5x5', 3]), normal_concat=range(2, 6), reduce=[('max_pool_3x3', 0), ('max_pool_3x3', 1), ('max_pool_3x3', 0), ('skip_connect', 2), ('skip_con-nect', 2), ('skip_connect', 3), ('skip_connect', 2), ('avg_pool_3x3', 0)], reduce_/_concat=range(2, 6))
150	Genotype(normal=[('dil_conv_3x3', 1), ('sep_conv_3x3', 0), ('skip_connect', 2), ('skip_connect', 0), ('skip_connect', 1), ('skip_connect', 2), ('skip_connect', 1), ('skip_connect', 2)], normal_concat=range(2, 6), reduce=[('max_pool_3x3', 1), ('max_pool_3x3', 0), ('skip_connect', 2), ('max_pool_3x3', 0), ('skip_connect', 2), ('skip_connect', 3), ('skip_connect', 3), ('skip_connect', 2)], reduce_con-cat=range(2, 6))
200	Genotype(normal=[('dil_conv_3x3', 1), ('sep_conv_3x3', 0), ('skip_connect', 2), ('skip_connect', 1), ('skip_connect', 1), ('skip_connect', 2), ('skip_connect', 1), ('skip_connect', 0)], normal_concat=range(2, 6), reduce=[('max_pool_3x3', 1), ('avg_pool_3x3', 0), ('skip_connect', 2), ('skip_connect', 3), ('skip_connect', 3), ('skip_connect', 2)], reduce_con-cat=range(2, 6))
Table 15: Architecture genotypes found by DARTS+PT on CIFAR-10 for 150 and 200 epochs, respectively. (with supernet tuning)
Epochs	Architecture genotype
150	Genotype(normal=[('sep_conv_3x3', 0), ('dil_conv_3x3', 1), ('skip_connect', 0), ('skip_connect', 2), ('skip_connect', 0), ('dil_conv_5x5', 3), ('skip_connect', 1), ('sep_conv_5x5', 4)], normal_concat=range(2, 6), reduce=[('max_pool_3x3', 0), ('max_pool_3x3', 1), ('avg_pool_3x3', 0), ('skip_connect', 2), ('dil_conv_5x5', 0), ('skip_connect', 3), ('dil_conv_5x5', 2), ('skip_connect', 4)], reduce_=concat=range(2, 6))
200	Genotype(normal=[('sep_conv_3x3', 0), ('dil_conv_3x3', 1), ('sep_conv_3x3', 0), ('skip_connect', 2), ('skip_connect', 0), ('dil_conv_5x5', 3), ('skip_connect', 1), ('dil_conv_5x5', 4]), normal_concat=range(2, 6), reduce=[('sep_conv_3x3', 0), ('max_pool_3x3', 1), ('max_pool_3x3', 0), ('skip_connect', 2), ('sep_conv_3x3', 1), ('skip_connect', 3), ('sep_conv_5x5', 0), ('skip_connect', 3)], reduce_=concat=range(2, 6))
(b) Reduction Cell
Figure 29: Normal and Reduction cells discovered by DARTS+PT on CIFAR-10 for 150 epochs in the standard DARTS' search space (with supernet tuning).
(b) Reduction Cell
Figure 30: Normal and Reduction cells discovered by DARTS+PT on CIFAR-10 for 200 epochs in the standard DARTS' search space (with supernet tuning).