Title: Operation-Level Early Stopping for Robustifying Differentiable NAS
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differentiable neural architecture search; Image classification; Failure of DARTS
Abstract: Differentiable NAS (DARTS) is a simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks.
% 
Nevertheless, DARTS still encounters several robustness issues, mainly the domination of skip connections.
% 
The resulting architectures are full of parametric-free operations, leading to performance collapse.
% 
Existing methods suggest that the skip connection has additional advantages in optimization compared to other parametric operations and propose to alleviate the domination of skip connections by eliminating these additional advantages.
% 
In this paper, we analyze this issue from a simple and straightforward perspective and propose that the domination of skip connections results from parametric operations overfitting the training data while architecture parameters are trained on the validation data, leading to undesired behaviors.
% 
Based on this observation, we propose the operation-level early stopping (OLES) method to overcome this issue and robustify DARTS without introducing any computation overhead.
% 
Extensive experimental results can verify our hypothesis and the effectiveness of OLES.

Operation-Level Early Stopping for Robustifying Differentiable NAS

Abstract

Differentiable NAS (DARTS) is a simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks. Nevertheless, DARTS still encounters several robustness issues, mainly the domination of skip connections. The resulting architectures are full of parametric-free operations, leading to performance collapse. Existing methods suggest that the skip connection has additional advantages in optimization compared to other parametric operations and propose to alleviate the domination of skip connections by eliminating these additional advantages. In this paper, we analyze this issue from a simple and straightforward perspective and propose that the domination of skip connections results from parametric operations overfitting the training data while architecture parameters are trained on the validation data, leading to undesired behaviors. Based on this observation, we propose the operation-level early stopping (OLES) method to overcome this issue and robustify DARTS without introducing any computation overhead. Extensive experimental results can verify our hypothesis and the effectiveness of OLES.

1 Introduction

Neural architecture search (NAS) aims to automatically design high-performance neural architectures Zoph and Le [2016] for given tasks in a data-driven manner. Deep learning models designed by NAS have achieved state-of-the-art performance in a variety of tasks, such as image classification Termritthikun et al. [2021], Real et al. [2018], Pham et al. [2018], semantic segmentation Liu et al. [2019], graph learning Zhu et al. [2022a, 2023], and recommender systems Chen et al. [2022], Zhu et al. [2022b]. Among various NAS methods, differentiable architecture search (DARTS) Liu et al. [2018] significantly improves the efficiency of NAS through parameter-sharing and continuous relaxation. DARTS converts discrete architecture selections into continuous architecture parameters α and optimizes them via gradient-based optimizers.

Despite its simplicity and efficiency, DARTS still has several issues that make it difficult to outperform other NAS methods, even a simple random search Li and Talwalkar [2020], Yu et al. [2019], in many cases. The first issue is the performance gap . Due to the discrepancies between discrete architectures and continuous architecture parameters, the derived model shows non-negligible performance degradation Liu et al. [2018]. Additionally, DARTS suffers from serious robustness problems Nayman et al. [2019], Chen et al. [2019], Liang et al. [2019], Chu et al. [2020], Zela et al. [2019], Wang et al.

[2021]. In particular, during the search phase of DARTS, as the number of training epochs increases, the searched architectures will fill with parametric-free operations Chu et al. [2019a], such as skip connections or even random noise, leading to performance collapse. These issues waste a lot of computational resources and seriously hinder the application of DARTS.

The domination of skip connections is the major robustness issue of DARTS Wang et al. [2021]. Many studies have investigated why the domination of skip connections happens and how to alleviate this issue. Most of them believe that skip connections exhibit unfair advantages in gradient-based optimization during the search phase Liang et al. [2019], Chu and Zhang [2020], Chu et al. [2019a] or stabilize the supernet training by constructing residual blocks Chu et al. [2020]. To address this problem, they propose eliminating these unfair advantages Chu et al. [2019a] or separating the stabilizing role Chu et al. [2020], mainly drawing inspiration from ResNet He et al. [2015]. However, the generic optimization in ResNet differs fundamentally from the bi-level optimization in DARTS. Therefore, observations and theories based on uni-level optimization are insufficient to support the claim that skip connections still have unfair advantages in DARTS. The work Wang et al. [2021] analyzes this problem from the perspective of architecture selection and proposes that the magnitude of the architecture parameter may not reflect the strength of the operation. Based on this assumption, a perturbation-based architecture selection method is proposed. However, this idea conflicts with the fundamental motivation of DARTS, which utilizes continuous relaxation to solve the combinatorial optimization problem approximately. Xie et al. [2021] suggests that overfitting in bi-level optimization is the primary reason for the catastrophic failure of DARTS and alayzes it from a unified theoretical framework, which is similar to our conjecture.

In contrast to the above sophisticated assumptions, in this paper, we take a simpler and more straightforward perspective to consider the reasons for the domination of skip connections. We observe that the robustness problem of DARTS is caused by overfitting. Specifically, the parameters of operations (e.g., convolution) in the supernet overfit the training data and gradually deviate from the validation data. Meanwhile, the architecture parameters are trained on the validation data, causing the advantages of parametric operations to weaken progressively and ultimately leading to the domination of parametric-free operations. It is evident that the supernet is always overparameterized. Additionally, the architecture parameters also affect the supernet training, with operations that exhibit advantages in the early stage of search being more prone to overfitting.

Based on the above observations, we further propose an operation-level early stopping (OLES) 1 method to address the domination of skip connections. The OLES method monitors each operation in the supernet to determine if it tends to overfit the training data and stops training the operation when overfitting occurs. Specifically, we utilize the gradient matching (GM) method Hu et al. [2022] to compare the gradient directions of operation parameters on the training data and the validation data. If the difference between them remains large in multiple consecutive iterations, the operation will stop training. This approach not only alleviates the domination of skip connections and but also sharpens the distribution of architecture parameters with more search iterations, thereby reducing the performance gap caused by discretization. Moreover, our method only needs to maintain the directions of operation gradients. Due to only minimal modifications to the original DARTS, the additional overhead is negligible.

In summary, we make the following contributions:

We analyze the robustness issue of DARTS from a new perspective, focusing on the overfitting of operations in the supernet.

We propose the operation-level early stopping method to address the domination of skip connections in DARTS with negligible computation overheads.

Extensive experiments demonstrate the effectiveness and efficiency of our method, which can achieve state-of-the-art (SOTA) performance on multiple commonly used image classification datasets.

2 Background and Related Work

2.1 Neural Architecture Search

Over the years, a variety of NAS methods have been proposed, including evolutionary-based Termritthikun et al. [2021], Real et al. [2018], Guo et al. [2020], reinforcement-learning-based Zoph and Le [2016], Pham et al. [2018], and gradient-based Liu et al. [2018], Xie et al. [2018], Cai et al. [2018] methods. They have achieved great success in many computer vision (CV) and natural language processing (NLP) So et al. [2019] tasks. As a pioneering work, NAS-RL Zoph and Le [2016] uses an RNN controller to generate architectures and employs the policy gradient algorithm to train the controller. However, this method suffers from low search efficiency. To improve the search efficiency, the cell-based micro search space Zoph et al. [2017] and weight-sharing mechanism Pham et al. [2018] have been proposed. DARTS Liu et al. [2018], which also takes advantage of weight-sharing, utilizes the gradient-based method to optimize architecture parameters and model weights alternatively. Recently, train-free NAS methods Chen et al. [2021a], Li et al. [2023] that utilize proxy metrics to predict the ranking or test performance of architectures without training have been proposed to further promote search efficiency.

2.2 Preliminaries of Differentiable Neural Architecture Search

We will now provide a detailed review of DARTS. The original DARTS Liu et al. [2018] relies on a cell-based micro search space, where each cell contains N nodes and E edges organized in a directed acyclic graph (DAG). Each node represents a feature map x ( i ) , and each edge is associated with an operation o ∈ O , where O denotes the set of candidate operations (e.g., skip connect, sep conv 3x3, etc.). During architecture search, instead of applying a single operation to a specific node, continuous relaxation is applied to relax the categorical choice of a specific operation to a mixture of candidate

parameters. DARTS then jointly optimizes architecture parameters α and model weights w with the following bi-level objective via alternative gradient descent:

Since DARTS follows the weight-sharing mechanism, we refer to the continuous relaxed network as the "supernet". At the end of the search phase, a discretization phase is performed by selecting operations with the largest α values to form the final architecture.

2.3 Robustifying DARTS

Despite its simplicity and efficiency, the undesired domination of skip connections is often observed in DARTS. Previous works have attempted to eliminate the domination of skip connections by diminishing their unfair advantages. DARTS+Liang et al. [2019] uses a fixed value to constrain the number of skip connections. Progressive DARTSChen et al. [2019] adds a Dropout after each skip connection operation. DARTS -Chu et al. [2020] introduces an auxiliary skip connection to separate the unfair advantage of the skip connection by analyzing it from the perspective of ResNet He et al. [2015]. Another strategy is to remove the exclusive competition of operation selection in DARTS. FairDARTSChu et al. [2019a] replaces the Softmax relaxation of DARTS with Sigmoid and offers an independent architecture parameter for each operation. NoisyDARTSChu and Zhang [2020] injects unbiased noise into the candidate operations to ensure that the good ones win robustly. However, DARTS+PT Wang et al. [2021] believes that the magnitude of the architecture parameter is not a good indicator of operation strength and proposes a perturbation-based architecture selection method.

Our work is also related to GM-NAS Hu et al. [2022], which falls in the field of Few-Shot NAS Zhao et al. [2021]. Although our method and GM-NAS both employ the GM score as a mathematical tool, they are entirely distinct lines of work. GM-NAS argues that due to coupled optimization between child architectures caused by weight-sharing, One-Shot supernet's performance estimation could be inaccurate, leading to degraded search results. As a result, GM-NAS proposes to reduce the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets.

0

5000

10000

15000

20000

25000

30000

35000

40000

Step

-

1

0

1

2

3

Value of Architecture Parameter

Skip-Connection

Sep-Conv-3x3

Sep-Conv-5x5

Dil-Conv-3x3

1

2

3

4

5

6

Validation Loss

Skip-Connection

Sep-Conv-3x3

Sep-Conv-5x5

Dil-Conv-3x3

(a) The architecture parameters and validation losses of different operations. The solid line represents the value of architecture parameter, while the dotted line represents the validation loss.

0

10000

20000

30000

40000

Step

-

1

0

1

2

3

Value of Architecture Parameter

Skip-Connection

None

Avg-Pool-3x3

GM-NAS utilizes GM scores to make splitting decisions, determining whether a module should be shared among child architectures. The GM scores in GM-NAS are computed based on the gradient information of different child architectures on shared parameters. In contrast, we aim to address the problem of skip connection donimation from a totally new perspective and employ the GM score as an indicator for early stopping, preventing operation parameter overfitting. The GM scores, in our approach, are calculated using the gradient information of parameters on training and validation data.

3 Motivation and Methodology

In this section, we argue that the domination of skip connections is due to the overfitting of operations in the supernet and demonstrate this hypothesis through a simple motivational experiment. Based on our observations, we propose an operation-level early stopping method to address this issue.

3.1 The Domination of Skip-Connection is Due to Overfitting

In DARTS, we observe that skip connections dominate due to the overfitting of operations in the supernet to the training data, causing them to increasingly deviate from the distribution of the validation data. However, the architecture parameters are trained on the validation data, thus operations are selected based on their validation performance. As the number of training iterations increases, the parametric operations overfit more seriously, and the parametric-free operations gradually dominate.

To verify the above conjecture, we present empirical evidence on the relationship between the overfitting of operations in the supernet and architecture parameters. We retain only architectural parameters on one edge in DARTS' search space and set all other architectural parameters to 0 to ensure that they will not be trained. In Figure 1, we record the architecture parameter and the validation loss with the corresponding output of each operation in this edge. From Figure 1a, we can observe that the validation losses of all parametric operations grow even faster as the training steps increase, and the architecture parameter gradually decreases, exhibiting an obvious negative correlation with the validation loss. Moreover, Figure 1bshows that the skip connection easily obtains higher architecture parameters, exhibiting advantages over other parametric-free operations. This is mainly due to the fact that the skip connection can always retain more information from the input.

Most existing works suggest that the domination of skip connections is due to their unfair advantage. To investigate this claim, we follow the settings of FairDARTS Chu et al. [2019a] and replace the Softmax relaxation in DARTS with the Sigmoid relaxation. The unfair advantage of skip connections will be eliminated by removing the exclusive competition among operations. We then rerun the experiment and present the results in Figure 2, which shows that even after removing the exclusive competition, the architecture parameters still exhibit a negative correlation with the validation losses, similar to that in Figure 1a. It illustrates that the unfair advantage of skip connections is not the fundamental reason for their domination. Instead, it is the effect of the overfitting of operations in the supernet on the training of architecture parameters.

3.2 Operation-Level Early Stopping

Based on the above observations, we propose an operation-level early stopping method to alleviate the domination of skip connections and enhance the robustness of DARTS. During the search phase, the architecture parameters have an effect on the supernet training, resulting in different operations with different architecture parameters being trained differently. Operations that exhibit advantages in the early stage will get more attention and are more prone to overfitting. Moreover, the operation is the basic unit of

0

5000

10000

15000

20000

25000

30000

35000

40000

Step

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Value of Architecture Parameter

Skip-Connection

Sep-Conv-3x3

Sep-Conv-5x5

Dil-Conv-3x3

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

Validation Loss

Skip-Connection

Sep-Conv-3x3

Sep-Conv-5x5

Dil-Conv-3x3

choice in DARTS, which selects operations to form the final architecture. Therefore, we perform early stopping for each operation. If early stopping can be performed for each operation during the supernet training, when the operation tends to overfit, the overfitting problem can have less impact on the training of architecture parameters, and the supernet can also be fully trained.

Overfitting occurs when a model is too complex and fits the training data distribution too closely, resulting in poor performance on the validating data. When a model is overfitting, the training loss continues to decrease while the validation loss increases, indicating that the optimization direction (i.e., the direction of gradients) of the model parameters on the training data will be inconsistent with that on the validation data. To identify overfitting, we propose using the gradient direction of the operation parameters. If the gradients on the training and validation data differ significantly in direction, we consider the operation to tend to overfit the training data.

Algorithm 1 DARTS with operation-level early stopping

Require: Supernet weights w ; architecture parameters α ; number of search iterations I ; number of consecutive iterations for GM M ; threshold of GM score for verifying overfitting ξ .

Ensure:

Searched architectures.

2: for each i ∈ [1 , I ] do

1: Construct a supernet A ;

3: Get a batch of training data B train ;

5: Record the training gradient of each operation g o train ;

4: Update weights w by ∇ w L train ( w,α,B train ) ;

6: Get a batch of validation data B val ;

8: Record the validation gradient of each operation g o val ;

7: Update parameters α by ∇ α L val ( w,α,B val ) ;

9: Compute the gradient matching score over consecutive M iterations based on Eq. 1.

11: end for

10: Stop the gradient of an operation o if GM ( g o train , g o val ) < ξ

12: Derive the final architecture based on learned α .

Concretely, for an operation o , let P o denote the parameters of o , and g o train and g o val denote the gradients of P o on a training batch and a validation batch, respectively. We calculate the cosine similarity between g o train and g o val to quantify the differences in the direction of the operations' gradients as follows:

where B train and B val denote batches of training data and validation data, respectively. The GM score is computed as the average cosine similarity between training gradients and validation gradients

over M consecutive steps. The gradient matching score here is similar to that in GM-NAS Hu et al. [2022]. When the GM score falls below a predefined threshold of ξ ∈ (0 , 1) within M steps, the operation is considered to be overfitting. Consequently, if GM < ξ for an operation o , the parameters of this operation will not be updated during subsequent supernet training. And the remaining operations and architecture parameters will be trained as usual.

Through the operation-level early stopping method, the overfitting of operations in the supernet can be alleviated. Benefiting from the attribute that the DARTS search process involves alternating between training the supernet on the training data and training architecture parameters on the validation data, we only need to maintain the gradient of each operation on a training data batch and a validation data batch in one iteration, with negligible additional computational overhead. As a result, OLES can alleviate the domination of skip connections, making DARTS more robust. This allows for training DARTS with more iterations to obtain better architectures and sharper architecture parameters, resulting in a smaller discretization gap and improving performance. The algorithm for DARTS with operation-level early stopping can be found in Algorithm 1.

4 Experiments

4.1 Search Spaces and Experimental Settings

To verify the effectiveness of OLES, we conduct experiments on three different search spaces: the standard DARTS' search space, NAS-Bench-201, and MobileNet-like search space. We keep all experimental settings the same as the original in each search space and only use the first-order optimization, as our method only needs to freeze the operation parameters during the supernet training. In addition to the standard hyperparameters in DARTS, our method introduces an additional hyperparameter, i.e., the overfitting threshold ξ , to stop operation training. We determine the threshold ξ by averaging the cosine similarity over 20 iterations for 30 randomly initiated architectures in each search space. And the gradient matching (GM) score is dynamically computed by averaging over every 20 iterations throughout the entire training process to reduce variance. Following Algorithm 1, we stop updating parameters of the corresponding operation o when GM ( g o train , g o val ) is lower than the threshold ξ .

4.2 Performance on DARTS' CNN Search Space

4.2.1 CIFAR-10

We conduct NAS on CIFAR-10 using DARTS' search space and then transfer the searched architectures to CIFAR-100. The threshold ξ is set to 0 . 3 . As shown in Table 1, our method OLES consistently outperforms other recent SOTA NAS methods. OLES achieves a top-1 test error of 2 . 30% on CIFAR-10 with nearly the same search cost as DARTS of 0 . 4 GPU-days. Moreover, the robustness of OLES is also guaranteed, as evidenced by the excellent average result of three independent runs. When the architectures searched on CIFAR-10 are transferred to CIFAR-100, our method still achieves a competitive test error of 16 . 30% . This indicates that the architectures searched by OLES are transferable and expressive, and the architecture parameters do not overfit the CIFAR-10 dataset.

4.2.2 CIFAR-100

We conduct NAS on CIFAR-100, and the results are presented in Table 2. The threshold ξ is set to 0 . 4 . OLES still exhibits significant advantages, improving the test error of DARTS from 20 . 58% to 17 . 30% without incurring extra search costs. It demonstrates that our method can adapt to datasets of different sizes and consistently improve the performance of DARTS. OLES not only consistently outperforms other methods that focus on the skip-connection domination issue, but also surpasses more sophisticated NAS methods (e.g., GM+DARTS Hu et al. [2022]) on CIFAR-10 and CIFAR-100 without requiring extensive code modifications or introducing additional computational overheads.

4.2.3 Transfer to ImageNet

To verify the transferability of architectures discovered by OLES, we transfer the best architecture derived from CIFAR-10 to ImageNet. Following the setting of DARTS, we train the searched

±

architecture from scratch for 250 epochs, and the results are presented in Table 3. Our method achieves 24 . 5% top-1 test error. Compared to other NAS methods, the architectures derived from OLES still exhibit competitive performance. Notably, we do not introduce any additional training techniques such as SE modules and Swish, but simply follow the original DARTS setting to transfer the architectures. Therefore, the performance of OLES on ImageNet can be further enhanced by incorporating these training tricks.

4.3 Performance on NAS-Bench-201

NAS-Bench-201 Dong and Yang [2020] provides a unified benchmark for analyzing various up-todate NAS algorithms. It contains 4 internal nodes with 5 operations (i.e., Zero, Skip Connection, 1 × 1 Conv, 3 × 3 Conv, 3 × 3 AvgPol). NAS-Bench-201 offers a similar cell-based search space that comprised a total of 15625 unique architectures. The architectures are trained on three datasets (i.e., CIFAR-10, CIFAR-100, and ImageNet16-120). The comparison results are shown in Table 4, and the threshold ξ is set to 0 . 7 . DARTS performs extremely poorly on NAS-Bench-201. We suppose that it is due to the fact that the search space of NAS-Bench-201 is extremely compact, contains few parametric operations, and the number of operation parameters varies widely, which makes it difficult for DARTS to adapt to NAS-Bench-201. OLES significantly outperforms DARTS on all datasets and is competitive among other NAS methods. It demonstrates the generality of OLES, which can adapt to different search spaces and datasets.

4.4 Performance on MobileNet-like Search Space

As described in FairDARTS Chu et al. [2019a], the domination of skip connections also exists in the MobileNet-like search space. The supernet is built on MobileNetV2 Sandler et al. [2018] and comprises 21 choice blocks, each with 7 candidate operations. As the MobileNet search space is not naturally designed for DARTS, we make minor modifications to the MobileNet search space as FairDARTS and perform NAS on the ImageNet dataset. The threshold

ξ is set to 0 . 4 . The results are summarized in Table 5, and the corresponding searched architecture is visualized in Appendix D.1. We can observe that OLES can also achieve highly competitive performance in the MobileNet search space.

5 In-Depth Analysis

5.1 Different Gradient Direction Similarity Measures

As described in Section 3.2, we use cosine similarity to measure the difference in the operation's gradient directions between the training and validation data. To examine the effect of different gradient direction similarity measures for computing the gradient matching score, we compare cosine similarity with three other measures: L 2 distance, direction distance, and hard threshold. The mathematical definitions of the other three measures can be found in Appendix B. The direction distance directly computes the fraction of elements that have different directions in two gradients. And the hard threshold does not use the averaged value of M iterations to compute the GM score but instead directly checks whether the direction distance can consistently be greater than a threshold in consecutive M iterations. As shown in Table 6, cosine similarity performs better than the other three methods. Therefore, we adopt cosine similarity as the gradient direction similarity measure of OLES.

5.2 Gradient Matching Score

0

10

20

30

40

50

60

70

Steps

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Cosine Similarity

DARTS CIFAR-10

DilConv3

SepConv3

SepConv5

DilConv5

55

60

65

70

75

80

85

Test acc

0

10

20

30

40

50

60

70

Steps

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Cosine Similarity

DARTS CIFAR-100

DilConv3

SepConv3

SepConv5

DilConv5

20

30

40

50

60

Test acc

0

20

40

60

80

100

Steps

0.3

0.4

0.5

0.6

0.7

0.8

Cosine Similarity

NAS-BENCH-201

ReLUConvBN1

ReLUConvBN3

40

50

60

70

80

Valid acc

In Figure 3, we present the trajectories of consine similarity between gradients on training data and validation data during the DARTS training. With the training process of DARTS, the cosine similarities of parametric operations gradually decrease, which indicates that the cosine similarity can reflect the overfitting of operations. And different parametric operations show similar trajectories. Consequently, we set an unified GM score threshold for all parametric operations. Moreover, when the number of training iterations continuously increases (>50), DARTS exhibits a noticeable decline in test performance. Figure 3c further corroborates this finding, as the problem of performance collapse due to the skip connection domination issue more easily appears in NAS-Bench-201. As a result, it manifests the test performance declines at an earlier stage.

5.3 Training With Longer Epochs

As mentioned in Bi et al. [2019] and Chu et al. [2020], training DARTS with longer epochs enables a better convergence of the supernet and architecture parameters. However, as shown in Figure 4a and 4b, training standard DARTS with longer epochs exhibit serious performance collapse with the domination of skip connections. The skip connection rapidly dominates after 50 epochs on CIFAR-10 and takes longer epochs on CIFAR-100. And we also find that other methods (e.g., FairDARTS, and DARTS-PT) fail with longer training epochs as well. In contrast, as shown in Figure 4c and 4d, OLES survives even after 200 epochs. The number of skip connections in OLES stays stable after a specific number of epochs since most parametric operations are frozen. It demonstrates that OLES is robust and can actually address the domination of skip connections. More experimental results and searched architectures with longer training epochs can be found in Appendix C.2 and Appendix D.2.

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

6

7

8

No. of Skips

normal

reduce

(a) DARTS on CIFAR-10.

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

6

7

8

No. of Skips

normal

reduce

(b) DARTS on CIFAR-100.

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

6

7

8

No. of Skips

normal

reduce

(c) OLES on CIFAR-10.

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

6

7

8

No. of Skips

normal

reduce

(d) OLES on CIFAR-100.

5.4 Comparison of Kendall Rank Correlation Coefficients

76

78

80

82

84

86

88

Search Accuracy(%)

96.7

96.8

96.9

97.0

97.1

97.2

Retraining Accuracy(%)

(78.54,96.72)

(75.42,96.76)

(83.90,96.91)

(88.06,96.99)

(83.87,97.03)

(81.38,97.02)

(76.46,97.09)

(81.09,97.09)

(84.07,97.16)

(81.79,97.24)

Kendall_tau=0.13

(a) DARTS (2nd Order)

82

83

84

85

86

87

88

89

90

Search Accuracy(%)

97.0

97.1

97.2

97.3

97.4

97.5

97.6

97.7

Retraining Accuracy(%)

(88.25,97.25)

(88.36,97.72)

(87.94,97.02)

(82.72,97.32)

(87.27,97.19)

(88.08,97.21)

(88.00,97.17)

(86.72,97.09)

(88.26,97.26)

(87.30,97.02)

Kendall_tau=0.41

(b) OLES (ours)

Figure 5 indicates that the early stopping mechanism does not hurt the ranking performance of DARTS. Additionally, we compare the Kendall rank correlation coefficients with other NAS methods, including RandomNAS, DARTS, GDAS, and SGAS, as shown in Table 7.

Notably, the Kendall coefficient of OLES aligns closely with that of SGAS Li et al. [2020], which aims to alleviate the effect of the degenerate search-retraining correlation problem. The results demonstrate that by mitigating operation parameter overfitting, differentiable NAS is able to focus on the potential of architectures themselves, thus enhancing the correlation between search metrics and the architectures discovered.

6 Conclusion

This paper attempts to find the fundamental reason for the domination of skip connections in DARTS from the new perspective of overfitting of operations in the supernet. We verified our conjecture through motivational experiments and proposed the operation-level early stopping (OLES) method to solve the domination of skip connections. OLES monitors the gradient direction of each operation and determines whether it is overfitting based on the difference between the gradient directions on the training data and the validation data. OLES can solve the domination of skip connections with negligible computational overheads without imposing any limitations. Moreover, OLES achieves state-of-the-art results of 2 . 30% test error on CIFAR-10 and 17 . 30% test error on CIFAR-100, meanwhile reporting superior performance when transferred to ImageNet and on search spaces of NAS-Bench-201 and MobileNet as well.

Acknowledgment

This work was supported by the National Natural Science Foundation of China (#62102177), the Natural Science Foundation of Jiangsu Province (#BK20210181), the Key R&D Program of Jiangsu Province (#BE2021729), and the Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China.

References

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016.

Chakkrit Termritthikun, Yeshi Jamtsho, Jirarat Ieamsaard, Paisarn Muneesawang, and Ivan Lee. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence , 104:104397, 2021.

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search, 2018. URL https://arxiv.org/abs/1802.01548 .

Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018. URL https://arxiv.org/abs/1802.03268 .

Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 82-92, 2019.

Guanghui Zhu, Wenjie Wang, Zhuoer Xu, Feng Cheng, Mengchuan Qiu, Chunfeng Yuan, and Yihua Huang. Psp: Progressive space pruning for efficient graph neural architecture search. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 2168-2181, 2022a. doi: 10.1109/ICDE53745.2022.00208.

Guanghui Zhu, Zhennan Zhu, Wenjie Wang, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. Autoac: Towards automated attribute completion for heterogeneous graph neural network. In 2023 IEEE 39th International Conference on Data Engineering (ICDE) , pages 2808-2821, 2023. doi: 10. 1109/ICDE55515.2023.00215.

Jingfan Chen, Guanghui Zhu, Haojun Hou, Chunfeng Yuan, and Yihua Huang. Autogsr: Neural architecture search for graph-based session recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , page 1694-1704, 2022.

Guanghui Zhu, Feng Cheng, Defu Lian, Chunfeng Yuan, and Yihua Huang. Nas-ctr: Efficient neural architecture search for click-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , page 332-342, 2022b.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2018. URL https://arxiv.org/abs/1806.09055 .

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence , pages 367-377. PMLR, 2020.

Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv:1902.08142 , 2019.

Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. Advances in neural information processing systems , 32, 2019.

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild, 2019. URL https://arxiv.org/abs/1912.10952 .

Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping, 2019. URL https://arxiv.org/abs/1909.06035 .

Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-: Robustly stepping out of performance collapse without indicators, 2020. URL https://arxiv.org/abs/ 2009.01027 .

Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search, 2019. URL https://arxiv. org/abs/1909.09656 .

Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable nas, 2021. URL https://arxiv.org/abs/2108.04392 .

Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search, 2019a. URL https://arxiv.org/abs/1911.12126 .

Xiangxiang Chu and Bo Zhang. Noisy differentiable architecture search, 2020. URL https: //arxiv.org/abs/2005.03566 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv.org/abs/1512.03385 .

Sirui Xie, Shoukang Hu, Xinjiang Wang, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Understanding the wiring evolution in differentiable neural architecture search. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 874-882. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/ xie21a.html .

Shoukang Hu, Ruochen Wang, Lanqing Hong, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. Generalizing few-shot nas with gradient matching, 2022.

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision , pages 544-560. Springer, 2020.

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 , 2018.

Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 , 2018.

David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning , pages 5877-5886. PMLR, 2019.

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition, 2017. URL https://arxiv.org/abs/1707.07012 .

Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102.11535 , 2021a.

Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. arXiv preprint arXiv:2301.11300 , 2023.

Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In International Conference on Machine Learning , pages 12707-12718. PMLR, 2021.

Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2016. URL https://arxiv.org/abs/1608.06993 .

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search, 2017. URL https://arxiv.org/abs/1712.00559 .

Li, G. Qian, I. C. Delgadillo, M. Muller, A. Thabet, and B. Ghanem. Sgas: Sequential greedy architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1617-1627, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10. 1109/CVPR42600.2020.00169. URL https://doi.ieeecomputersociety.org/10.1109/ CVPR42600.2020.00169 .

Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search, 2021b.

Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. β -darts: Beta-decay regularization for differentiable architecture search, 2022. URL https://arxiv.org/abs/2203. 01665 .

Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours, 2019. URL https://arxiv.org/abs/1910.04465 .

Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Theory-inspired path-regularized differential network architecture search. CoRR , abs/2006.16537, 2020. URL https://arxiv. org/abs/2006.16537 .

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision, 2015. URL https://arxiv.org/abs/ 1512.00567 .

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks, 2018. URL https://arxiv.org/abs/ 1801.04381 .

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017. URL https://arxiv.org/abs/1707. 01083 .

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile, 2018. URL https: //arxiv.org/abs/1807.11626 .

Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search, 2019. URL https://arxiv.org/abs/1907.05737 .

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=HJxyZkBKDr .

Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: direct neural architecture search without parameter retraining. CoRR , abs/2002.09128, 2020. URL https://arxiv.org/abs/2002.09128 .

Miao Zhang, Steven W. Su, Shirui Pan, Xiaojun Chang, Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. CoRR , abs/2106.10784, 2021a. URL https://arxiv.org/abs/2106.10784 .

Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. CoRR , abs/2101.11834, 2021b. URL https://arxiv.org/abs/2101.11834 .

Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CoRR , abs/1812.03443, 2018. URL http://arxiv. org/abs/1812.03443 .

Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. CoRR , abs/1907.01845, 2019b. URL http://arxiv. org/abs/1907.01845 .

Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing darts with amended gradient estimation on architectural parameters, 2019. URL https://arxiv.org/ abs/1910.11831 .

A The Selection of Parametric-Free Operations on Various Search Spaces

Zela et al. [2019] designed four variants of the DARTS' search space (S1-S4) and pointed out skip connections tend to dominate compared with two other parametric-free operations, i.e., zero and noise . To investigate whether the robustness issue in DARTS arises from the domination of skip connections rather than other parametric-free operations, we extend the experimental settings in Zela et al. [2019] to seven search spaces, including four types of parametric-free operations( avg_pool_-3 × 3, skip_connect, noise, zero ). These search spaces only differ from DARTS' original space in the number and types of operations on each edge. We follow the same experimental setup in Liu et al. [2018] and execute DARTS on each search space. Figures 6-8 and 13-15 display the results on S1-S3. We observe that skip connections tend to dominate more severely than other parametric-free operations. By designing different combinations of parametric-free operations in S4-S7 search spaces, Figures 9-12 and 16-19 show that skip connections also dominate under fair competition with other parametric-free operations when the searched cells contain more than one type of parametric-free operations. The corresponding performance for each search space is shown in Table 8. As a result, the domination of skip connections is the primary cause of the shallow architecture of DARTS with poor generalization and robustness.

S1 consists of two operations: skip_connect and sep_conv_3 × 3 .

S2 consists of two operations: avg_pool_3 × 3 and sep_conv_3 × 3 .

S3 consists of two operations: noise and sep_conv_3 × 3 . The noise operation replaces each input value with a random Gaussian noise for N (0 , 1) . Differentiable NAS algorithms generally avoid this operation because it hurts the performance of discretized architectures.

S4 consists of three operations: skip_connect , noise , and sep_conv_3 × 3 .

S5 consists of three operations: skip_connect , avg_pool_3 × 3 , and sep_conv_3 × 3 .

S6 consists of three operations: skip_connect , zero , and sep_conv_3 × 3 . The zero operation replaces each input value with zeros.

S7 consists of five operations: skip_connect , avg_pool_3 × 3 , noise , zero , and sep_conv_3 × 3 .

c_{k-2}

0

skip_connect

1

sep_conv_3x3

c_{k-1}

skip_connect

2

skip_connect

3

skip_connect

skip_connect

c_{k}

skip_connect

skip_connect

(a) Normal Cell

c_{k-2}

0

skip_connect

1

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

skip_connect

2

sep_conv_3x3

skip_connect

sep_conv_3x3

c_{k}

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

3

sep_conv_3x3

avg_pool_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

avg_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

avg_pool_3x3

avg_pool_3x3

avg_pool_3x3

c_{k}

sep_conv_3x3

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

2

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

noise

noise

noise

3

noise

c_{k}

noise

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

c_{k}

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

skip_connect

c_{k-1}

skip_connect

skip_connect

2

skip_connect

c_{k}

noise

3

noise

noise

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

skip_connect

sep_conv_3x3

2

sep_conv_3x3

skip_connect

c_{k}

sep_conv_3x3

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

skip_connect

skip_connect

c_{k}

sep_conv_3x3

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

c_{k-1}

skip_connect

1

avg_pool_3x3

skip_connect

2

skip_connect

3

skip_connect

c_{k}

skip_connect

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

skip_connect

skip_connect

c_{k}

sep_conv_3x3

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

c_{k-1}

skip_connect

1

avg_pool_3x3

skip_connect

2

skip_connect

3

skip_connect

c_{k}

skip_connect

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

skip_connect

3

skip_connect

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

noise

c_{k}

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

skip_connect

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

skip_connect

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

c_{k-2}

0

skip_connect

1

sep_conv_3x3

2

skip_connect

c_{k-1}

sep_conv_3x3

sep_conv_3x3

skip_connect

c_{k}

3

skip_connect

skip_connect

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

2

sep_conv_3x3

skip_connect

3

sep_conv_3x3

c_{k}

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

sep_conv_3x3

sep_conv_3x3

2

sep_conv_3x3

c_{k}

sep_conv_3x3

3

sep_conv_3x3

sep_conv_3x3

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

sep_conv_3x3

sep_conv_3x3

2

sep_conv_3x3

3

sep_conv_3x3

c_{k}

sep_conv_3x3

sep_conv_3x3

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

c_{k}

sep_conv_3x3

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

skip_connect

c_{k-1}

skip_connect

skip_connect

2

noise

c_{k}

noise

3

noise

noise

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

1

skip_connect

2

sep_conv_3x3

skip_connect

sep_conv_3x3

sep_conv_3x3

c_{k}

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

c_{k-1}

sep_conv_3x3

skip_connect

2

skip_connect

c_{k}

sep_conv_3x3

3

sep_conv_3x3

sep_conv_3x3

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

c_{k-1}

skip_connect

1

skip_connect

sep_conv_3x3

2

skip_connect

3

skip_connect

c_{k}

skip_connect

sep_conv_3x3

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

skip_connect

2

skip_connect

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

c_{k}

sep_conv_3x3

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

3

skip_connect

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

2

sep_conv_3x3

c_{k}

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

skip_connect

3

skip_connect

c_{k-1}

skip_connect

skip_connect

skip_connect

skip_connect

c_{k}

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

skip_connect

3

sep_conv_3x3

c_{k-1}

sep_conv_3x3

sep_conv_3x3

sep_conv_3x3

skip_connect

c_{k}

(b) Reduction Cell

B Gradient Distance Similarity Measures

In Section 5.1, four gradient distance measures are present. Next, we provide a comprehensive mathematical definition for each measure. Let g o train and g o val denote the gradients of the operation parameters P o on a training batch and a validation batch, respectively.

The GM score with L 2 distance can be formulated as follows:

And the GM score with direction distance can be formulated as follows:

where ⊙ denotes the element-wise product, and ∥ g o train ⊙ g o val < 0 ∥ 1 represents the number of elements whose directions are different.

However, the hard threshold method does not use the averaged gradient distances over M steps to calculate the GM score. Instead, it directly checks whether the direction distance can consistently exceed a threshold over the consecutive M steps. Let ξ denote the threshold. The hard threshold method checks whether ∥ g o train ⊙ g o val < 0 ∥ 1 ∥ P o ∥ 0 > ξ can still be satisfied for M consecutive steps.

C Supplementary Experimental Results

C.1 Quantitative Analysis on Negative Correlation between Validation Loss and Architecture Parameter

The results of our motivation experiments, as depicted in Figure 1, indicate a negative correlation between validation loss and architecture parameter. To further provide a more rigid analysis, we calculate the Kendall-correlations between validation loss and architecture parameter for each operation and present the results in the Table 9. It can be observed that there is a clear negative correlation between validation losses and architecture parameters, particularly for parametric operations.

-

-

-

-

C.2 Results with Longer Training Epoches

60

80

100

120

140

160

180

200

Epochs

88

90

92

94

96

98

100

OLES

DARTS-PT

DARTS

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

6

7

No. of Skips

DARTS

DARTS+PT

OLES

(a) Normal Cell

0

25

50

75

100

125

150

175

200

Epochs

0

1

2

3

4

5

No. of Skips

DARTS

DARTS+PT

OLES

(b) Reduction Cell

Despite previous attempts to investigate the origins of the skip connection domination issue and its corresponding solutions, it remains an open problem. These methods have put forth their hypotheses but have not fully resolved the issue. To further exemplify the efficacy of addressing the skip connection domination issue, we increase the number of iterations of supernet training. As depicted in Figures 20 and 21, DARTS-PT Wang et al. [2021] fails to address the skip connection domination issue when extending to very long training epoches, whereas OLES remains stable.

C.3 Hyperparameter Analysis

In OLES, the gradient matching score is dynamically computed by averaging over every 20 iterations throughout the entire training process. The threshold ξ is determined based on the GM scores within the initial 20 iterations. We conduct experiments to assess various iterations along with their corresponding thresholds across different search spaces to explain the rationale behind selecting 20 iterations. The results, as shown in Table 10, demonstrate that computing the GM score using an average across every 20 iterations yields the highest accuracy.

C.4 Results on DARTS S1-S4

DARTS S1-S4 are simplified variations of the DARTS search space proposed by R-DARTS Zela et al. [2019] to investigate DARTS' failure mode. In Appendix A, we have further expanded these search spaces and presented DARTS' failure mode (both in terms of performance and the searched architectures) in these extended search spaces. In Table 11, we additionally provide a performance comparison between OLES and other methods on DARTS S1-S4. The results demonstrate that OLES also consistently outperforms DARTS, as well as R-DARTS Zela et al. [2019] and DARTS-PT Wang et al. [2021] which are also dedicated to addressing the skip connection domination issue.

C.5 Comparison with Train-Free NAS Methods

Table 12 compares OLES with two recently proposed train-free NAS methods, TE-NAS Zhao et al. [2021] and ZiCo Li et al. [2023]. It is evident that OLES still outperforms train-free methods in most cases. While train-free methods exhibit slightly superior performance on NAS-Bench-201 due to its compact search space, in larger search spaces such as the DARTS search space, OLES performs better than TE-NAS and ZiCo. Train-free NAS is a highly promising research direction, as it not only greatly enhances NAS efficiency but also promotes a deeper understanding of neural networks.

±

±

D Searched Architectures

D.1 Architecture on MobileNet-like Search Space

Following the experimental settings in Chu et al. [2019a], we modify the MobileNetV2 Sandler et al. [2018] search space and perform OLES on ImageNet in this search space. The searched architecture is illustrated in Figure 22.

MB3  5

×

5

MB3  3

×

3

Identity

MB3 5

×

5

MB6 3

×

3

Identity

MB6 7

×

7

MB6 3

×

3

MB3 3

×

3

MB6 3

×

3

MB6 3

×

3

MB6 3

×

3

MB6 3

×

3

Identity

MB3 3

×

3

MB6 3

×

3

MB6 7

×

7

MB6 7

×

7

MB6 7

×

7

MB6 7

×

7

MB6 3

×

3

D.2 Training With Longer Epochs

We train OLES and several other differentiable architecture search methods with 200 epochs to verify whether they are robust enough in this case. According to our conjecture, as the number of training epochs increases, the operations in the supernet become more prone to overfitting, leading to a more serious issue of the domination of skip connections. In this section, we present some additional results of the searched architectures using different numbers of training epochs.

D.2.1 Results on OLES

As described in Section 5.3, OLES can survive with longer training epochs. Specifically, the number of skip connections in OLES keeps stable even when the number of training epochs reaches 150 or 200. The corresponding architecture genotypes on CIFAR-10 and CIFAR-100 are visualized in Figures 23-26.

c_{k-2}

0

sep_conv_3x3

1

sep_conv_5x5

2

sep_conv_3x3

3

skip_connect

c_{k-1}

sep_conv_3x3

dil_conv_5x5

dil_conv_3x3

sep_conv_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

max_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

max_pool_3x3

skip_connect

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_5x5

2

sep_conv_3x3

3

skip_connect

c_{k-1}

sep_conv_3x3

dil_conv_5x5

dil_conv_3x3

sep_conv_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

max_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

max_pool_3x3

skip_connect

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

skip_connect

2

skip_connect

3

skip_connect

c_{k-1}

sep_conv_3x3

sep_conv_3x3

dil_conv_3x3

sep_conv_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

avg_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

avg_pool_3x3

dil_conv_3x3

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

skip_connect

2

skip_connect

3

skip_connect

c_{k-1}

sep_conv_3x3

sep_conv_3x3

dil_conv_3x3

sep_conv_3x3

c_{k}

(a) Normal Cell

c_{k-2}

0

avg_pool_3x3

1

avg_pool_3x3

2

avg_pool_3x3

3

avg_pool_3x3

c_{k-1}

avg_pool_3x3

dil_conv_3x3

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

D.2.2 Results on FairDARTS

FairDARTS Chu et al. [2019a] adopts the Sigmoid instead of the exclusive Softmax for architecture parameters and selects operations through a threshold value. However, our experiments show that no operation can achieve the pre-defined threshold value in the normal cell when FairDARTS is trained for 150 or 200 epochs. Following FairDARTS' settings, we train it for 50, 150, and 200 epochs. Table 13 shows the architecture genotypes, which can be visualized in Figure 27 and Figure 28.

c_{k-2}

0

sep_conv_3x3

1

dil_conv_3x3

c_{k-1}

dil_conv_5x5

dil_conv_3x3

2

sep_conv_5x5

c_{k}

(a) Normal Cell

c_{k-2}

0

skip_connect

1

sep_conv_3x3

2

sep_conv_3x3

3

skip_connect

c_{k-1}

dil_conv_5x5

skip_connect

skip_connect

skip_connect

c_{k}

(b) Reduction Cell

c_{k-2}

0

max_pool_3x3

1

max_pool_3x3

2

max_pool_3x3

c_{k-1}

max_pool_3x3

skip_connect

skip_connect

3

skip_connect

c_{k}

skip_connect

(a) Epoch=150

c_{k-2}

0

max_pool_3x3

1

max_pool_3x3

2

max_pool_3x3

c_{k-1}

max_pool_3x3

skip_connect

skip_connect

3

skip_connect

c_{k}

skip_connect

(b) Epoch=200

D.2.3 Results on DARTS+PT

DARTS+PT Wang et al. [2021] evaluates operation strength by measuring its contribution to the supernet's performance instead of the magnitude of the architecture parameter. Following DARTS+PT's settings, we train DARTS with the permutation-based architecture selection method for 50, 150, and 200 epochs, and the corresponding architecture genotypes are shown in Table 14. When extending to over 150 epochs, the resulting architectures are also full of skip connections. Furthermore, to avoid inaccurate evaluation of the operation strength during discretization, DARTS+PT needs to fine-tune the remaining supernet until it converges again. The architecture genotypes with tuned supernets are shown in Table 15, Figure 29, and Figure 30. It's worth noting that the test error of the final resulting architecture for 200 epochs on CIFAR-10 will drop to 6 . 28% .

c_{k-2}

0

sep_conv_3x3

1

skip_connect

2

skip_connect

c_{k-1}

dil_conv_3x3

3

skip_connect

skip_connect

c_{k}

dil_conv_5x5

sep_conv_5x5

(a) Normal Cell

c_{k-2}

0

max_pool_3x3

1

avg_pool_3x3

2

dil_conv_5x5

c_{k-1}

max_pool_3x3

skip_connect

3

dil_conv_5x5

c_{k}

skip_connect

skip_connect

(b) Reduction Cell

c_{k-2}

0

sep_conv_3x3

1

sep_conv_3x3

2

skip_connect

c_{k-1}

dil_conv_3x3

3

skip_connect

skip_connect

c_{k}

dil_conv_5x5

dil_conv_5x5

(a) Normal Cell

c_{k-2}

0

sep_conv_3x3

1

max_pool_3x3

3

sep_conv_5x5

c_{k-1}

max_pool_3x3

2

sep_conv_3x3

skip_connect

c_{k}

skip_connect

skip_connect

(b) Reduction Cell

Peer reviews:
comments: SUMMARY:
This paper studies the robustness issue of DARTS from the perspective of overfitting. It uses gradient matching scores to measure the overfitting issues, and proposes an early-stop strategy to address the problem of saturated skip connections in normal DARTS. The proposed approach has been evaluated on a number of search spaces, showing comparable results to the state of the art.

STRENGTHS:
+ The idea of using similarity between gradient directions of training vs validation batch makes sense. 
+ Extensive experiments on various search spaces and benchmarks. 
+ Decent results on DARTS C10 space comparing to SOTA.

WEAKNESSES:
- Although the idea to use early stopping to robustify DARTS is interesting, the metric used in this paper is very similar to that in GM-NAS [1]. Therefore it seems to me that the novelty of this paper is somewhat discounted. 
- It is good to see experiments on a number of search spaces and benchmarks. However, the results on DARTS S1-S4 space is missing. 
- Some discussion on overfitting is not very clear. e.g. from Fig.1, it is difficult to see quantitively how the negative correlation between arch params and val loss: indeed the curvature of the lines are different, but it would be better to have some more rigid analysis. 


[1] Generalizing Few-Shot NAS with Gradient Matching. Shoukang Hu*, Ruochen Wang*, Lanqing Hong, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. ICLR 2022.

QUESTIONS:
* How the proposed approach performs on DARTS S1-S4?
* It seems this approach doesn't work so well on MobileNet space, any possible reasons for that? 
* How this approach may compare to the train-free methods, like TE-NAS and ZiCo?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The authors are adressing the issue of converging to a degenerated solution (many skip connections) using DARTS. The authors connect this behavior to an overfitting to the train data. To remedy this issue, they suggest to apply early stopping based on the correlation of gradients during the architecture parameters training and the wekths parameters training.

STRENGTHS:
The paper is trying to tackle an interesting problem in an elegant way. The story is clearly stated and the introduced approach is a nice extension of the classical early stopping. The performance of this method is on par or better than many other more sophisticated (and to some extent complicated) method.

WEAKNESSES:
* The authors are claiming that the dominance of skip connections is due to the quick overfitting of weighted operations to the train data. The whole story of the paper relies on this assumption which is unfortunately not sufficiently supported. While curves on figure 1(a) are supporting this claim, figures 3(a) and 3(b) are a bit confusing.  I would have expected the test performance to drop for both cases.

* There is an issue regarding the reported numbers: the authors are claiming that their method benefits operations with parameters and the test error is dropping (as expected), I am wondering why the number of parameters is barely different.

QUESTIONS:
Adressing the mentioned weaknesses appropriatly would lead to a higher rating

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

comments: SUMMARY:
This paper demonstrates the fundamental reason for the domination of skip connections in DARTS from the new perspective of overfitting of operations in the supernet, using preliminary experiments,  and proposed the operation-level early stopping method to mitigate this phenomenon by using the GM score metric during the searching.

STRENGTHS:
The idea of using GM metric to decide whether to update the OPs is broadly used in efficient-training papers.  This paper proposes to use this metric to early stop the updating of specific OPs during the NAS procedure, based on their novel overfitting observations. 

The comparison is intensive, showing the superiority of this proposed method, in terms of time cost, and accuracy.

WEAKNESSES:
1. The accuracy metric is one of the metrics to measure a NAS method. We also consider the Kendall rank correlation coefficient. Please compare with previous methods using this metric, because I am not sure whether the proposed early stopping mechanism will hurt the ranking or not. Usually, we say the ranking performance of a NAS method may be more important than the accuracy of the searched model. 

2. About the overfitting threshold, "we determine the threshold by averaging the cosine similarity over 20 iterations for 30 randomly initiated architectures in each search space"    Is the initial 20 iterations ok or enough for determining the threshold? The initial stages may have dramatic changes in gradient. Also, the threshold is fixed during the NAS procedure, shouldn't it be adaptive or scheduled?

QUESTIONS:
See Weakness

LIMITATIONS:
Yes, the authors adequately addressed the limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the robustness issues in differentiable NAS, specifically the domination of skip connections. It first analyzes the issue from a novel perspective, proposing that the domination of skip connections arises due to the overfitting of operations in the supernet during training. Then, the paper proposes the operation-level early stopping method, which monitors each operation in the supernet and stops its training when it tends to overfit. The paper employs a gradient matching approach to detect overfitting, comparing the gradients' directions of operations on training and validation data. A significant deviation in direction is considered an indication of overfitting. The proposed OLES addresses the domination of skip connections with negligible additional overhead. Extensive experiments demonstrate the effectiveness of OLES on different datasets and search spaces.\n\nSTRENGTHS:\nS1. The paper demonstrates good originality. It provides a comprehensive analysis of the issue of the domination of skip connections in the differentiable NAS by dopting a new perspective. Although straightforward, the perspective makes sense and is interesting. Specifically, the paper aims to explain the cause of this issue through the overfitting of operations in the supernet. The paper proposes the operation-level early stopping (OLES) method, which introduces gradient matching to address this matter. OLES elegantly and effectively resolves the domination of skip connections, incurring negligible additional overhead. \n\nS2. The experiments are thorough and well-organized, containing experiments in different search spaces and an in-depth analysis of the proposed algorithm. The empirical results demonstrate that the proposed OLES  achieves state-of-the-art performance on CIFAR. The availability of open-source codes further facilitates reproducibility.\n\nS3. The presentation of ideas and algorithms is clear, while the references and background knowledge are comprehensive. The background knowledge and the issue to be solved are adequately introduced.\n\nS4. The perspective and idea about the overfitting of operations have profound significance in uncovering the underlying causes of the domination of skip connections in DARTS. These novel perspectives, ideas, and algorithms contribute to a deep understanding of differentiable architecture search and may inspire future research in the field.\n\nWEAKNESSES:\nW1. It needs a thorough explanation of the used gradient matching method. The authors should provide a more detailed introduction to gradient matching and clarify the differences from other methods.\n\nW2. While the proposed method demonstrates significant improvements over the original DARTS in the experiments, it does not have a competitive advantage compared to other state-of-the-art methods. \n\nW3. The paper specifically focuses on addressing the domination of skip connections through operator-level early stopping. The authors are suggested to discuss how the concept of operator-level early stopping can be applied to other scenarios beyond differentiable architecture search.\n\nQUESTIONS:\nQ1. Are there any limitations to the methodology presented in this paper? Can it be applied to other differentiable NAS methods beyond the original DARTS?\n\nQ2. Why does the proposed method not exhibit a competitive advantage over other state-of-the-art NAS methods?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a method namely operation-level early stopping to address the skip-connection domination issue in domain of differentiable architecture search (DARTS). Though this problem is heavily explored in the past, the authors believe that the key reason of skip-connection domination is because of they try to overfit the validation set used in DARTS based method, and their proposed operation-level early stop that conceptually stop the training of architecture parameter if the overfitting is observed.\n\nSTRENGTHS:\nThe problem of skip-connection dominance is a long standing problem in DARTS domain, and novel method to address this has a clear motivation. \nThe hypothesis of overfitting is the root cause of skip-connection domination is novel and interesting. \nAnalysis of architecture with validation loss is quite clear and justify their hypothesis.\nI appreciate the extensive and honest experiments on all kinds of settings, even though many of the results does not surpass the state-of-the-art.\n\nWEAKNESSES:\nI have several questions regarding this paper and hope to hear back from the authors.\n\n1. Utilizing gradient matching as an indicator to perform early stop seems okay, but this paper lacks of sufficient analysis of what is the key difference between GM+DARTS and their approach, especially regarding why OLES is better. In the related work section, it only describes GM-NAS introduces gradient match score into NAS literature, but this is not enough to let readers understand the difference. \n2. In essence, use early stop to avoid overfitting in DARTS is novel, but experiments seem to show that this OLES does not surpass previous DARTS+PT, which is another indicator to select DARTS operation. I do not understand the urgency to accept another indicator work with similar performance. In addition, similar as above, I do not see much comparison of how OLES surpass DARTS+PT. I agree that OLES can surpass original DARTS, but if it cannot surpass other methods that aims to address the skip-connection domination issue, this is inadequate to appear in a top-tier conference in my humble opinion. \n3. In general, the experiments compared to state-of-the-art is inferior. \nTo the best of my knowledge, this approach has a close relationship with GM+DARTS and DARTS+PT, where in all experiments, the authors should compare to. However, for example, in Table 2, none of these methods exists. In addition, when solely compared to GM+DARTS, in Table 1, it is 0.05 test error better than GM+DARTS. In table 3, GM+DARTS is basically identical as OLES (24.5 v.s. 24.5). In Table 4, OLES surpass GM+DARTS by a margin of 0.2, which is kind of significant. However, in Table 5, GM + ProxylessNAS surpasses OLES again. Why we should accept a paper using the exact gradient matching score in a different manner, which seems inferior than the original approach?\n\nIn addition, I would like to see how this OLES address skip-connection domination with other approaches that aim the same target, not in terms of their final performance, but with respect to their performance to address skip-connection domination issue. After all, this is not the first paper aiming to address it.\n\nQUESTIONS:\nSame as above\n\nLIMITATIONS:\nModerately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}