Operation-Level Early Stopping for Robustifying Differentiable NAS
Authors: Shen Jiang
Zipeng Ji
Guanghui Zhu
Chunfeng Yuan
Yihua Huang
Venue: NeurIPS 2023 poster (2023)
Keywords: Differentiable neural architecture search; Image classification; Failure of DARTS
TL;DR: Analyzing the domination of skip connections issue in DARTS from the perspective of overfitting of operations in the supernet and proposing the operation-level early stopping method to robustify DARTS.

Abstract: Differentiable NAS (DARTS) is a simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks.
% 
Nevertheless, DARTS still encounters several robustness issues, mainly the domination of skip connections.
% 
The resulting architectures are full of parametric-free operations, leading to performance collapse.
% 
Existing methods suggest that the skip connection has additional advantages in optimization compared to other parametric operations and propose to alleviate the domination of skip connections by eliminating these additional advantages.
% 
In this paper, we analyze this issue from a simple and straightforward perspective and propose that the domination of skip connections results from parametric operations overfitting the training data while architecture parameters are trained on the validation data, leading to undesired behaviors.
% 
Based on this observation, we propose the operation-level early stopping (OLES) method to overcome this issue and robustify DARTS without introducing any computation overhead.
% 
Extensive experimental results can verify our hypothesis and the effectiveness of OLES.

Operation-Level Early Stopping for Robustifying Differentiable NAS
Shen Jiang"
Zipeng Ji
Guanghui Zhu
Chunfeng Yuan
Yihua Huang
State Key Laboratory for Novel Software Technology, Nanjing University {jiangshen, (azipeng)$smale nju. edu. en, {zgh, cfyuan, yhuangHonju. edu.
Abstract
Differentiable NAS (DARTS) is simple and efficient neural architecture search method that has been extensively adopted in various machine learning tasks. Never- theless, DARTS still encounters several robustness issues, mainly the domination of skip connections. The resulting architectures are full of parametric- free operations, leading performance collapse. Existing methods suggest that the skip connection has additional advantages in optimization compared to other parametric operations and propose to alleviate the domination of skip connections by eliminating these additional advantages. In this paper, we analyze this issue from a simple and straightff wraighhforward perspective and propose that the domination of skip connections results from parametric operations overfitting the training data while architecture parameters trained on the validation data, leading undesired behaviors. Based on this observation, we propose the operation-level early stopping (OLES) method overeome this issue and robustify DARTS without introducing any computa- tion overhead. Extensive experimental results can verify our hypothesis and the effectiveness of OLES.
Introduction
Neural architecture search (NAS) aims to automatically design high- performance neural architec- tures Zoph and Le (016) for given tasks in a data-driven manner. Deep learning models designed by NAS have achieved state-of-the-arr performance in variety of tasks, such image classifica- tion Termrithhikn et al. [202T) Real et al. [2018) Pham et (2018)) semantic segmentation Liu
al. [2019), graph learning Zhu al. [2022a 2023 and recommender systems Chen al. [2022]. Zhu et al. [2022b]. Among various NAS methods, differentiable architecture search (DARTS) Liu
al. 2018 significantly improves the efficiency of NAS through parameter-sharing and continuous
relaxation. DARTS converts discrete architecture selections into continuous architecture parameters and optimizes them via gradient- praddeet--based optimizers.
Despite its simplicity and efficiency. DARTS still has several issues that make it difficult to outperform other NAS methods, even simple random search Li and Tal Talwalkar 2020 Yu al. 2019 many
cases. The first issue the performance gap. Due to the discrepancies between discrete architectures and continuous architecture parameters, the derived model shows non-negligible performance degra- dation Liu et all (2018). Additionally, DARTS suffers from serious robustness problems Nayman
al. 2019] Chen [2019] Liang al. (2019) Chu et al. 2020) Zela al. (2019] Wang et al.
"Equal contribution. *Coresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(02T]. In particular, during the search phase of DARTS, as the number of training epochs increases, the searched architectures will fill with parametric-free operations Chu et al.] (2019a]. such as skip connections or even random noise, leading to performance collapse. These issues waste lot of computational resources and seriously hinder the application of DARTS.
The domination of skip connections is the major robustness issue of DARTS Wang et al. 202
Many studies have investigated why the domination of skip connections happens and how to alleviate this issue. Most of them believe that skip connections exhibit unfair advantages in gradient-
optimization during the search phase Liang et [2019] Chu and Zhang [2020], Chu al. (209aa
or stabilize the supernet training by constructing residual blocks Chu et al. [2020]. To address
this problem. they propose eliminating these unfair advantages Chu et al. 2019 or separating
the stabilizing role Chu et al.) [2020], mainly drawing inspiration from ResNet He et al. (2015).
However, the generic optimization ResNet differs fundamentally from the bi-level optimization in DARTS. Therefore, observations and theories based on uni-level optimization are insufficient to support the claim that skip connections still have unfair advantages DARTS. The work [ang] al. [2021] analyzes this problem from the perspective of architecture selection and proposes that the magnitude of the architecture parameter may not reflect the strength the operation. Based on this assumption, a cntuurbtioonbasee architecture selection method is proposed. However, this idea conflicts with the fundamental motivation of DARTS, which utilizes continuous relaxation to solve the combinatorial optimization problem approximately. Xie et al. [2021) suggests that overfitting in bi-level optimization is the primary reason for the catastrophic failure of DARTS and alayzes from unified theoretical framework, which is similar to conjecture.
In contrast to the above sophisticated assumptions, in this paper, we take simpler and more straightforward perspective to consider the reasons for the domination of skip connections. We observe that the robustness problem of DARTS is caused by overfitting. Specifically, the parameters of operations (e.g., convolution) in the supernet overfit the training data and gradually deviate from the validation data. Meanwhile, the architecture parameters are trained on the validation data, causing the advantages of parametric operations to weaken progressively and ultimately leading to the domination of parametric free operations. It is evident that the supernet is always overparameterized. Additionally, the architecture parameters also affect the supernet training, with operations that exhibit advantages in the early stage of search being more prone to overfitting.
Based on the above observations, we further propose an operation-level early stopping (OLES) | method to address the domination of skip connections. The OLES method monitors each operation in the supernet to determine if it tends to overfit the training data and stops training the operation when overfitting occurs. Specifically,, we utilize the gradient matching (GM) method Hu et al. [2022] to compare the gradient directions of operation parameters on the training data and the validation data. If the difference between them remains large in multiple consecutive iterations, the operation will stop training. This approach not only alleviates the domination of skip connections and but also sharpens the distribution of architecture parameters with more search iterations, thereby reducing the performance gap caused by discretization. Moreover, method only needs to maintain the directions of operation gradients. Due to only minimal modifications to the original DARTS, the additional overhead is negligible.
In summary, we make the following contributions:
We analyze the robustness issue of DARTS from new perspective, focusing the overfit- ting of operations in the supernet.
We propose the operation-level early stopping method to address the domination of skip connections in DARTS with negligible computation overheads.
Extensive experiments demonstrate the effectiveness and efficiency of our method, which can achieve state-of-the-art (SOTA) performance on multiple commonly used image classifi- cation datasets.
'Open-source code be found at https:/gahhub.com/Pssaaabbbb

---

2 Background and Related Work
2.1 Neural Architecture Search
Over the years, variety of NAS methods have been proposed, including evolutionary- -based Termrit-
[hikun et al. [202T] Real et al. 2018), Guo et al. 2020]. reinforcement-leann wereeeett-aanning-bbee -baddd Zoph and Le
[. Pham et al.
2018 and gradient-basee
Liu et al. 2013). Xie et al. [2018] Cai et al. [2018]
methods. They have achieved great success in many computer vision (CV) and natural language
processing (NLP) So et al. [2019] tasks. As pioneering work, NAS-RL Zoph and Le [2016] uses an RNN controller to generate architectures and employs the policy gradient algorithm to train the controller. However, this method suffers from low search efficiency. To improve the search efficiency,
the cell-based miero search space Zoph et al. (2017) and weight-sharing mechanism |Pham et al. [0018] have been proposed. DARTS et al.] 2018], which also takes advantage of weight-sharing, utilizes the gradient- based method to optimize architecture parameters and model weights alterna- tively. Recently, train-free NAS methods Chen et al. [2021a]. et [2023] that utilize proxy metries to predict the ranking or test performance of architectures without training have been proposed to further promote search efficiency.
2.2 Preliminaries Differentiable Neural Architecture Search
We will now provide detailed review of DARTS. The original DARTS Liu et al. [2018) relies cell-based micro search space, where each cell contains nodes and E edges orgamized in a directed acyelic graph (DAG). Each node represents a feature map ,:()), and each edge is associated with an operation E O, where O denotes the set of candidate operations (e.g., skip connect, sep conv 3x3, etc.). During architecture search, instead of applying single operation to a specific node, continuous relaxation is applied to relax the categorical choice of a specific operation to mixture of candidate expío (f:2))
operations, i.c., (z) = EoEo Ed'oo exp(o Da o(x). = {a(t))) } serves as the architecture
parameters. DARTS then jointly optimizes architecture parameters a and model weights w with the following bi-level objective via alternative gradient descent:
min Lvol(w" (.)
S. t. W = arg min Lerain (w, a)).
Since DARTS follows the weight-sharing mechanism, we refer to the continuous relaxed network as the "supernet". At the end of the search phase, discretization phase is performed by selecting operations with the largest a values to form the final architecture.
2.3 Robustitying DARTS
Despite its simplicity and efficiency, the undesired domination of skip connections is often observed in DARTS. Previous works have attempted to climinate the domination of skip connections by diminishing their unfair advantages. DARTS
al. 2019 uses a fixed value to constrain the number of skip connections. Progressive DARTSChen et al. [0119 adds Dropout after each skip connection operation. DARTS -CCh et al. (220) introduces an auxiliary skip connection to separate the unfair advantage of the skip connection by analyzing it from the perspective of ResNer He et al. [2015). Another strategy is to remove the exclusive competition operation selection DARTS. FairDART&Chu et al. [209a] replaces the Softmax relaxation of DARTS with Sigmoid and offers an independent architecture parameter for each operation. NoisyDART&Chu and Zhang (2020) injects unbiased noise into the candidate operations to ensure that the good ones win robustly. However, DARTS+PT Wang et al. [2021] believes that the magnitude of the architecture parameter is not a good indicator of operation strength and proposes a perturbation-banl
architecture selection method.
Our work also related to GM-NAS et al. [2022), which falls the field of Few-Shot NAS Zhao et al. [021]. Although our method and GM-NAS both employ the GM score as mathematical tool, they are entirely distinct lines of work. GM-NAS argues that due to coupled optimization between child architectures caused by weight-sharing. One-Shot supernet's performance estimation could be inaccurate, leading to degraded search results. As result, GM-NAS proposes to reduce the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets.
3

---

Supomaaco Sep C
Skip Conneciion Sgy-Coar-3d 3d3
Valoo Aocaioottrr Patutatt
Conn-Sid
Skip Consectio
None
Valaco
3000
100000
15000
200003
25000
30000
30000
40000
110000
20003
90000
40000
Sxp
Stop
(a) The architecture parameters and validation losses of different operations. The solid line represents the value of architecture param- eler, while the dotted line represents the validation loss.
(b) The architecture parameters of dif- ferent parametric-free operations.
The motivational experiment to vefiry the domination of skip connections is due to the overfitting of operations.
Figure 1:
GM-NAS utilizes GM scores make splitting decisions, determining whether module should be shared among child architectures. The GM scores in GM-NAS are computed based on the gradient information of different child architectures problem of skip connection donimation from a totally new perspective and employ the GM score as an indicator for early stopping, preventing operation parameter overfitting. The GM scores, in our approach, are calculated using the gradient information of parameters on training and validation data.
shared parameters. In contrast, we aim to address the
3 Motivation and Methodology
In this section, we argue that the domination of skip connections is due to the overfitting of operations in the superett and demonstrate this hypothesis through a simple motivational experiment. Based on our observations, we propose an operation-level early stopping method to address this issue.
3.1 The Domination of Skip-Connection is Due to Overfitting
In DARTS, we observe that skip connections dominate due to the overfitting of operations in the supernet to the training data, causing them increasingly deviate from the distribution of the validation data. However, the architecture parameters are trained on the validation data, thus operations are selected based on their validation performance. As the number of training iterations increases, the parametric operations overfit more seriously, and the parametric-free operations gradually dominate.
To verify the above conjecture, we present empirical evidence on the relationship between the overfitting of operations the supernet and architecture parameters. We retain only architectural parameters on one edge in DARTS' search space and set all other architectural parameters to 0 to ensure that they will not be trained. In Figure [. we record the architecture parameter and the validation loss with the corresponding output of each operation in this edge. From Figure Tal we can observe that the validation losses of all parametric operations grow even faster as the training steps increase, and the architecture parameter gradually decreases, exhibiting an obvious negative correlation with the validation loss. Moreover, Figure Tbšhows that the skip connection easily obtains higher architecture parameters, exhibiting advantages over other parametric- operations. This is mainly due to the fact that the skip connection can always retain more information from the input.
Most existing works suggest that the domination of skip connections is due to their unfair ad advantage. To investigate this claim, we follow the settings of FairDARTS Chu al. [2019a] and replace the Softmax relaxation in DARTS with the Sigmoid relaxation. The unfair advantage of skip connections will be eliminated by removing the exclusive competition among operations. We then rerun the experiment and present the results in Figure?) 2 which shows that even after removing the exclusive competition, the architecture parameters still exhibit negative correlation with the validation losses, similar to that in Figure Ta It illustrates that the unfair advantage of skip connections is not the fundamental reason for their domination. Instead, it the effect of the overfitting of operations the supernet on the training of architecture parameters.

---

3.2 Operation-Level Early Stopping
Based on the above observations, we propose an operation-level early stop- ping method to alleviate the domina- tion of skip connections and enhance the robustness of DARTS. During the search phase, the architecture param- eters have an effect on the supernet training, resulting in different oper- ations with different architecture pa- rameters being trained differently. Op- erations that exhibit advantages in the early stage will get more attention and are more prone to overfitting. More- over, the operation is the basic unit of choice in DARTS, which selects operations to form the final architecture. Therefore, we perform early stopping for each operation. If early stopping be performed for each operation during the supernet training, when the operation tends to overfit, the overfitting problem can have less impact on the training of architecture parameters, and the supernet can also be fully trained.
210
Cometri DilOrends)
30
Figure 2: The architecture parameters and validation losses of different operations in FairDARTS.
Overfitting occurs when a model is too complex and fits training data distribution too closely, resulting in poor performance the validating data. When model is overfitting, the training loss continues to decrease while the validation loss increases, indicating that the optimization direction (i.e.. the direction of gradients) of the model parameters the training data will be inconsistent with that on the validation data. To identify overfitting, we propose using the gradient direction of the operation parameters. If the gradients the training and validation data differ significantly in direction, we consider the operation to tend to overfit the training data.
Algorithm 1 DARTS with operation-level early stopping
Require: Supernet weights w; architecture parameters ot number of search iterations /; number of consecutive iterations for GM M; threshold of GM score for verifying overfitting Ensure: Searched architectures.
Construct a supernet A:
2: for each i [1, do
3: Get batch of training data Btrain:
4: Update weights w by V Ltrain (w, a, Btrain);
5: Record the training gradient each operation Dtroin'
6: Get batch of validation data Brat;
7:
Update parameters a by VaLuai a, Buat);
8: Record the validation gradient of each operation Piati
9: Compute the gradient matching score over consecutive M iterations based on Eq.
10: Stop the gradient of an operation o if G. (Mfraii Poat) &
11: end for
12: Derive the final architecture based learned
Concretely, for an operation o, let P. denote the parameters of o, and Pirain and Peat denote the gradients of P. training batch and validation batch, respectively. We calculate the cosine similarity between Piaan and Poat to quantify the differences in the direction of the operations' gradients follows:
M COS(givain Phat)
GM Peat) m==1
M
Piaan Peal
where COS(grrain Peat)
(1)
1grain 11
Piain O ILtrain (w, O. Berain) g
0L (W, a, Bwat)
aP.
OP.
where Brrain and Bus denote batches of training data and validation data, respectively. The GM score is computed as the average cosine similarity between training gradients and validation gradients
5

---

over M consecutive steps. The gradient matching score here is similar to that in GM-NAS Hu et. [022]. When the GM score falls below predefined threshold of Ç E (0, 1) within M steps. the operation is considered to be overfitting. Consequently, if GM < E for an operation o, the parameters of this operation will not be updated during subsequent supernet training. And the remaining operations and architecture parameters will be trained as usual.
Through the operation-level early stopping method, the overfitting of operations in the supernet can be alleviated. Benefiting from the attribute that the DARTS search process involves alternating between training the supernet on the training data and training architecture parameters on the validation data, we only need to maintain the gradient of each operation on training data batch and a validation data batch in one iteration, with negligible additional computational overhead. As a result, OLES can alleviate the domination of skip connections, making DARTS more robust. This allows for training DARTS with more iterations to obtain better architectures and sharper architecture parameters, resulting in smaller discretization gap and improving performance. The algorithm for DARTS with operation-level early stopping can be found in Algorithm T
4 Experiments
4.1
Search Spaces and Experimental Settings
To verify the effectiveness of OLES, we conduct experiments on three different search spaces: the standard DARTS' search space, experimental settings the same as the original in each search space and only use the first-order optimization, our method only needs freeze the operation parameters during the supernet training. In addition to the standard hyperparameters in DARTS, our method introduces an additional hyperparameter, i.e., the overfitting threshold Ç, to stop operation training. We determine the threshold & by averaging the cosine similarity over 20 iterations for 30 randomly initiated architectures in each search space. And the gradient matching (GM) score is dynamically computed by averaging over every 20 iterations throughout the entire training process to reduce variance. Following Algorithm| [] we stop updating parameters of the corresponding operation o when GM (gtrain Peat) is lower than the threshold E.
and MobileNet-like search space. We keep all
4.2
Performance on DARTS' CNN Search Space
4.2.1
CIFAR-10
We conduct NAS CIFAR-10 using DARTS' search space and then transfer the searched archi- tectures to CIFAR-100. The threshold & is set to 0.3. As shown in Table [ our method OLES consistently outperforms other recent SOTA NAS methods. OLES achieves a top-1 test error of 2.30% on CIFAR-10 with nearly the same search cost as DARTS of 0.4 GPU-days. Moreover, the robustness of OLES is also guaranteed, as evidenced by the excellent average result of three independent runs. When the architectures searched on CIFAR- 10 are transferred to CIFAR- 100, our method still achieves a competitive test error of 16.30% This indicates that the architectures searched by OLES are transferable and expressive, and the architecture parameters do not overfit the CIFAR-10 dataset.
4.2.2 CIF
We conduct NAS on CIFAR-100, and the results are presented in Table 2 The threshold & is set to 0.4. OLES still exhibits significant advantages, improving the test error of DARTS from 20.58% to 17.3 17.00% without incurring extra search costs. It demonstrates that our method can adapt to datasets of different sizes and consistently improve the performance of DARTS. OLES not only consistently outperforms other methods that focus on the skip-connection domination issue, but also surpasses more sophisticated NAS methods (e.g., GM+DARTS Hu et al. [2022) on CIFAR-10 and CIFAR- 100 without requiring extensive code modifications or introducing additional computational overheads.
4.2.3 Transfer to ImageNet
To verify the transferability of architectures discovered by OLES, we transfer the best architecture derived from CIFAR-10 10 to ImageNet. Following the setting of DARTS, train the searched

---

Table I: Comparison with state-of-the-art NAS methods CIFAR-10, 10, and the transferred results on CIFAR-I00 100 are also reported. 9 reported by Chen et al. 2019 *: produced by rerunning the
published code. + transfer architectures discovered on CIFAR 10 to CIFAR-100.
NAS method
Test Em(()))
Params
Search Cost (GPU-days)
Search Method
CIFAR-10
CIFAR-100
DenseNet-BC Huang al.
2016
3,46
25.6
Manual
ResNet CutOa al
2015
4.61
17.8
Manual
NASNet-A CurOut Zaph
2017
2.65
1800
RI.
AmochaNct-A Real è 01
2018
3.34 0.06
3.2
3150
Evolution
AmechaNet-B
2,55
3150
Evolution
PNAS
2017
3.41 + 0.09
3.2
225
MDL
ENAS
2018
RL
DARTS(Is() + CutOul Läu
2018
3.00 1. 0.14
17.76
3.3
0.4
Gradient
DARTS(2nd1 CurQuiraleer
2.85 0.08"
Gradient
P-DARTS al 2010
2,50
16.55
3,4
0,3
Gradicnt
R-DARTSU 25 2/
2019
2.05
Gradient
FairDARTS Chirct
2019
2.54
2.8
0,4
Gradicnt
DARTS-
4020
2.50
0.13
Gradient
SGA
2020
2,39
3.8
0,25
Gradicnt
DARTS Wang cl
2021
2.48/2.61 0.08)
19.05
3.0
0.8
Gradient
DINAS Chen
2.46 1 0.03
4.1
0.6
Gradient
8-DARTS al 4122
2.53 + 0.08
16.24 + 0.22
3.75
0.15
0.4
Gradient
GM-DARTST
2022
2.35
16.45
1.1
Gradient
OLES
2.30 0.11)
16.30016.35 0.05)
0.4
Gradient
Table 2: Comparison with NAS methods on CIFAR- 100. 9 reported by Dong and Yang 2019
reported by Zela et al. 2019]. *: produced by rerunning the published code.
Params
Search Cost (GPU-days)
NAS method
Test Err (%)
Search Method
(M)
ResNet CueOu al. 2015
10%
1.7
Manual
PNAS Liu et al.
2017
19.53
3.2
150
MDL
ENAS CutOut Pham al. 2018
19.43
4.6
0.45
RL
DARTS Liu al. 20188
20.58 0.44*
3,4
0.4
Gradient
GDAS Dong and Yang
2019
18.38
3,4
0.2
Gradient
P-DARTS Chen et 2019
17.46*
3.6
0.3
Gradient
R-DARTS/L2
2019
18.24
1.6
Gradient
DARTS- Chu èt al.
2020
17.51 + 0.25
3.3
0.4
Gradient
PR-DARTS Zhou al.
2020
20.10"
3.6
0.17
Gradient
DARTS+PT Wang 3-DARTS Ye ct al. 2022
2021
18.78*
3.4
0.8
Gradient
17.33*
3.83
+ 0.08
0.4
Gradient
GM+DART et al.
2022
17.42
3.6
1.3
Gradient
OLES
17.30
3.4
0.4
Gradient
architecture from scratch for 250 epochs, and the results are presented in Table 3 achieves 24.5% top-1 test Compared to other NAS methods, the architectures derived from OLES still exhibit competitive performance. Notably, we do not introduce any additional training techniques such as SE modules and Swish, but simply follow the original DARTS setting transfer the architectures. Therefore, the performance of OLES on ImageNet can be further enhanced by incorporating these training tricks.
Our method
4.3 Performance on NAS-Bench-201
NAS-Bench-201 Dong and Yang 2020) provides a unified benchmark for analyzing various up-to- date NAS algorithms. contains 4 internal nodes with 5 operations (i.e., Zero, Skip Connection, 1x1 1 Conv, 3x3 Conv, 3x3 AvgPol). NAS-Bench-201 offers a similar cell-based search space that comprised total of 15625 unique architectures. The architectures CIFAR-10 10, CIFAR- and ImageNet16 120). The comparison results are shown in Table and 0.7. DARTS performs extremely poorly on NAS-Bench-201.. We suppose
trained on three datasets (i.e..
the threshold ş is that it is due to the fact that the search space of HAS-Bench-200 is extremely compact, contains few parametric operations, and the number of operation parameters varies widely, which makes it difficult for DARTS to adapt to NAS-Bench-201. OLES significantly outperforms DARTS on all datasets and competitive among other NAS methods. It demonstrates the generality of OLES, which can adapt to different search spaces and datasets.
7

---

Table 3: Comparison with NAS methods on ImageNet. o: reported by Chu al. 2019a]. Other results are from their original papers. DARTS- and NoisyDARTS- A are directly searched on ImageNet and use SE modules and Swish improve performance. Other methods in the bottom block are searched on CIFAR-10 and then transferred to ImageNet.
Test Err (%)
Params (M)
Search Cost
NAS method
Search Method
Top-1
Top-5
(GPU-days)
Inception-v/ Szegedy et al. 20155
30.2
10.1
6.6
Manual
MobileNet V2 Sandler ct al. 2018
28.0
3.4
Manual
ShuffleNet Zhang al.
2017
26.3
Manual
NASNet-A Zoph
al.
2017
26.0
8.4
5.3
1800
RL
AmochaNet-A Real et al.
2018
25.5
8.0
5.1
3150
Evolution
AmoehaNet-BB Real et al. 2018
26.0
8.5
5.3
3150
Evolution
PNAS Liu al. 2017
25.8
8.1
5.1
225
SMBO
MnasNet-92 Tan et al.
2018
25.2
8.0
4.4
1667
RI.
DARTS((nd) + CutOur Liu et al.
2018
26.9
8.7
4.7
0.4
Gradient
SNAS Xie et al. 2018
27.3
9.2
4.3
1.5
Gradient
GDAS:Dong
Yang
2019
25.0
8.5
5.3
0.2
Gradient
P-DARTS Chen et al
2019
24.4
7.4
5.1
0.3
Gradient
PC-DARTS Xu el al. 2019
25.1
7.8
5.3
3.8
Gradient
FairDARTS Chu
201
24.9
7.5
4.8
Gradient
NoisyDARTS A Chu and
Zhang
2020
22.1
6.0
5.5
12
Gradient
DARTS- Chu et al.
2020
23.8
6.1
5.5
4.5
Gradient
DARTS+PT Wang al.
2021
23.5
8.0
4.6
0.8
Gradient
DINAS Chen el al.
2021
23.7
7.1
5.7
4.6
Gradient
B-DARTS Ye el al.
2022
23.9
7.0
5.5
0.4
Gradient
GM DARTS(2nd)
Hu et al. 2022
24.5
7.3
5.1
2.7
Gradient
OLES
24.5
7.4
4.7
0.4
Gradient
Table 4: Results on NAS-Bench201.
NAS method
CIFAR-10
CIFAR- 100
ImageNet16-120
valid
lest
valid
test
valid
test
DARTS(2nd) Liu
15.03
SNAS Xie et ar 2013
90.10 1.044
92.77
1: 0.83
69.69
2.39
69.31
1.98
42.81
1.79
43.16
2.61
DSNAS
2010
89.96 0.29
83.08
0.13
30.87
1: 16.40
31.01
1: 16.38
40.61
0.09
41.07
0.09
PC-DARISİXU ol
2019
89.96 0.15
93.41
0.30
67.12
1: 0.33
67.48
0.89
41.83
0.08
11.31
1 0.22
DARTS Chir X020
91.03 0.44
93.48
1 0.40
71.36
1: 1.51
71.53
1: 1.51
44.87
1.46
45.12
0.82
iDARTS
2012
89.86 0.60
93.58
RLNASirantin lihel Zhang
2021b
89.94 0.00
93.35
0.00
70.98 1: 0.00
70.71 4: 0.00
43.86
0.00
43.70
0.00
DiNAS Chen al 2021b
91.36 0.00
94.36 0.00
73.49 +: 0.00
73.51 + 0.00
46.37
3:0000000 0.00
46.34
0.00
3-DARTS
73.49
0.00
GM + DAKTSHo Her 2022
91 03 0.24
93.
0.12
71.61
0.62
7L.83 0.97
42.19 0.00
42.00 0.00
OLES
90.85 0.10
83.70 0.15
70.56
0.28
4.4 Performance on MobileNet-like Search Space
As described in FairDARTS Chu jet al.] [2019a]. the domination of skip connections also exists in the MobileNet-like search space. The su- pernet is built MobileNet Sann- dler et al. 20188 and comprises 2T choice blocks, each with candidate operations. As the MobileNet search space is not naturally designed for DARTS, we make minor modifica- tions to the MobileNet search space as FaiTDARTS and perform NAS on ImageNet dataset. The threshold Ç is set to 0.4. The results are summarized in Table and the corresponding searched architecture visualized Appendix [.T]] We can observe that OLES can also achieve highly competitive performance in the MobileNet search space.
Table Results on ImageNet using the MobileNet-like search space.
method
Top
Top-5
Params(M)
Ploxyless
2018
24.9
7.5
7.1
FBNer-C Wu 2018
25.1
7.9
4.4
FairNAS-A
2019b
24.7
4.6
FairDARTS
20T9a
24.4
4,3
RLNAS RLNAS 202 lb
24.4
7.4
5.3
CMMPPoxyeesNAS Hu et 2022
23.4
7.0
4.9
OLES
24.7
7.6
4.7

---

In-Depth Analysis
5.1
Different Gradient Direction Similarity Measures
Table 6: Test accuracy(%) with different gradient direction similarity measures on NAS- 20S-
Dataset
Cosine Similarity
Tn Distance Direction Distance
Hard Threshold
CIFAR-10 93.77
92.35
90.59
93.65
CIFAR-100
70.11
69.39
67.40
70.19
ImageNet
44.23
42.38
38.28
43.86
use cosine similarity to measure the difference in the operation's 's gradient directions between the training and validation data. To examine the effect of different gradient direction similarity measures for computing the gradient matching score, we compare cosine similarity with three other measures: L2 distance, direction distance, and hard threshold. The mathematical definitions of the other three measures distance directly computes the fraction of elements that have different directions in two gradients. And the hard threshold does not use the averaged value of M iterations compute the GM score but instead directly checks whether the direction distance can consistently be greater than a threshold in consecutive M iterations. As shown Table 6 cosine similarity performs better than the other three methods. Therefore, we adopt cosine similarity as the gradient direction similarity measure of OLES.
As described in Section 3.2.
be found in Appendix [ The direction
5.2 Gradient Matching Score
23
(a) Operations' cosine similarity and test accuracy on CIFAR-I0.
(b) Operations' cosine similarity and test accuracy on CIFAR-I00
(c) Operations cosine similarity and test accuracy on CIFAR-10 (NAS- Bench-201).
Figure 3: Trajectories of the cosine similarity of parametric operations and test accuracy during the training of DARTS.
In Figure 3 we present the trajectories of consine similarity between gradients on training data and validation data during the DARTS training. With the training process of DARTS, the cosine similarities of parametric operations gradually decrease, which indicates that the cosine similarity can reflect the overfitting of operations. And different parametric operations show similar trajectories. Consequently, set an unified GM score threshold for all parametric operations. Moreover, when the number of training iterations continuously increases (>50), DARTS exhibits a noticeable decline in test performance. Figure 3e further corroborates this finding, the problem of performance collapse due to the skip connection domination issue more easily appears in NAS-Bench Bench-2011 2011. As a result, it manifests the test performance declines an earlier stage.
5.3 Training With Longer Epochs
As mentioned in Bi al. (2019) and Chu al. 2222). training DARTS with longer epochs enables a better convergence of the supernet and architecture parameters. However, as shown in Figure 4a
and 4bb] training standard DARTS with longer epochs exhibit serious performance collapse with the domination of skip connections. The skip connection rapidly dominates after 50 epochs on CIFAR-10 and takes longer epochs on CIFAR- 100. And we also find that other methods (e.g., FRRSDARTS, and DARTS-PT) fail with longer training epochs as well. In contrast, as shown in Figure 4c and 4d OLES survives even after 200 epochs. The number of skip connections in OLES stays stable after a specific number of epochs since most parametric operations are frozen. It demonstrates that OLES is robust and can actually address the domination of skip connections. More experimental results and searched architectures with longer training epochs can be found in Appendix C.2 and Appendix D.2

---

()) DARTS on CIFAR-10.
(b) DARTS on CIFAR-100.
(c) OLEE on CIFAR-10.
(d) OLEES om CIFAR- 100.
Figure 4: Trajectories of the numbers of skip connections in the normal cell and the reduction cell on CIFAR and CIFAR-100.
5.4
Comparison of Kendall Rank Correlation Coefficients
Kendsl Dayed
Kondsl Oh
87.
17.1
37.
Tot 33.37
10.77,9100
197.7 57 39
168
357
37.3
Search Accumra
Search
(a) DARTS (2nd Order)
(b) OLES (ours)
Figure 5: Comparison of search-retraining Kendall rank correlation coefficients т. The results are derived from 10 randomly selected architectures from single run of DARTS and OLES.
Table Kendall-7 coefficients architecture ranks based on searching and retraining metrics. sample 4 excellent architectures in 4 rounds. Other methods randomly select 10 architectures from single run.
RandomNAS
DARTS
GDAS
SGAS
OLES
0.0909
0.13
-0.1818
0.42
0.41
Figure 5 indicates that the early stopping mechanism does not hurt the ranking performance of DARTS. Additionally, we compare the Kendall rank correlation coefficients with other NAS methods, including RandomNAS, DARTS, GDAS, and SGAS, as shown in Table 7
Notably, the Kendall coefficient of OLES aligns closely with that of SGAS Li et al. [020]. which aims to alleviate the effect of the degenerate search-retraining correlation problem. The results demonstrate that by mitigating operation parameter overfitting, differentiable NAS is able focus the potential of architectures themselves, thus enhancing the correlation between search metrics and the architectures discovered.
Conclusion
This paper attempts find the fundamental reason for the domination of skip connections in DARTS from the new perspective of overfitting of operations in the supernet. We verified our conjecture through motivational experiments and proposed the operation-level early stopping (OLES) method to solve the domination of skip connections. OLES monitors the gradient direction of each operation and determines whether is overfitting based on the difference between the gradient directions on the training data and the validation data. OLES can solve the domination of skip connections with negligible computational overheads without imposing any limitations. Moreover, OLES achieves state-of-the-an results of 2.30% test error CIFAR-10 10 and 17.30% test enror CIFAR- 100, meanwhile reporting superior performance when transferred ImageNet and on search spaces of 2AS-Bench-201 and MobileNet well.
10

---

Acknowledgment
This work was supported by the National Natural Science Foundation of China (#62102177), the Natural Science Foundation of Jiangsu Province (#BK20210181), the Key R&D Program of Jiangsu Province (#BE2021729), and the Collaborative Innovation Center Novel Software Technology and Industrialization, Jiangsu, China.
References
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.0157,, 2016.
Chakkrit Termrithikun, Yeshi Jamtsho, Jirarat leamsaard, Paisarn Muneesawang, and Ivan Lee. Eeea-net: An early exit evolutionary neural architecture search. Engineering Applications of Artificial Intelligence, 104:104397, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search, 2018. httl:/tttts/////tii http:///w/wwoww../abbbbboe
org hargg/bb//te 11.802.01545
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018. htt.::ttts://axii wwwp:/r./g///aaaee/e wrgg/ab//80/..32222222 1atp:/1802.33666.
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 82-92, 2019.
Guanghui Zhu, Wenjie Wang, Zhuoer Xu, Feng Cheng, Mengchuan Qiu, Chunfeng Yuan, and Yihua Huang. Psp: Progressive space pruning for efficient graph neural architecture search. 2022 IEEE 38th International Conference Data Engineering (ICDE), pages 2168-2181, 2022a. doi: 10.1 10111//DDE3375522222200200
Guanghui Zhu, Zhennan Zhu, Wenjie Wang, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. Autoac: Towards automated attribute completion for heterogeneous graph neural network. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 2808 28281, 2023. doi: 10. 1109/CCDE55515.2223.000
Jingfan Chen, Guanghui Zhu, Haojun Hou, Chunfeng Yuan, and Yihua Huang. Autogsr: Neural architecture search for graph-based session recommendation. In Proceedings of the 45th Inter- national ACM SIGIR Conference Research and Development in Information Retrieval, page 1694-1704, 2022.
Guanghui Zhu, Feng Cheng, Defu Lian, Chunfeng Yuan, and Yihua Huang. Nas-ctr: Efficient neural architecture search for tlick-through rate prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 332-342, 2022b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang, Darts: Differentiable architecture search, 2018.
URL
www.ariwi..i.oroeooooeooe abs/10/s/660/ 09055
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in artificial intelligence, pages 367-377. PMLR, 2020.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv: 0902.9002000 0.00200/44 /22, 2019.
Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. Advances neural information processing systems, 32, 2019.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild. 2019. URLhtttps //tr//aaa //trxiv..vv////////a//eaae/aaavv org/ abs/ /abb///91..19555

---

Hanwen Liang. Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping, 2019. URL abs/ 06035
https://arxiv.ott www.aaiwwwo.ab//////e.ee
Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. Darts-: Robustly stepping out of performance collapse without indicators, 2020. URL htt.btttpp::aattttee wwwpriiiw...rrooeoooooeee abs/ 22009. 01027
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search, 2019. URL atttps///axxx org/abs/1909 09bbb//999900966666
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable nas, 2021. URL httpt:ttttt::...ooooto h/tg//ab///// 2108 04392
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search, 2019a. URL httptt:::/aiiiv.org//aaaaaa 1911 1911.12126
Xiangxiang Chu and Bo Zhang. Noisy differentiable architecture search, 2020. URL. https
//arxiv. org/ abaaaba//000/ 03566
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL https://arxiv..rrr org/ abs/ 11///1111.00 03385
Sirui Xie, Shoukang Hu, Xinjiang Wang, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Understanding the wiring evolution in differentiable neural architecture search. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, URL https: https://prooceddnng mlr hAlrppesss/v10/
pages
PMLR, 13-15 Apr 2021.
xie21a. html
Shoukang Hu, Ruochen Wang, Lanqing Hong, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. General- izing few-shot nas with gradient matching, 2022.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision, pages 544-560. Springer, 2020.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.009966 2018.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv: /0l:::/12.0033 2018.
David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning, pages 5877-5886. PMLR, 2019.
Barret Zoph, Vijay Vasudevan, Jonathon Shiens, and Quoe V. Le. Learning transferable architectures for scalable image recognition, 2017. URL http:::////axiiv..orr//a//////// 07012
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: theoretically inspired perspective. arXiv preprint arXiv:2102.11533 202
Guihong Li, Yuedong Yang, Kartikeya Bhardwaj. and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation on gradients. arXiv preprint arXiv:2301. 200.0000000 2023.
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In International Conference on Machine Learning, pages 12707-12718. PMLR, 2021.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2016. URL htRp::ts::://eet http///axvvvv.. org/ abs////a///1000 06993
12

---

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search, 2017. URL
alwlg//bb//11.....m. 00059
G. Li, G. Qian, I. C. Delgadillo, M. Muller, A. Thabet, and B. Ghanem Sgas: Sequential greedy architecture search. In 2020 JEEF/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pages 1617-1627, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10. 11090CVPR42200.2220.. 201220 CVPR42600 200...002000000
URI. otteo//oototeeeeooeeoeeeeeeoooor
org/ 1109/
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search, 2021b.
Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. 3-darts: Beta-decay regularization for differentiable architecture search, 2022. http:tts: /atpvv////vvvv//. www.:/rrg/bbb////ee 01665.
Xuanyi Dong and Yang. Searching for a robust neural architecture in four gpu hours, 2019, URL https://arxiv..
19465
04465
Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Theory-inspired path-regularized differential network architecture search. CoRR, abs/2006. 16537, 2020. URL https://arxtt
org/abs/2000 16537
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015. htttps/////ax
1512.00567
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen, Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2018. URL https://arxiv.orrt
1801. 04681
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017. URL htttps:////a http//aaiiv.orgab//////a///////
01083.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-ave aware neural architecture search for mobile, 2018. URL https
//arxiv. , org/ abs/abb/. 11626
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficient architecture search, 2019. URL
https://arxiv. org/ abs/ http//bb/1990000555 1977.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural ar- chitecture search. In International Conference on Learning Representations, 2020. URL
https://openreeveuu wwwpaarwwwiiie.eetoooorr.......o
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: direct neural architecture search without parameter retraining. CORR, abs/2002.09128, 2020. htt.tttss://
org/ abs/ abs/ 2002.00020000 09128
Miao Zhang, Steven W. Su, Shirui Pan, Xiaojun Chang, Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. CoRR, abs/2106. 10784, 2021a. URL http:t:tt://araa wttp:/wwwwwrrg///////e////// 10784
Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. CoRR, abs/2101.11834. 2021b. httpps/////a .oreiiivvorggeoo abtp//bb./10..... 11834
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CoRR, 234/18223333 2018. URI, (ttp://tt://arxi htggabss18112 13444033433 03443
13

---

Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. CoRR, 0///1907001845 2019b. bttt://a/ar org/abs/ 1100/ 19077018450050
Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing darts with amended gradient estimation on architectural parameters, 2019. URL BtLhhttss:// // //ax/axvvvoo.// org/ abs 0bss1911 /9100 11831
14

---

A The Selection of Parametric-Free Operations on Various Search Spaces
Zela ct al. 2019 designed four variants of the DARTS' search space (S1-S4) and pointed out skip connections tend to dominate compared with other parametric-free operations, i.e., zero and noise. To investigate whether the robustness issue in DARTS arises from the domination of skip connections rather than other parametric-free operations,
extend the experimental settings in Zela ct al. 2019] to seven search spaces, including four types of parametric-free operations( avg_pool__ 3x3, kiip_connect, noise, zero). These search spaces only differ from DARTS' original space in the number and types of operations on each edge. We follow the same experimental setup in Liu al. 2018 and execute DARTS on each search space. Figures O and I3 15 display the results on S1-S3. We observe that skip connections tend to dominate more severely than other parametric- free operations. By designing different combinations of parametric-fre operations S4-S7 search spaces, Figures 9 12 and [9 show that skip connections also dominate under fair competition with other parametric-fre operations when the searched cells contain more than one type of parametric-fres operations. The corresponding performance for each search space is shown in Table 8 As result, the domination of skip connections the primary cause of the shallow architecture DARTS with poor generalization and robustness.
SI consists of two operations: skip_connee and sep_conv_3e xx3.
S2 consists of two operations: avg_pool_3x3 and sep_conv_3x3
S3 consists of two operations: noise and sep_conv c33._333. The noise operation replaces each input value with random Gaussian noise for NO0, Differentiable NAS algorithms generally avoid this operation because it hurts the performance of discretized architectures.
S4 consists of three operations: skip_connect noise, and cop__oov_3x3.
S5 consists of three operations: skip_connect, avg_pool_33x and _ap_c_ov333_3
S6 consists of three operations: skip_ stip_connet,, zero, and sep_conv com3_3x3. The zero operation replaces each input value with zeros.
S7 consists of five operations: skip_ connect, avg_ frs_pool_33 3x3, noise, zero, and sep_conv 3x3.
Table Performance of DARTS on search spaces S1-S7.
Search Space
CIFAR-10
CIFAR-100
SI
3,28
26.05
S2
2.55
25,40
S3
3.69
24.70
S4
3.05
21.35
S5
3.03
23.77
S6
2.84
27.90
S7
2.70
22.17
CORMNT
aex 253
KJ
(a) Normal Cell
(b) Reduction Cell
Figure 6: Normal and reduction cells found by DARTS on CIFAR-10 in search space $1.
15

---

onmy 3x3
Mpp only
KpLase 363
peel
C.
AspLesmr__ 3x3
c.
oug pael 3x3
ing_poo
3xx
ong pool 3x3
(a) Normal Cell
(b) Reduction Cell
Figure 7: Normal and reduction cells found by DARTS CIFAR-10 in search space S2.
sep sep_comv conv
scp conv
sep 3x3
[k-2]
scp_ comy. 3x3
c_(k]
sep_comv
c_ [kl
6_
sep_ cony
nose
scp. 3x3
scp_com 363
sep_ conv
3x3
(a) Normal Cell
(b) Reduction Cell
Figure Normal and reduction cells found by DARTS on CIFAR- 10 search space S3.
skip
sep_2007: 333
skipuooemazt
S 21
stip coonectt
upp oow
c.
skip
6.0k]
Skip connect
c [b-3]
Xpposom 3i3
ck 11
kkp cornect
cone 3x3
(a) Normal Cell
(b) Reduction Cell
Figure Normal and reduction cells found by DARTS on CIFAR-10 10 search space $4.
303 4bt.cme
(a) Normal Cell
(b) Reduction Cell
Figure 10: Normal and reduction cells found by DARTS on CIFAR-10 in search space S5.
16

---

10.00
CJKI
xp AeP.
(a) Normal Cell
(b) Reduction Cell
Figure 11: Normal and reduction cells found by DARTS CIFAR-10 in search space S6.
sep_ con_cony 3x3
skip connect
scp Lww_ooov____ _3x3
c_ [k
skip
(k)
nutss
skip contiss
c_
scp ccp_conv__oo con__3x33
c_
cony
conmest
c.
scp wwl.cooy_cco ____oonvv33
sep Sgg_no0..33
skip contect
scp wwl.coov__aa _o3
F [b-
Normal Cell
(b) Reduction Cell
Figure 12: Normal and reduction cells found by DARTS on CIFAR-10 in search space S7.
36
Sy
(a) Normal Cell
(b) Reduction Cell
Figure 13: Normal and reduction cells found by DARTS on CIFAR- 100 in search space $1.
(a) Normal Cell
(b) Reduction Cell
Figure 14: Normal and reduction cells found by DARTS CIFAR-100 in search space $2.
17

---

sep vony
scp_ cony 3x.3
(_-22
scp
sep_conv 3x3
sep 3x3
c_(kk
sep_stem_.0
c [k11]
scp ecp_con _33
c0L J13
coBY 3K3
sep _e__ooovv_33
3x!
c.
sep 2ony_oov__33
40p man
Normal Cell
(b) Reduction Cell
Figure 15: Normal and reduction cells found by DARTS on CIFAR- 100 in search space S3.
sop_comv
e [1-2]
dtipeconeet
poise
sep scpo
neise
skmp_comect
noise
compe
c_|1-
comeco
Ikki
sorr 333
(a) Normal Cell
(b) Reduction Cell
Figure 16: Normal and reduction cells found by DARTS on CIFAR- in search space S4.
(a) Normal Cell
(b) Reduction Cell
Figure 17: Normal and reduction cells found by DARTS CIFAR- 1IFAR-100 in search space S5.
seip commect
stp cone 3x3
sep_somm_333
skip
conY 3x3
2k
sop_somy 3x3
cone 163
sep_com 3x3
spp cone
c.lk]
spp _03
c.lk]]
sppsomw_3x
sep_conv_ 3x3
sp_comi_11
up crmy
(a) Normal Cell
(b) Reduction Cell
Figure 18: Normal and reduction cells found by DARTS CIFAR-100 in search space S6.
18

---

sep vuny
skip connect
skip connect
[k-2)
skip connect
Sip_ronned posnee
skip commect
c_(kk
c_ k_
sep coey 363
skip skipeon
coty Scc
c_|k=|
sep
ssp_sney_5dd 3x3
skip connect
spp. Lunv
skip comnect
Rukk 11
sep. SoLeco
up cone
(a) Normal Cell
(b) Reduction Cell
Figure 19: Normal and reduction cells found by DARTS on CIFAR- 100 in search space S7.
B Gradient Distance Similarity Measures
In Section 5.1 four gradient distance measures present. Next, we provide comprehensive mathematical definition for each measure. Let Prain and goat denote the gradients of the operation parameters P. on training batch and validation batch, respectively.
The GM score with L2 distance can be formulated as follows:
EmE
M
I Peutll2
GM PErain Peal)
(2)
M
And the GM score with direction distance can be formulated follows:
M
Pallo
GMäirentio (Birains Poat)
(3)
M
where @ denotes the element-wise product, and I1g @ Phat < O|l1 represents the number of elements whose directions are different.
However, the hard threshold method does not use the averaged gradient distances over M steps to calculate the GM score. Instead, it directly checks whether the direction distance can consistently exceed a threshold over the consecutive steps. Let & denote the threshold. The hard threshold method checks whether I1 00% & can still be satisfied for M consecutive steps.
P.llo
C Supplementary Experimental Results
C.1 Quantitative Analysis on Negative Correlation between Validation Loss and Architecture Parameter
The results of our motivation experiments. as depicted in Figure D indicate a negative correlation between validation loss and architecture parameter. To further provide a more rigid analysis, we cal- culate the Kendall-correlations between validation loss and architecture parameter for each operation and present the results in the Table It can be observed that there is clear negative correlation between validation losses and architecture parameters, particularly for parametric operations.
Table Kendall-T correlations for architecutre parameters and validation losses of different opera- tions in Figure Ta
Skip Connections
Sep-Cone Sep-Conv-3x3
Sep- Sop-Conv-5x5
Dil-Conv-3x
-0.36
-0.59
-0.73
-0.72
19

---

C.2 Results with Longer Training Epoches
DAVIS-T
33-
33
Rachh
Figure 20: Test accuracy of searched architectures CIFAR-I0 with longer search epochs.
BATTS
10 075000
(a) Normal Cell
(b) Reduction Cell
Figure 21: The number of skip connections in searched architectures on CIFAR-10. Figure 21b there are some overlaps among the curves of DARTS and OLES.
Despite previous attempts to investigate the origins of the skip connection domination issue and its corresponding solutions, remains open problem. These methods have put forth their hypotheses but have not fully resolved the issue. To further exemplify the efficacy of addressing the skip connection domination issue, we increase the number of iterations of supernet training. As depicted in Figures 20 []] DARTS-PT paans et al. (2021) fails to address the skip connection domination issue when extending to very long training epoches, whereas OLES remains stable.
C.3 Hyperparameter Analysis
In OLES, the gradient matching score is dynamically computed by averaging over every 20 iterations throughout the entire training process. The threshold & is determined based on the GM scores within the initial 20 iterations. We conduct experiments to assess various iterations along with their corresponding thresholds across different search spaces to explain the rationale behind selecting 20 iterations. The results, as shown Table [0] demonstrate that computing the GM score using an average across every 20 iterations yields the highest accuracy.
C.4 Results on DARTS S1-S4
DARTS S1-$4 are simplified variations of the DARTS search space proposed by R-DARTS Zela al. [019] investigate DARTS' failure mode. In Appendix [A] we have further expanded these search spaces and presented DARTS' failure mode (both in terms of performance and the searched architectures) in these extended search spaces. In Table TT we additionally provide performance comparison between OLES and other methods DARTS S1-S4. The results demonstrate that OLES also consistently outperforms DARTS, well as R-DARTS Zela et al. (2099) and DARTS-PT Wang al. [2021 which also dedicated to addressing the skip connection domination issuc.
20

---

Table 10: Performance on various search spaces using different numbers iterations calculate the
GM scores.
DARTS Search Space CIFAR-10
MAS-BENCH-AA Search Space CIFAR-10
Iterations
CIFAR-100
CIFAR-100 ImageNet16- 1-0
10
97.33
80.83
80.57
47.93
26.29
20
97.70
82.70
93.89
70.75
44.38
30
97.05
82.39
91.93
66.97
38.87
40
97.19
81.32
89.76
64.69
32.73
Table 11: Performance comparison on the DARTS S1-S4 search spaces.
Datasets
Spaces
DARTS
R-DARTS
DARTS-PT
OLES
S1
3.28
3.31
2.79
2.76
CIFAR-10
S2
2.55
2.44
2.47
2.43
S3
3.69
3.56
2.64
2.57
S4
3.05
3.05
2.92
2.73
S1
26.05
22.24
23.16
22.10
CIFAR-100
S2
25.40
23.34
22.10
22.18
S3
24.70
21.94
20.80
20.70
S4
21.35
20.70
19.98
19.18
C.5
Comparison with Train-Free NAS Methods
Table 12 compares OLES with two recently proposed train-free NAS methods, TE-NAS Zhao et al. 2021 and ZiCo Li et al. [023]. It is evident that OLES still outperforms train-free methods in most cases. While train-free methods exhibit slightly superior performance on NAS-Bench-201 due to its compact search space, in larger search spaces such as the DARTS search space, OLES performs better than TE-NAS and ZiCo. Train-free NAS a highly promising research direction, as not only greatly enhances NAS efficiency but also promotes deeper understanding of neural networks.
Table 12: Comparison with TE-NAS and ZiCo. The test errors are reported.
NAS-Bench-22
DARTS Search Space
ImageNet
CIFAR-10
CIFAR-100
16-120
CIFAR-10
top-1
top-5
TE-NAS
6.1400.7
78.56
57.62-10.46
2.63400066
26.2
8.3
ZiCo
6.0+0..4
28.9
58.20-10.3
2.45+00.11
OLES
6.310.15
29.60+1.22
56.03 + 0.38
2.41 + 0.11
24.5
7.4
D Searched Architectures
D.1 Architecture on MobileNet-likee Search Space
Following the experimental settings in Chu et al. 2019a modify the MobileNerV2 Sandler et al. [2018) search space and perform OLES on ImageNet this search space. The searched architecture illustrated in Figure 22
21

---

MB3 x
MB3 3
MS2 a 8
MBs 3
MB6 a 8
MBS 3
MBS 3
MB6 x
MBS x
MB3 7
MBS 3 x
Iden
MB3 5
Identiy
MBB 3
MB3
Figure 22: The architecture searched on the MobileNet-like search space.
D.2 Training With Longer Epochs
We train OLES and several other differentiable architecture search methods with 200 epochs to verify whether they are robust enough in this case. According to our conjecture, as the number of training epochs increases, the operations in the supernet become more prone to overfitting, leading more serious issue of the domination skip connections. In this section, we present some additional results of the searched architectures using different numbers of training epochs.
D.2.1 Results on OLES
As described in Section 5.3 OLES survive with longer training epochs. Specifically, the number of skip connections in OLES keeps stable even when the number of training epochs reaches 150 or 200. The corresponding architecture genotypes on CIFAR-I0 and CIFAR-100 are visualized in
Figures 23 26
avg_
Lon 515
ag poot
[--]
dRLeomy_363
stip connies
ssp_ramv_3t3
2
skip sumee
stp _o3___o__
c.
c_lk]
ay puol
stip connect
wpp care.
5x5
(a) Normal Cell
(b) Reduction Cell
Figure 23: Normal and Reduction cells discovered by OLES on CIFAR-10 for 150 epochs in the standard DARTS' search space.
diLenmy_5e
[-1]
URLoonv_313
conãess
2 mus paol 363
spp. _x3
C.
aod 363
<ipp. ponnect
k.lkk
pool 3t
1--2
ssp_romv_3ts
3x3
spp comy
(a) Normal Cell
(b) Reduction Cell
Figure 24: Normal and Reduction cells discovered by OLES on CIFAR-10 10 for 200 epochs in the standard DARTS' search space.
22

---

[1+1]
dil Trs
skip_come
2.
sip_counest
c.
skip_someet
pupp h2
(I-21
2D cp_comw.uu 3x3
skip contant
[b1]
(a) Normal Cell
(b) Reduction Cell
Figure 25: Normal and Reduction cells discovered by OLES on CIFAR-100 for 150 epochs in the standard DARTS' search space.
3x3
sqg_pool_s3
[K-1]
diLsun
skip
pool Se3
Hip_stmnes
skip pornect
Stp_comme connect
[k-3]
3k_ 3k3
skip contect
C (b-1
(a) Normal Cell
(b) Reduction Cell
Figure 26: Normal and Reduction cells discovered by OLES on CIFAR-100 for 200 epochs in the standard DARTS' search space.
D.2.2 Results on FairDARTS
FairDARTS Chu et al. [019a] adopts the Sigmoid instead of the exclusive Sofimax for architecture parameters and selects operations through a threshold value. However, our experiments show that no operation achieve the pre-defined threshold value in the normal cell when FairDARTS is trained for 150 or 200 epochs. Following FairDARTS' settings, we train it for 50, 150, and 200 epochs. Table 13 shows the architecture genotypes, which can be visualized in Figure 27 and Figure 28
Table 13: Architecture genotypes found by FairDARTS on CIFAR- 10 for 50, 150 and 200 epochs, respectively. Epochs
Architecture genotype
Genotype(normal-((xepp.. conv 3x3', 2, 0), ('dil conv 5x5', 2, 1), ('dil_ "dil_conv___
3x3', 3, 0), ('dil_ccom) [an_3333',,_0 3, 1), ('sep conv _55___x5'_____ 4. 1)1, normal concaltcooctttttteaaa
range(2, 6), reduce= ("skip_ed bokip_connect', 2, 0),
255', 2, 1), ('skip_-
connect', 3, 2), bep__cov_33x31 3, 0), ('skip corip_ccnneet,,.. 4, 2), ('ee_conv_3xx)
4, 0), ('skip_connect", 5, 2), ('skip_connect", 5, 0)), reduce c 6))
Genotyyeenormall[[] normal coainal__cocat-aaaee22.. 6), (edacc==("max___] _ppp__ox0333
150
2, 0), fatax_poool_3__ _ooo__33'',_______ 1), sakkp__coneett. 3, 2), faax_ppoo_3x333 0),
('skip_connect connect', 4, 2), "max_pool_3xx" 4, 0), ("skip_connect', 5, 3), ('skip_- connect', 2)), reduce_concat- range(2. 6))
0). C 1aaa_aoo____3_____ _ool_3x3', . 2, 1), ('skip_conned) 3. 2), Laaaaaa_o_o________ ____3333',,____ 2, _skip_connect', 4, 3). ('skip_conned) 2), ("sep_conv33x3" 5, 0), ('dil_conv_ range(2, 6))
Genotype(normal=[], J. coniraaa__ccataraeee2...2 6), reduce=[('avg
200
3, 0),
3x3', 5, 1)],
23

---

dil 3x3
(k-2)
sup__oov__3xs
skip
dil_conv_33d3
C_
Jkp.
Hiip conrect
[_
dil Leoerv 5xx5
c_(k)
sonnect
sop_conv_5xx 5xx
(a) Normal Cell
(b) Reduction Cell
Figure 27: Normal and Reduction cells discovered by FairDARTS CIFAR- 10 for 50 epochs in the standard DARTS' search space.
2:111
X1
(a) E5och=150
(b) Epoch=200
Figure 28: Reduction cells discovered by FairDARTS CIFAR- 10 for 150 and 200 epochs in the standard DARTS' search space. Note that normal cells are empty when FairDARTS is trained for 150 and 200 epochs.
D.2.3 Results on DARTS+PT
DARTS+PT Wang et al. 2021 evaluates operation strength by measuring its contribution to the su- pemet's performance instead of the magnitude of the architecture parameter. Following DARTS+PT's settings, we train DARTS with the permutation baase architecture selection method for 50, 150, and 200 epochs, and the corresponding architecture genotypes over 150 epochs, the resulting architectures are also full of skip connections. Furthermore, to avoid inaccurate evaluation of the operation strength during discretization, DARTS+PT needs to fine-tune the remaining supernet until it converges again. The architecture genotypes with tuned supernets are shown in Table 15] Figure 29 and Figure 30 It's worth noting that the test error of the final resulting architecture for 200 epochs on CIFAR- 10 will drop to 6.28%.
shown in Table I4] When extending to
Table 14: Architecture genotypes found by DARTS+PT on CIFAR-10 for 50, 150, and 200 epochs, respectively. (without supernet tuning)
Epochs
Architecture genotype
Genotypo(normal=(((sg conv _xx'3333330 0), ('dil_comy conv__ovvvv__ 1), ('sep_com conv _m33,,
0). ('dil_ccoy cicii__oov_____________ _333'',, 2), ('dil_commi conv non__3x3', 3), ('skip cecipppconntt'._.m 0), ('dil_conv
conv 5x5', o5n5_5x5',
50
6), reduce=[("ma) ("mxx Booa>_ooo_3xx____ _313333'', 4), cdii__cov__52_ ___5',3))] noomal_coccat-raggen
0), 1aaxx_pool_3331 1), (max_pool_3xx" 0), ('skip_connee 2), ('skip con- nect', 2),
3), ('skip_connect'. 2), Lavg__ool_3$a_ 0)], reduce_-
concat-range(2. 6))
Genotype(normal-((14 cacl__oov____ conv_3x3',, 1). "'ep__cov_3x3" 0), 'skip_cconectt 2).
('skip_connect', 0), ('skip_connect', 1), ('skip_cone) _oaii__connett,, 2), "'skip_conned" 1), (ssiipconneett' 2)), parmal_concat-ranged 6). reduce=[(" max max__ool__3a_ _0o'_3x3',,
150
Baix__poo____00 mool_3x3',, 0), ("skip_con ccti__ccneett, ('max_pool__) aool_3x3',,__ 0), ('skip_connedd) 2), ("max
2), bskkp_cooneect, 3), "'skip_connec"" 3), "'skip_comned"" 2)], reduce_ con-
cat=range(2, 6))
Genotype(normal-(C _onl___oo________lleelll cenv_ooovv33 1), ('sep_conv pap__cooo_33""__ 3x3',.00 0), "'kki__conned" 2). [ww__333',,0 ('skip_connect' 1), (. "skip_connectt 1), 'sski__connee 2), _akip_connect's 1), ('skip_connect', 0)], normal_concat-rrngg
200
6), reduce=[(( mma__pool_3x3 I),
('avg_pool_ 3x3', 0), ('skip connect', 2), ('avg pool 3x3', 0), ('skip connect', 2), ('skip_com copppcctn"t", 3), ('skip_ook connec'' 3), Sakip_connect, 2)], reduce_ con-
cat=range(2, 6))

---

Table 15: Architecture genotypes found by DARTS+PT on CIFAR-I0 for 150 and 200 epochs,
respectively. (with supernet tuning)
Epochs
Architecture genotype
Genotyp((normal(((ppp see__eee____eeeeeeeeeeeeeeeeee [tV_3333',,
(033',0) 0), ('dil conv ___3333''______ 1), fakkp_conneet'' 0). ('skip_connect' connect', 2). ('skip_connect', 0), ( ('dil conv 5x5', 3), Sakip_connect
150
1). ("sep_com conv w_____55'_____ 4) normal concat=range(2, 6), reduce=[( (mm0__a__ppppp_ 1aaaaaooaa_aa__aaaaaaaa pool_3x3'',
1), lav__poo__3aa ___x333', 0), 5x5', 0), (assii_sccnneeeeeee 3), ("dil_cony, cconv_5x5',, 2),
0),
2), ('dil_ ('dil_covv_
4)], reduce_-
concat-range(2, 6))
(eenoypeenorralll(seppp.oo _aee___ov_333_3 "onv_3x3", 0), ('dil_conv _cooc_ocv____________ [ww__333',, 1), ('sep_cony, conv__cvv _3x3',
0), ""skip_connec',, 2), "skki__comnee 0). ('dil conv_5x5" 3), "sski__conne""
200
1). ('dil_ccov__S 4)], normal_conenn commaa_cocatt-aagge22.. 6), reduce=[(" (ea=((((s__ooo)m _env____v_____ __33',,'"
0), ("max_pool_33) 1),
"xoo__xx3', 0), ('skip_connect', connect', 2), ('sep_conv_= 3x3', 1), ('siip_connect', 3), [sep_conv_5x55 0), ( 'skip_connect', 3)J, reduce_- concat-range(2, 6))
(a) Normal Cell
(b) Reduction Cell
Figure 29: Normal and Reduction cells discovered by DARTS+PT on CIFAR-10 for 150 epochs in the standard DARTS' search space (with supernet tuning).
K1,10 30
CI
(a) Normal Cell
(b) Reduction Cell
Figure 30: Normal and Reduction cells discovered by DARTS+PT on CIFAR-10 for 200 epochs in the standard DARTS' search space (with supernet tuning).
25


Reviews:
[{"review_id": "WpvH2H05FG", "reviewer": "Reviewer_fTac", "summary": "\nThis paper propose a method namely operation-level early stopping to address the skip-connection domination issue in domain of differentiable architecture search (DARTS). Though this problem is heavily explored in the past, the authors believe that the key reason of skip-connection domination is because of they try to overfit the validation set used in DARTS based method, and their proposed operation-level early stop that conceptually stop the training of architecture parameter if the overfitting is observed. ", "questions": "Same as above", "limitations": "Moderately discussed. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem of skip-connection dominance is a long standing problem in DARTS domain, and novel method to address this has a clear motivation. \nThe hypothesis of overfitting is the root cause of skip-connection domination is novel and interesting. \nAnalysis of architecture with validation loss is quite clear and justify their hypothesis.\nI appreciate the extensive and honest experiments on all kinds of settings, even though many of the results does not surpass the state-of-the-art. ", "weaknesses": "\nI have several questions regarding this paper and hope to hear back from the authors.\n\n1. Utilizing gradient matching as an indicator to perform early stop seems okay, but this paper lacks of sufficient analysis of what is the key difference between GM+DARTS and their approach, especially regarding why OLES is better. In the related work section, it only describes GM-NAS introduces gradient match score into NAS literature, but this is not enough to let readers understand the difference. \n2. In essence, use early stop to avoid overfitting in DARTS is novel, but experiments seem to show that this OLES does not surpass previous DARTS+PT, which is another indicator to select DARTS operation. I do not understand the urgency to accept another indicator work with similar performance. In addition, similar as above, I do not see much comparison of how OLES surpass DARTS+PT. I agree that OLES can surpass original DARTS, but if it cannot surpass other methods that aims to address the skip-connection domination issue, this is inadequate to appear in a top-tier conference in my humble opinion. \n3. In general, the experiments compared to state-of-the-art is inferior. \nTo the best of my knowledge, this approach has a close relationship with GM+DARTS and DARTS+PT, where in all experiments, the authors should compare to. However, for example, in Table 2, none of these methods exists. In addition, when solely compared to GM+DARTS, in Table 1, it is 0.05 test error better than GM+DARTS. In table 3, GM+DARTS is basically identical as OLES (24.5 v.s. 24.5). In Table 4, OLES surpass GM+DARTS by a margin of 0.2, which is kind of significant. However, in Table 5, GM + ProxylessNAS surpasses OLES again. Why we should accept a paper using the exact gradient matching score in a different manner, which seems inferior than the original approach?\n\nIn addition, I would like to see how this OLES address skip-connection domination with other approaches that aim the same target, not in terms of their final performance, but with respect to their performance to address skip-connection domination issue. After all, this is not the first paper aiming to address it. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will help to address your concerns and improve your view of the paper.\n\n**W1.**\nThank you for pointing this out. We apologize for not providing a detailed explanation for GM-NAS in the related work due to space constraints. Although, as you mentioned, both methods employ the GM Score as a mathematical tool, GM-NAS and our approach are entirely distinct lines of works. Please refer to the global rebuttal for further detail.\n\nWe hope our answers address your concerns. Thanks again for the comment, and we will consider adding the differences to our revised version.\n\n**W2.**\nThanks for the comment. Actually, our approach almost consistently outperforms DARTS-PT and other NAS methods that address the skip connection domination issue. For your convenience, we highlight the comparison between our approach and other NAS methods in the following table. And in most cases, our method remains competitive compared to the SOTA results. \n\n| NAS method | CIFAR-10 | CIFAR-100 (Transfer) | CIFAR-100 | ImageNet|\n|---|---|---|---|---|\n| P-DARTS | 2.50 | 16.55 | 17.46 | 24.4 |\n| R-DARTS(L2) | 2.95 $\\pm$ 0.21 | - | 18.24 | - |\n| FairDARTS | 2.54 | - | - | 24.9 |\n| DARTS- | 2.59 $\\pm$ 0.08 | - | 17.51 $\\pm$ 0.25 | 23.8 |\n| $\\beta$-DARTS | 2.53 $\\pm$ 0.08 | 16.24 $\\pm$ 0.22 | 17.33 | 23.9 |\n| DARTS+PT | 2.48(2.61 $\\pm$ 0.08) | 19.05 | 18.78 | 25.5 |\n| **OLES** | **2.30(2.41 $\\pm$ 0.11)** | 16.30(16.35 $\\pm$ 0.05) | **17.30** | 24.5 |\n\nDespite existing efforts to explore the origins of the skip connection domination issue and the corresponding solutions, it remains an open problem. These methods have put forth their hypotheses but have not fully resolved the issue. Compared to existing methods, the main contribution of our approach is that we offer a novel perspective on the origins of the skip connections domination and propose an effective solution. It is essential to note that our approach is remarkably simple, involving only minimal modifications to the original DARTS, thus incurring negligible additional overhead. We believe that the current performance of OLES is sufficient to validate our hypothesis and demonstrate that our method effectively resolves the skip connection domination issue in differentiable NAS.\n\nTo further demonstrate the effectiveness of addressing the skip connection domination issue, we increase the number of iterations of supernet training. As depicted in Figures 1&2 in our attached PDF, DARTS-PT fails to solve the skip connection domination issue, indicating that their hypotheses do not entirely reveal the fundamental causes of this problem, or rather, they merely delay the occurrence of skip connection domination.\n\nTherefore, our work is meaningful as we offer a novel perspective and idea that not only proves effective but is also simpler and more straightforward to comprehend than previous perspectives. Instead of introducing yet another indicator, we aspire to stimulate further exploration in this field based on our insightful observations regarding the overfitting phenomenon. We believe this can lead to a deeper understanding of the differentiable architecture search process. We hope our answers address your concerns, and we will highlight the differences between our approach and other methods that focus on the skip-connection domination issue.\n\n**W3.**\nThank you for pointing this out. For both GM-NAS and DARTS-PT, the results on CFIAR-100 are not found in their original papers. For more comprehensive comparisons, we reproduced the results of these two methods on CFIAR-100 using their open-source codes, and the results are also displayed in the table below.\n\n||DARTS+PT|GM+DARTS|OLES|\n|---|---|---|---|\n| CIFAR-100 (Transfer) | 19.05 | 16.45 | **16.30** |\n| CIFAR-100 |18.78 | 17.42 | **17.30** |\n\nIt can be observed that the proposed OLES performs better than GM+DARTS and DARTS+PT on CIFAR-100. As previously stated, GM-NAS is a much more complex method that falls in a distinct line of research. Due to its few-shot nature, it involves higher time and space complexities. Moreover, GM-NAS is orthogonal to our approach. OLES focuses on addressing the skip connections domination issue in differentiable NAS. We aim to uncover the fundamental causes of this issue and require only minimal modifications to the original DARTS, incurring negligible additional overhead. Despite this, our method remains highly competitive compared to GM-NAS. Furthermore, our main contribution is the insightful observation and hypothesis regarding overfitting, as well as the proposal of the early stopping method. The GM score, in this context, serves merely as an indicator for early stopping.\n\nTo demonstrate the effectiveness of our approach in different search spaces, we also perform OLES on the MobileNet search space. As it is not naturally designed for DARTS, we had to make several modifications (ProxylessNAS has adapted to this search space, and FairDARTS did not open-source their search process on MobileNet). Therefore, we believe there is still room for optimization in the results presented. Nevertheless, OLES achieved highly competitive performance in the MobileNet search space. \n\n**W4.**\nThanks for the suggestion. Besides the final performance, we display the number of skip connections in the searched architecture as the number of training iterations increases in Figure 2 of our attached PDF. It is evident that DARTS exhibits a severe skip connection domination issue, while OLES effectively resolves this problem. When training with an extended number of training epochs. Methods like DARTS-PT tend to have an increasing trend of skip connections, whereas OLES remains stable, which further verifies the insightful observations on the fundamental causes of the skip connection domination issue and the effectiveness of the proposed solution."}, {"review_id": "L8MaBdHgIK", "reviewer": "Reviewer_XYdd", "summary": "The paper focuses on the robustness issues in differentiable NAS, specifically the domination of skip connections. It first analyzes the issue from a novel perspective, proposing that the domination of skip connections arises due to the overfitting of operations in the supernet during training. Then, the paper proposes the operation-level early stopping method, which monitors each operation in the supernet and stops its training when it tends to overfit. The paper employs a gradient matching approach to detect overfitting, comparing the gradients' directions of operations on training and validation data. A significant deviation in direction is considered an indication of overfitting. The proposed OLES addresses the domination of skip connections with negligible additional overhead. Extensive experiments demonstrate the effectiveness of OLES on different datasets and search spaces. ", "questions": "Q1. Are there any limitations to the methodology presented in this paper? Can it be applied to other differentiable NAS methods beyond the original DARTS?\n\nQ2. Why does the proposed method not exhibit a competitive advantage over other state-of-the-art NAS methods?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The paper demonstrates good originality. It provides a comprehensive analysis of the issue of the domination of skip connections in the differentiable NAS by dopting a new perspective. Although straightforward, the perspective makes sense and is interesting. Specifically, the paper aims to explain the cause of this issue through the overfitting of operations in the supernet. The paper proposes the operation-level early stopping (OLES) method, which introduces gradient matching to address this matter. OLES elegantly and effectively resolves the domination of skip connections, incurring negligible additional overhead. \n\nS2. The experiments are thorough and well-organized, containing experiments in different search spaces and an in-depth analysis of the proposed algorithm. The empirical results demonstrate that the proposed OLES  achieves state-of-the-art performance on CIFAR. The availability of open-source codes further facilitates reproducibility.\n\nS3. The presentation of ideas and algorithms is clear, while the references and background knowledge are comprehensive. The background knowledge and the issue to be solved are adequately introduced.\n\nS4. The perspective and idea about the overfitting of operations have profound significance in uncovering the underlying causes of the domination of skip connections in DARTS. These novel perspectives, ideas, and algorithms contribute to a deep understanding of differentiable architecture search and may inspire future research in the field.", "weaknesses": "W1. It needs a thorough explanation of the used gradient matching method. The authors should provide a more detailed introduction to gradient matching and clarify the differences from other methods.\n\nW2. While the proposed method demonstrates significant improvements over the original DARTS in the experiments, it does not have a competitive advantage compared to other state-of-the-art methods. \n\nW3. The paper specifically focuses on addressing the domination of skip connections through operator-level early stopping. The authors are suggested to discuss how the concept of operator-level early stopping can be applied to other scenarios beyond differentiable architecture search.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review. We are glad to receive your positive response and acknowledgment of our work. Next, we will answer your questions one by one, and we hope this will improve your acceptance of our paper. \n\n**W1.**\nThanks for the valuable advice. Gradient matching aims to leverage the gradient information of parameters to assist in making decisions and selections. For example, GM-NAS utilizes gradient matching scores to make splitting decisions, determining whether a module should be shared among child architectures. The gradient matching scores are computed based on the gradient information of different child models on shared parameters. There are also methods that utilize GM for dataset condensation. \nIn our proposed OLES, we employ the GM score as an indicator for early stopping, preventing operation parameter overfitting. \nThe gradient matching scores, in our approach, are calculated using the gradient information of parameters on both training and validation data. We hope our answers address your concerns.\n\n**W2 \\& Q2.**\nThank you for pointing this out. Compared to other sophisticated NAS methods, we aim to solve the skip connection domination issue in differentiable NAS and uncover the fundamental causes of this issue. OLES offers a novel perspective on the origins of the skip connection domination and proposes simple yet effective solutions. It is essential to note that our approach requires only minimal modifications to the original DARTS, thus incurring negligible additional overhead. \nPractitioners can easily enhance the performance of OLES by integrating it with other state-of-the-art differentiable NAS algorithms, conducting more fine-tuning, and leveraging advanced training tricks such as the SE module. We hope our answers address your concerns.\n\n**W3 \\& Q1.**\nMany thanks for the suggestion. OLES can also be employed as a plugin module for other differentiable NAS methods. It is especially well-suited for scenarios where the NAS method alternately trains on the training and validation datasets. Depending on the specific differentiable NAS methods and search spaces, it may be necessary to adjust the early stopping threshold setting to achieve optimal performance."}, {"review_id": "ymP4C2KBdK", "reviewer": "Reviewer_nDad", "summary": "This paper demonstrates the fundamental reason for the domination of skip connections in DARTS from the new perspective of overfitting of operations in the supernet, using preliminary experiments,  and proposed the operation-level early stopping method to mitigate this phenomenon by using the GM score metric during the searching.  ", "questions": "See Weakness", "limitations": "Yes, the authors adequately addressed the limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using GM metric to decide whether to update the OPs is broadly used in efficient-training papers.  This paper proposes to use this metric to early stop the updating of specific OPs during the NAS procedure, based on their novel overfitting observations. \n\nThe comparison is intensive, showing the superiority of this proposed method, in terms of time cost, and accuracy. ", "weaknesses": "1. The accuracy metric is one of the metrics to measure a NAS method. We also consider the Kendall rank correlation coefficient. Please compare with previous methods using this metric, because I am not sure whether the proposed early stopping mechanism will hurt the ranking or not. Usually, we say the ranking performance of a NAS method may be more important than the accuracy of the searched model. \n\n2. About the overfitting threshold, \"we determine the threshold by averaging the cosine similarity over 20 iterations for 30 randomly initiated architectures in each search space\"    Is the initial 20 iterations ok or enough for determining the threshold? The initial stages may have dramatic changes in gradient. Also, the threshold is fixed during the NAS procedure, shouldn't it be adaptive or scheduled? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**W1.**\nThanks for the valuable suggestion. As shown in Figure 3 in our attached PDF, we compare the Kendall rank correlation coefficients for OLES and DARTS. The results indicate that the early stopping mechanism does not hurt the ranking performance of DARTS. In the following table, we also conduct a comparison with other NAS methods, including RandomNAS, DARTS, GDAS, and SGAS. (RandomNAS and GDAS sample 4 excellent architectures in 4 rounds, and other methods randomly select 10 architectures from a single run.)\n\n| RandomNAS$^*$ | DARTS | GDAS$^*$ | SGAS | OLES |\n|---|---|---|---|---|\n| 0.0909 | 0.13 | -0.1818 | 0.42 | 0.41 |\n\nNotably, the Kendall coefficient of OLES closely aligns with SGAS. SGAS [1]  aims to alleviate the effect of the degenerate search-retraining correlation problem. The results demonstrate that by mitigating operation parameter overfitting, differentiable NAS could focus on the potential of architectures themselves, thus enhancing the correlation between search metrics and the architectures discovered. We hope our answers address your concerns, and we will consider adding the comparison of the Kendall rank correlation coefficients into the revised version.\n\n**W2.**\nThank you for pointing this out. Instead of relying on search or empirical knowledge to determine the early stopping threshold, we intend to employ an adaptive approach to determining this critical hyperparameter. It is important to clarify that our gradient matching (GM) score is dynamically computed by averaging over every 20 iterations throughout the entire training iterations, rather than being limited to the initial 20 iterations. We apologize for the confusion here. The selection of 30 randomly initiated architectures is carried out to initiate the experiment 30 times, enabling the identification of the optimal iteration number. As you pointed out, the initial stages may have dramatic changes in gradient values. Therefore, in our experiments, we compute the GM score throughout the entire training process to ensure the effectiveness of early stopping.\n\nTo elucidate the rationale behind selecting 20 iterations, we conduct experiments to assess various iterations along with their corresponding thresholds across different search spaces. As shown in the following table, the accuracy attained through GM score computation over every 20 iterations achieves the highest accuracy.\n\nWe have not yet found a better method to determine the threshold in a more adaptive or schedulable manner, and we will leave it for future work. We sincerely appreciate your valuable suggestions and we will make more clear introduction about the selection of  the overfitting threshold in our revised version.\n\n| Iterations |DARTS Search Space|| NAS-Bench-201 Search Space |||\n|---|---|---|---|---|---|\n|| CIFAR-10 | CIFAR-100 | CIFAR-10 | CIFAR-100 | ImageNet-16-120 |\n| 10 | 97.33 | 80.83 | 80.57 | 47.93 | 26.29 |\n| 20 | **97.70** | **82.70** | **93.89** | **70.75** | **44.38** |\n| 30 | 97.05 | 82.39 | 91.93 | 66.97 | 38.87 |\n| 40 | 97.19 | 81.32 | 89.76 | 64.69 | 32.73 |\n\n\n[1]  Snas: stochastic neural architecture search, ICLR 2019."}, {"review_id": "Dk0rYYZgGh", "reviewer": "Reviewer_5e2k", "summary": "The authors are adressing the issue of converging to a degenerated solution (many skip connections) using DARTS. The authors connect this behavior to an overfitting to the train data. To remedy this issue, they suggest to apply early stopping based on the correlation of gradients during the architecture parameters training and the wekths parameters training. ", "questions": "Adressing the mentioned weaknesses appropriatly would lead to a higher rating ", "limitations": "None", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 1, "strengths": "The paper is trying to tackle an interesting problem in an elegant way. The story is clearly stated and the introduced approach is a nice extension of the classical early stopping. The performance of this method is on par or better than many other more sophisticated (and to some extent complicated) method. ", "weaknesses": "* The authors are claiming that the dominance of skip connections is due to the quick overfitting of weighted operations to the train data. The whole story of the paper relies on this assumption which is unfortunately not sufficiently supported. While curves on figure 1(a) are supporting this claim, figures 3(a) and 3(b) are a bit confusing.  I would have expected the test performance to drop for both cases.\n\n* There is an issue regarding the reported numbers: the authors are claiming that their method benefits operations with parameters and the test error is dropping (as expected), I am wondering why the number of parameters is barely different. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your considerate feedback. We sincerely appreciate your engagement in the review. Next, we will address your concerns one by one, and we hope this will improve your view of the paper.\n\n**W1.**\nThanks a lot for your careful and insightful observation. In fact, in Figures 3(a) and 3(b), the number of iterations of DARTS has not yet reached the threshold where the test performance starts to drop, indicating that the overfitting issue has not yet appeared. In Figure 4 of our attached PDF, we have redrawn Figures 3(a) and 3(b) and extended the number of training iterations. Consequently, it becomes apparent that when the number of training iterations continuously increases (>50), DARTS exhibits a noticeable decline in test performance. Moreover, Figure 3(c) in the original paper further corroborates this finding, as the problem of performance collapse due to the skip connection domination issue more easily appears in NAS-Bench-201. As a result, it manifests the test performance declines at an earlier stage. We hope our answers address your concerns, and we will consider improving Figures 3(a) and 3(b) in the revised version.\n\n**W2.**\nThanks for pointing this out. Our approach aims to address the issue of performance collapse caused by the abnormal aggregation of skip connections in differentiable NAS. NAS methods fundamentally seek the optimal model architecture. In reality, models with more parameters are not necessarily superior to those with fewer parameters. The phenomenon you described, i.e., that the number of parameters in OLES is barely different with other methods, actually serves to illustrate that OLES can resolve the issue of abnormal skip connection domination, rather than invariably benefit operations with parameters.\n\nFurthermore, in our comparisons, we presented the optimal results of other methods (mostly from the original paper's reported results). These results were obtained before the occurrence of skip connection domination.  Once the skip connection domination issue emerges, there will be a noticeable performance collapse. As shown in Figures 1&2 in our attached PDF and Section C.2 in our supplementary material and, DARTS and other methods like DARTS-PT encounter skip connection domination and performance collapse after training for longer epochs.  Thus, there are no significant differences in the number of parameters.\n\nWe hope our answers address your concerns. Thanks again for this insightful comment. "}, {"review_id": "eMtYDt59je", "reviewer": "Reviewer_nZNk", "summary": "This paper studies the robustness issue of DARTS from the perspective of overfitting. It uses gradient matching scores to measure the overfitting issues, and proposes an early-stop strategy to address the problem of saturated skip connections in normal DARTS. The proposed approach has been evaluated on a number of search spaces, showing comparable results to the state of the art. ", "questions": "* How the proposed approach performs on DARTS S1-S4?\n* It seems this approach doesn't work so well on MobileNet space, any possible reasons for that? \n* How this approach may compare to the train-free methods, like TE-NAS and ZiCo? ", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The idea of using similarity between gradient directions of training vs validation batch makes sense. \n+ Extensive experiments on various search spaces and benchmarks. \n+ Decent results on DARTS C10 space comparing to SOTA.", "weaknesses": "- Although the idea to use early stopping to robustify DARTS is interesting, the metric used in this paper is very similar to that in GM-NAS [1]. Therefore it seems to me that the novelty of this paper is somewhat discounted. \n- It is good to see experiments on a number of search spaces and benchmarks. However, the results on DARTS S1-S4 space is missing. \n- Some discussion on overfitting is not very clear. e.g. from Fig.1, it is difficult to see quantitively how the negative correlation between arch params and val loss: indeed the curvature of the lines are different, but it would be better to have some more rigid analysis. \n\n\n[1] Generalizing Few-Shot NAS with Gradient Matching. Shoukang Hu*, Ruochen Wang*, Lanqing Hong, Zhenguo Li, Cho-Jui Hsieh, and Jiashi Feng. ICLR 2022. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful and insightful review, which is very helpful for us to further improve this paper. Next, we will answer your questions one by one, and we hope this will improve your acceptance of the paper.\n\n**W1.**\nThank you for pointing this out. We apologize for not providing a detailed explanation for GM-NAS in the related work due to space constraints. In reality, GM-NAS and our approach are entirely distinct lines of work. Please refer to the global rebuttal for further detail.\n\nWe hope our answers address your concerns, and we will consider adding the differences to our revised version.\n\n**W2 \\& Q1.**\nMany thanks for the valuable advice. DARTS S1-S4 are not standard search spaces. They are simplified variations of the DARTS search space proposed by R-DARTS to investigate DARTS' failure mode. In our supplementary material, we further expanded these search spaces and presented DARTS' failure mode (both in terms of performance and the searched architectures) in these extended search spaces. To address your concerns, in the following table, we provide an additional performance comparison between OLES and other methods on DARTS S1-S4.\nIt is evident that OLES also consistently outperforms DARTS, as well as R-DARTS and DARTS-PT which are also dedicated to addressing the skip connection domination issue.\n\n| Datasets | Spaces | DARTS | R-DARTS | DARTS-PT | OLES |\n|---|---|---|---|---|---|\n| CIFAR-10 | S1 | 3.28 | 3.31 | 2.79 | **2.76** |\n|  | S2 | 2.55 | 2.44 | 2.47 | **2.43** |\n|  | S3 | 3.69 | 3.56 | 2.64 | **2.57** |\n|  | S4 | 3.05 | 3.05 | 2.92 | **2.73** |\n| CIFAR-100 | S1 | 26.05 | 22.24 | 23.16 | **22.10** |\n|  | S2 | 25.40 | 23.34 | 22.10 | 22.18 |\n|  | S3 | 24.70 | 21.94 | 20.80 | **20.70** |\n|  | S4 | 21.35 | 20.70 | 19.98 | **19.18** |\n\n**W3.**\nThanks for the suggestion. In Figure 1, through the curves of validation losses and architecture parameters over search iterations, it is evident that there exists a negative correlation between validation loss and architecture parameter. To provide a more rigid analysis, we calculated the Kendall-$\\tau$ correlations between validation loss and architecture parameter for each operation. The results are presented in the following table. \n\n| Skip-Connections | Sep-Conv-3x3 | Sep-Conv-5x5 | Dil-Conv-3x |\n|---|---|---|---|\n| -0.36 | -0.59 | -0.73 | -0.72 |\n\nIt can be observed that there is a clear negative correlation between validation losses and architecture parameters, particularly for parametric operations, where the negative correlation is more noticeable.\n\nFurthermore, we simplify the scenario by considering only one set of candidate operations $\\{o_1, o_2, o_3\\}$.  Suppose their corresponding validation losses and architecture parameters are $\\{l_1, l_2, l_3\\}$ and $\\{\\alpha_1, \\alpha_2, \\alpha_3\\}$, respectively. The overall validation loss can be expressed as:\n$$\nl = l_1 \\cdot \\frac{e^{\\alpha_1}}{e^{\\alpha_1} + e^{\\alpha_2} + e^{\\alpha_3}} + l_2 \\cdot \\frac{e^{\\alpha_2}}{e^{\\alpha_1} + e^{\\alpha_2} + e^{\\alpha_3}} + l_3 \\cdot \\frac{e^{\\alpha_3}}{e^{\\alpha_1} + e^{\\alpha_2} + e^{\\alpha_3}}\n$$\nthen\n$$\n\\frac{\\partial l}{\\partial \\alpha_1} = \\frac{(l_1 - l_2)e^{\\alpha_1 \\alpha_2} + (l_1 - l_3) e^{\\alpha_1 \\alpha_3}}{(e^{\\alpha_1} + e^{\\alpha_2} + e^{\\alpha_3})^2}\n$$\nHence, when $l_1$ is larger, the corresponding $\\frac{\\partial l}{\\partial \\alpha_1}$ will be larger as well, leading to a more significant decrease in $\\alpha_1$ during gradient descent. Other operations are similar. Although this scenario is significantly straightforward, it illustrates some connections between validation losses and architecture parameters. We hope our answers address your concerns.\n\n**Q2.**\nThanks for the comment. Indeed, in the MobileNet search space, our approach has achieved competitive performance compared to state-of-the-art methods, slightly trailing behind several more complex approaches. As the MobileNet search space is not naturally designed for DARTS, we had to make several modifications (ProxylessNAS has adapted to this search space, and FairDARTS did not open-source their search process on MobileNet). Therefore, we believe there is still room for optimization in the results presented. Here, the purpose of using the MobileNet search space is to demonstrate the effectiveness of our approach in different search spaces.\n\nFurthermore, our approach is remarkably simple, involving only minimal modifications to the original DARTS, thus incurring negligible additional overhead. It is easy to enhance the performance of OLES by integrating it with other more advanced differentiable NAS algorithms, conducting more fine-tuning, and leveraging advanced training tricks.\n\n**Q3.**\n\nMany thanks for the suggestion. Train-free NAS utilizes proxy metrics to predict the test performance or ranking of architectures without the need for training. However, current train-free methods still slightly lag behind traditional NAS methods in terms of performance, especially in large search spaces.\n\nIn the following table, we compare OLES with TE-NAS and ZiCo. It is evident that OLES still outperforms train-free methods in most cases. Benefiting from the compact search space, train-free methods exhibit slightly superior performance on NAS-Bench-201. However, in large search spaces such as DARTS search space, OLES performs better than TE-NAS and ZiCo. Train-free NAS is a highly promising research direction, as it not only greatly enhances NAS efficiency but also promotes a deeper understanding of neural networks.\n\n|| NAS-Bench-201 ||| DARTS Search Space |||\n|---|---|---|---|---|---|---|\n|||||| ImageNet |||\n|| CIFAR-10 | CIFAR-100 | ImageNet-16-120 | CIFAR-10 | top-1 | top-5 |\n| TE-NAS | 6.1 $\\pm$ 0.47 | 28.76 $\\pm$ 0.56 | 57.62 $\\pm$ 0.46 | 2.63 $\\pm$ 0.064 | 26.2 | 8.3 |\n| ZiCo | 6.0 $\\pm$ 0.4 | 28.9 $\\pm$ 0.3 | 58.20 $\\pm$ 0.3 | 2.45 $\\pm$ 0.11 | - | - |\n| OLES | 6.3 $\\pm$ 0.15 | 29.60 $\\pm$ 0.22 | **56.03 $\\pm$ 0.38** | **2.41 $\\pm$ 0.11** | **24.5** | **7.4** |\n"}]

Author rebuttal:
Dear AC and reviewers,

We would like to thank all the reviewers for their great efforts, insightful comments, and valuable suggestions, which are very helpful for us to further improve this paper. To the best of our efforts, we’ve diligently tried to address all the specific comments, including the minor ones that have been raised by each reviewer, by adding additional experiments and detailed analyses. Next, we would like to respond to the questions raised by the reviewers point by point, and we will incorporate these clarifications in the future version of our paper. We sincerely hope that our responses can address reviewers' concerns and translate into higher scores.

In the attached PDF, we present some additional experimental results to provide further clarification for addressing the concerns of the reviewers. In Figure 1, we compare the performance of our OLES against other methods when training for longer epochs. Similarly, Figure 2 displays the number of skip connections obtained by different methods. In Figure 3, we present the search-retraining Kendall coefficients of OLES and DARTS. Lastly, Figure 4 shows the results of the original paper's Figure 1(a) and Figure 1(b) under extended training iterations. We kindly recommend the esteemed reviewers refer to the content provided in the attached PDF while reviewing our rebuttal.

Here we provide general answers to similar questions mentioned by the reviewers.

**1. Distinctions between OLES and GM-NAS**

In reality, GM-NAS and our approach are entirely distinct lines of work, addressing completely different problems. They just both employ the GM Score as a mathematical tool. Specifically, GM-NAS falls within the field of Few-Shot NAS research. They argue that due to coupled optimization between child architectures caused by weight-sharing, One-Shot supernet’s performance estimation could be inaccurate, leading to degraded search results. As a result, they propose to reduce the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets. GM-NAS utilizes gradient matching scores to make splitting decisions, determining whether a module should be shared among child architectures. The gradient matching scores are computed based on the gradient information of different child architectures on shared parameters.

In contrast, the proposed OLES aims to address the problem of skip connection donimation from a totally new perspective and employ the GM score as an indicator for early stopping, preventing operation parameter overfitting. The gradient matching scores, in our approach, are calculated using the gradient information of parameters on training and validation data.

Also, while GM-NAS offers performance enhancements to some extent, the time complexity of GM-NAS is larger, leading to higher search costs compared to OLES. As shown in Table 1 and Table 3, the search costs of GM-DARTS is 3 times and 6 times higher than OLES on CIFAR-10 and ImageNet, respectively. Due to only minimal modifications to the original DARTS, our approach incurs negligible additional overhead.

Furthermore, due to different lines of works, our approach and GM-NAS are orthogonal, enabling their combined usage to further enhance performance. We will try it in our future work.

Thanks again to all the reviewers for their constructive suggestions. We will do our best to improve our paper and address all concerns raised. We sincerely hope our answers can address the all the concerns reviewers have raised and improve the score of our paper.

Best regards,

Authors of Submission 11229

Meta-review: This paper addresses the robustness issues in differentiable NAS via a new operation-level early stopping to address the skip-connection domination issue in domain of DART. Experiments show the effectiveness of OLES on different datasets and search spaces and it resolves the domination of skip connections with negligible additional overhead. Reviewer fTac questions the difference between GM+DARTS and OLES. The general rebuttal gives detailed and convincing comparison. Reviewers' concerns about experiments have been responded such as, OLES does not surpass previous DARTS+PT, which is wrong understanding that has been clarified in W2. Reviewer 5e2k raised the issues about reported numbers and test performance, which have been addressed by authors. Both Reviewer fTac and Reviewer 5e2k are not involved in the discussion. From Area chair's understanding, the authors have addressed the problems. Thus the whole paper is good enough for Neurips.
Decision: Accept (poster)