DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
Authors: Xiangxiang Chu
Xiaoxing Wang
Bo Zhang
Shun Lu
Xiaolin Wei
Junchi Yan
Venue: ICLR 2021 Poster (2021)
Keywords: neural architecture search
DARTS stability

Abstract: Despite the fast development of differentiable architecture search (DARTS), it suffers from a standing instability issue regarding searching performance, which extremely limits its application. Existing robustifying methods draw clues from the outcome instead of finding out the causing factor. Various indicators such as Hessian eigenvalues are proposed as a signal of performance collapse, and the searching should be stopped once an indicator reaches a preset threshold.
However, these methods tend to easily reject good architectures if thresholds are inappropriately set, let alone the searching is intrinsically noisy. In this paper, we undertake a more subtle and direct approach to resolve the collapse. 
We first demonstrate that skip connections with a learnable architectural coefficient can easily recover from a disadvantageous state and become dominant.  We conjecture that skip connections profit too much from this privilege, hence causing the collapse for the derived model. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. Extensive experiments on various datasets verify that our approach can substantially improve the robustness of DARTS. Our code is available at https://github.com/Meituan-AutoML/DARTS-

Published as a conference paper at ICLR 2021
DARTS-: ROBUSTLY STEPPING OUT OF PERFOR- MANCE COLLAPSE WITHOUT INDICATORS
Xiangxiang Chu', Xiaoxing Wang Bo Zhang', Jhaang
Shun Lu 13. Xiaolin Wei Junchi Yan" 'Meituan, *Shanghai Jiao Tong University. Unniersitty of Chinese Academy of Sciences {chuxi cchuxiangxiang, zhangbo97, weixiaolin02 @me iineitt.n.... com {figurel wxx, yan junchi es jtu. edu. lushun1 9@mails ucas. ac cn
ABSTRACT
Despite the fast development of differentiable architecture search (DARTS), it suffers from long-standing performance instability, which extremely limits its ap- plication. Existing robustifying methods draw clues from the resulting deterio- rated behavior instead of finding out its causing factor. Various indicators such as Hessian eigenvalues are proposed formance collapses. However, these indicator andeed methods tend to easily reject good architectures if the thresholds are inappropriately set, let alone the search- ing is intrinsically noisy. In this paper, undertake a more subtle and direct approach to resolve the collapse. We first demonstrate that skip connections have clear advantage over other candidate operations, where it can easily recover from a disadvantageous state and become dominant. conjecture that this priv- ilege is causing degenerated performance. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. We call this approach DARTS-. Extensive experiments datasets verify that it can substantially improve robustness. Our code is available at https:: / thub. com /om/Meeii Me tuan Aut.oM/DAARTS.. /tt.://ttttaaAoooO/A//
signal to stop searching before the per-
various
INTRODUCTION
Recent studies (Zela et al., 2020; Liang et al., 2019; Chu et al., 2020b) have shown that one critical issue for differentiable architecture search (Liu et al., 2019b) regarding the performance collapse due to superfluous skip connections. Accordingly, some empirical indicators for detecting the occurrence of collapse have been produced. R-DARTS (Zela al., 2020) shows that the loss landscape has more curvatures (characterized by higher Hessian eigenvalues w.r.t. architectural weights) when the derived architecture generalizes poorly. By regularizing for lower Hessian eigenvalue, Zela et al. (2020); Chen & Hsich (2020) attempt to stabilize the search process. Mcanwhile, by directly constraining the number of skip connections to fixed number (typically 2). the collapse issue becomes less pronounced (Chen et al., 2019b; Liang et al., 2019). These indicator-basee approaches have several main drawbacks. Firstly, robustness relies heavily on the quality of the indicator. An imprecise indicator either inevitably accepts poor models or mistakenly reject good ones. Secondly, indicators impose strong priors by directly manipulating the inferred model, which is somewhat suspicious, akin to touching the test set. Thirdly, extra computing cost (Zela et al., 2020) or careful tuning of natural ask the following questions:
(Chen et al., 2019b; Liang et al., 2019) are required. Therefore, it's
Can we resolve the collapse without handcrafted indicators and restrictions to interfere with the searching and/or discretization procedure?
Is it possible achieve robustness in DARTS without tuning extra hyper-paramettrs?
WWork done as an intern at Meituan Inc.
"oorrsponddet Author.

---

Published as a conference paper at ICLR 2021
To answer the above questions, propose an effective and efficient approach to stabilize DARTS. Our contributions can be summarized as follows:
Arge
Maxt
sept:
skap
AraP
Mea
SeyC
DRC
Skip
Ausiiarr Skip
New Paradigm to Stabilize DARTS. While empirically observing that current indicators (Zela et al., 2020; Chen & Hsieh, 2020) can avoid performance collapse at a cost of reduced exploration coverage the search space, we propose a novel indicator-free approach to sta- bilize DARTS, referred to as DARTS-1, which involves an auxiliary skip connection (see Fig- ure 1) to remove the unfair advantage (Chu et al., 2020b) in the searching phase.
(a) DARTS
(b) DARTS-
Figure I: Schematic illustration about (a) DARTS and (b) the proposed DARTS-, featuring auxil- iary skip connection (thick red line) with a decay rate 3 between every two nodes to remove the po- tential unfair advantage that leads to performance collapse.
Strong Robustness and Stabilization. We conduct thorough experiments across seven search spaces and three datasets to demonstrate
the effectiveness of our method. Specifically, approach robustly obtains state-of-the-arr results on 4 search space with 3x fewer search cost than R-DARTS (Zela et al., 2020), which requires four independent runs report the final performance.
Seamless Plug-in Combination with DARTS Variants. We conduct experiments to demonstrate that our approach can work together seamlessly with other orthogonal DARTS variants by removing their handcrafted indicators without extra overhead. In particular, our approach is able to improve 0.8% accuracy for P-DARTS, and 0.25% accuracy for PC-DARTS on CIFAR-10 10 dataset.
2 RELATED WORK
Neural architecture search and DARTS variants. Over the years, researchers have sought to automatically discover neural architectures for various deep learning tasks to relieve human from the tedious effort, ranging from image classification (Zoph et al., 2018), objection detection (Ghiasi et al.. 2019), image segmentation (Liu et al., 2019a) to machine translation (So et al., 2019) etc. Among many proposed approaches, Differentiable Architecture Search (Liu et al., 2019b) features weight-sharing and resolves the searching problem via gradient descent, which is very efficient and easy to generalize. A short description of DARTS can be found in A. Since then, many subsequent works have been dedicated to accelerating the process (Dong & Yang. 2019b), reducing memory cost (Xu et al., 2020), or fostering its ability such as hardware- awarensss (Cai et al., 2019; Wu et al., 2019), finer granularity (Mei et al., 2020) and so However, regardless of these endeavors, fundamental issue of DARTS over its searching performance collapse remains not properly solved, which extremely prohibits its application.
Robustifying DARTS. AS DARTS (Liu et al., 2019b) is known to be unstable as a result of per- formance collapse (Chu et al., 2020b), some recent works have devoted to resolving it by either designing indicators like Hessian eigenvalues for the collapse (Zela et al., 2020) or adding pertur- bations to regularize such an indicator (Chen & Hsieh, 2020). Both methods rely heavily on the indicator's ' accurateness, i.e to what extent does the indicator correlate with the performance col- lapse? Other methods like Progressive DARTS (Chen al., 2019b) and DARTS+ (Liang et al., 2019) employ a strong human prior, i.e., limiting the number of skip-connections to be a fixed value. Fair DARTS (Chu et al., 2020b) argues that the collapse results from the unfair advantage in an ex- clusive competitive environment, from which skip connections overly benefit to cause abundant aggregation. To suppress such an advantage from overshooting, they convert the competition into collaboration where each operation is independent of others. It is however an indirect approach. SGAS (Li et al., 2020), instead, circumvents the problem with a greedy strategy where the unfair advantage be prevented from taking effect. Nevertheless, potentially good operations might be pruned out too early because of greedy underestimation.
We name it so we undertake an inward way, as opposed to those outward ones who design new indicators, add extra cost and introduce new hyper-parameters.
2

---

Published as a conference paper at ICLR 2021
3 DARTS-
3.1 MOTIVATION
We start from the detailed analysis of the role of skip connections. Skip connections were proposed to construct a residual block in ResNet (He et al., 2016), which significantly improves training stability. is even possible deepen the network up to hundreds of layers without accuracy degra- dation by simply stacking them up. In contrast, stacking the plain blocks of VGG has degenerated performance when the network gets deeper. Besides, Ren et al. (2015); Wei et al. (2017); Tai et al. (2017); Li et al. (2018b) also empirically demonstrate that deep residual network can achieve better performance on various tasks.
From the gradient flow's perspective, skip connection is able alleviate the gradient vanishing problem. Given a stack of n residual blocks, the output of the (i + 1)th residual block X,+1 can be computed X1+1 1x+,(X,, Wi+1)= + X;, where fi+1 denotes the operations of the 1 residual block with weights W1+1- Suppose the loss function of the model is L, and the gradient of X; can be obtained as follows (1 denotes a tensor whose items are all ones):
ac OC
Oc (DiT- +1 +xx=+1)=-0 0Xi+s ax:
afi-k
+1)
II k=1
0X;
0X,+1
0x
We observe that the gradient of shallow layers always includes the gradient of deep layers, which mitigates the gradient vanishing of W,. Formally we have,
ac Oc OW; 0x,+j
J I' k=1
( Ofttk
+1) Of aw,
(axit
To analyze how skip connect affects the performance of residual networks, we introduce trainable coefficient 3 on all skip connections in ResNet. Therefore, the gradient of X; is converted to:
0c OC (ofi1) (xxx
+B)
ax,
0X,1
Once 3 < 1, gradients of deep layers gradually vanish during the back-propagation (BP) towards shallow layers. Here B controls the memory of gradients in BP to stabilize the training procedure.
We conduct a confirmatory experiment on ResNet50 and show the result in Fig 2. By initializing B with {0, 0.5, 1.0). we can visualize the tendency of 3 along with training epochs. We observe that 33 converges to- wards after 40 epochs demonstrates that the residual structure learns to push 3 rather large value alleviate gradient vanishing.
matter the initialization, which
190
156
22-10
Epock
100
120
140
100
Similarly, DARTS (Liu et al., 2019b) utilizes a trainable parameter Sakip to denote the importance of skip connec- tion. However, In the search stage, 3skip can generally increase and dominate the architecture parameters, and finally leads to performance collapse. We analyze that large Bakip in DARTS could result from two aspects: On the one hand, as the supernet automatically learns to alleviate gradient vanishing, pushes Bskip a proper large value; On the other hand, the skip connection is indeed an important connection for the target network, which should be selected in the discretization stage. As a consequence, the skip connection in DARTS plays two-fold roles: as an auxiliary connection to stabilize the supernet training, and candidate operation build the final network. Inspired by the above observation and analysis, we propose to stabilize the search process by distinguishing the two roles of skip connection and handling the issue of gradient flow.
Figure 2: Tendency trainable coef- ficient 3 (initialized with {0, 0.5, I)) the skip connection in ResNet50 and test accuracy (inset figure) vs. epochs. The residual structure is proved to learn large / ease training in all three cases. All models are trained and tested on CIFAR-10.
3

---

Published as a conference paper at ICLR 2021
3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE
To distinguish the two roles, we introduce an auxiliary skip connection between every two nodes in cell, see Fig. 1 (b). On the one hand, the fixed auxiliary skip connection carries the function of stabilizing the supernet training, even when Sskip is rather small. On the other hand, it also breaks the unfair advantage (Chu al., 2020b) as the advantageous contribution from the residual block is factored out. Consequently, the learned architectural parameter Sskip can be freed from the role of controlling the memory of gradients, and is more precisely aimed to represent the relative importance of skip connection candidate operation. In contrast to Eq. 7, the output feature map expla
of edge ((.-)) can now be obtained by Eq. 4, where 3y3 Ed'eo exp(a denotes the normalized
importance, and 33 is coefficient independent from the architecture parameters.
Moreover, to eliminate the impact of auxiliary connection on the discretization procedure, we pro- pose decrease to 0 in the search phase, and method can be degenerated standard DARTS at the end of the search. Note that our method insensitive to the type of decay strategy, so we choose linear decay by default for simplicity.
gloo) (x) B3x Σ (( = B OEO
x+ Σ opskip
3lt5) olx)
(4)
We then analyze how the auxiliary skip connection handles the issue of gradient flow. Referring to the theorem of a recent work by Zhou et al. (2020), the convergence network weight W in the supernet can heavily depend on Bskip- Specifically, suppose only three operations (none, skip connection, and convolution) are included the search space and MSE loss is utilized as training loss, when architecture parameters Bi are fixed optimize W via gradient descent, training loss can decrease by ratio (1 m)/4) at step with probability at least 1 8, where 7) is the learning rate that should be bounded by 6, and follows Eq. 5.
(((..)) 2 skip
A
cone
II E
where h is the number of layers of the supernet. From Eq. 5, observe that À relies much on Bakip than Beone, which indicates that the network weights W can converge faster with a large Sakip- However, by involving an auxiliary skip connection weighted 3, Eq. 5 can be refined as follows:
h-2
2 i-1
A x
(B(CA-1) Poone
II (=0
((6.5) shop
+ 8) 8)²
(6)
where B >> Sekip making A insensitive to Baksps that the convergence of network weights W depends more on Bcone. In the beginning of the search, the common value for fskip is 0.15 while /3 is 1.0. From the view of convergence theorem (Zhou et al., 2020), the auxiliary skip connection alleviates the privilege of Baksp and equalize the compe- tition among architecture parameters. Even when 3 gradually decays, the fair competi- tion still holds since network weights W have been converged to an optimal point. Con- sequently, DARTS- is able to stabilize the search stage of DARTS.
Algorithm 1 DARTS-
Require:
Network weights w; Architecture parameters o;
Number of search epochs E;
Decay strategy for Be, E 21, E).
Ensure:
Searched architecture parameters a.
I: Construct super-network by stacking cells in which there an auxiliary skip connection be- tween every two nodes of choice for each [1, F do
Update weights by Vw.Ctrain (w, Ot, 3x)) Update parameters a by aa (w, Ot, Be)
5: end for
Extensive experiments are performed to demonstrate the efficiency of the proposed auxiliary skip connection, and emphasize that our method is flexible to combine with other methods to further improve the stabilization and searching performance. The overall algorithm is given in Alg. 1.
6: Derive the final architecture based on learned a from the best validation supernet.
4

---

Published as a conference paper at ICLR 2021
3.3 RELATIONSHIP TO PRIOR WORK
Our method is aimed to address the performance collapse in differentiable neural architecture search. Most previous works (Zela et al., 2020; Chen & Hsich, 2020: Liang et al., 2019) concentrate on de- veloping various criteria or indicators characterizing the occurrence of collapse. Whereas, we don't t study or rely on these indicators because they can mistakenly reject good models. Inspired by Chu et al. (2020b), our method focuses on calibrating the biased searching process. The underlying phi- losophy is simple: if the biased process is rectified, the searching result will be better. In summary, our method differs from others in two aspects: being process-oriented and indicator-free. Distinct from Chu et al. (2020b) that tweaks the competitive environment, our method can be viewed as one that breaks the unfair advantage. Morcover, we don't introduce any handcrafted indicators to represent performance collapse, thus greatly reducing the burden of shifting to different tasks.
EXPERIMENTS
4.1 SEARCH SPACES AND TRAINING SETTINGS
For searching and evaluation in the standard DARTS space (we name it as SO for simplicity). we
keep the same settings as in DARTS (Liu et al., 2019b). We follow R-DARTS (Zela et al., 2020) for their proposed reduced spaces $1- S4 (harder than S0). However, the inferred models are trained with two different settings from R-DARTS (Zela et al., 2020) and SDARTS (Chen & Hsieh, 2020). The difference lies in the number of layers and initial channels for evaluation on CIFAR-100. R-
DARTS sets 8 layers and 16 initial channels. Instead, SDARTS uses 20 and 36 respectively. For the proxyless searching on ImageNet, we instead search in MobileNet S5) proposed in FBNet (Wu et al., 2019). We use the SGD optimizer for weight and Adam (31 = 0.5 and B2 = 0.999, 0.001 learning rate) for architecture parameters with the batch-size of 768. The initial learning rate is 0.045 and decayed 0 within 30 epochs following the cosine schedule. We also use L2 regularization with le-4. It takes about 4.5 GPU days on Tesla V100. More details are provided in the appendix. We also use NAS-Bench-20 (S6) since DARTS performs severely bad. In total, we use 7 different search spaces to conduct the experiments, which involves three datasets.
search space (we name it
4.2 SEARCHING RESULTS
CIFAR-10 and CIFAR-100. Following the set- tings of R-DARTS (Zela et al., 2020), we obtain an average top-1 accuracy of 97.41% on CIFAR- 10, as shown in Table 2. Moreover, our method is very robust since out of 5 independent runs the search- ing results quite stable. The best cells found on CIFAR-10 (97.5% are shown in Figure 9 (B). Results on CIFAR- 100 are presented in Table 10 (see 1. Moreover, our method has much lower searching cost (3x less) than R-DARTS (Zela et al., 2020), where four independent searches with different regularization settings are needed to generate the best architecture. In other words, its robustness comes from the cost of more CO2 emissions.
Table I: Comparison of searched CNN in the DARTS search space on two different datasets.
Dataset
DARTS
R-DARTS (L22
Ours
C10 C100 (SO) 20.58++0.44
2.91+00.25
2.95+00211
2.63=0000
18.01+0.26
+..51
ImageNet. To further verify the efficiency of DARTS-, we directly search on ImageNet in S5 and compare our results with the state-of-the-art models under the mobile setting in Table The visualization of the architecture is given in Fig 10. DARTS-A A obtains 76.2% top- accuracy on the ImageNet validation dataset. By contrast, direct applying DARTS on this search space only obtains 66.4% (Chu et al., 2020b). Moreover, it obtains 77.8% top-1 accuracy after being equipped with auto-augmentation (Cubuk et al., 2019) and squeeze-and-excitatee (Hu et al., 2018), which are also used in EfficientNet.
MAS-Bench-225 Apart from standard search spaces, benchmarking with known optimal in a lim- ited setting is also recommended. NAS-Bench-201 (Dong & Yang. 2020) consists of 15,625 archi- tectures in a reduced DARTS- -ike search space, where it has 4 internal nodes and 5 operations per node. We compare ground-truth performance with found genotypes on various test sets. Remarkably, we achieve a new state of the art, the best of which almost touches the optimal.
method with prior work in Table 3. We search on CIFAR-10 and look up the
5

---

Published as a conference paper at ICLR 2021
Table 2: Comparison of the state-of-the-art models on CIFAR-10 (left) and ImageNet (right). On CIFAR-10 10 dataset, our average result is obtained on 5 independently searched models to assure the robustness. For ImageNet, networks in the top block are directly searched on ImageNet; the middle indicates that architectures are searched on CIFAR-10 and then transferred to ImageNet; the bottom indicates models have SE and Swish. We search SO for CIFAR- 10 and S5 for ImageNet.
Models
FL.PPs Params Top-I Top-5
Cost
Models
Params
FLOPs
Ace (%)
Cost
(M)
(M)
(%)
(%)
(GPU days)
(M)
(M)
GPU Days AmrehaNet-A (2019)
555
5.1
74.5
92.0
3150
NASNet-A (2018)
3,3
608
97.35
2000
MnnsNet-92 (2019)
388
3.9
74.79
92.1
3791
ENAS (2018)
4.6
626
97.11
FBNet C (2019)
FairNAS-A (2019h)
375
388
4.6
5.5
75.3
74.9
92.3
92.4
12
DARTS (2019h)
97.00110.
SCARLET-C (2019a)
365
6.7
76.9
93.4
(2019)
422
97.15
FairDARTS D (2020h)
440
4.3
75.6
92.6
GDAS (2019b)
3,4
5191
97.07
0.2
PC-DARTS (2020)
597
5.3
75.8
92.7
P-DARTS (20196)
3,4
5321
97,5
0.3
DARTS- (ours)
467
4.9
76.2
93.0
4.5
PC-DARTS (2020)
3.6
558
97.43
0.1
DARTS- (best)
3.5
568
97.5
NASNet-A (2018)
564
5.3
74.0
91.6
2000
DARTS (2019b)
574
4.7
73.3
91.3
0.4
P-DARTS (2019h)*
3.3.
19
(2019)
522
90.8
R-DARTS (2020)
0.
1.6
PC DARTS (2020)
586
5.3
74.9
92.2
0.1
ADARTS-ADV (2020)
3.3
0.002
1.3
FairDARTS-B (2020b)
541
4.8
75.1
92.5
DARTS- (avg.) 1. 58
97.41
0.4
MobileNet33 (2019)
219
5.4
75.2
92.2
R:3000
Based on the provided genotypes.
MoGiA-A (2020a)
304
5.1
92.8
5 independent searches using their released code.
MixNet M (2019)
360
5.0
77.0
93.3
#:3000
Training the best searched model for several times (whose average doesn't indicate the stability of the DARTS- (ours) method)
EfficientNet (2019)
390
5.3
76.3
93.2
Ra3000
NoisyDARTS-A
449
5.5
77.9
94.0
12
470
5.5
77.8
93.9
Estimated by Wu ot al. (2019).
SE modules and Swish enabled.
Table 3: Searching performance on
(Dong Yang, 2020). Our method robustly obtains new SOTA. Averaged on 4 runs of searching. 1st, first-order, 2nd, second-order Method
Cost (hours)
CIFAR-10
CIFAR-100 100
ImageNet1 120 test
valid
test
valid
test
valid
DARTS (2019b)
3.2 39.77=0.00 54.30+0.00
15.03++0.00
15.61=0.000
16.43=0.00
16.32:0.00
Dad (2019h)
10.2 39.77.10.00
54.30-10.00
15.03:10.00
15.61 16.43.1.000
16.32.10.00
GDAS (2019b)
8.7 89 89.89=00008 93.61+0.09
71.34++0.04
70.70=0.30
41.59=1.33
41.71=:0.98
SETN (2019a)
9.5 84.04=0.28
87.6440.00
58.86+00.06
59.05=#0224
33.06=10.02
32.52=0.21
DARTS-
3.2 91.03=10.44
93.80=0.40
1..36::1.51
71.53=
44.87=1446
45.12+:0822
DARTS- (best)
3.2
91.55
94.36
73.49
73.51
46.37
46.34
optimal
n/a 91.61
94.37
73.49
73.51
46.77
47.31
Transfer results on objection detection We further evaluate the transfer-ability of our models down-stream object detection task by replacing the backbone of RetinaNet (Lin et al., 2017) on MMDetection toolbox platform (Chen et al., 2019a). Specifically, with the same training setting as Chu et al. (2020b), our model achieves 32.5% mAP on the COCO dataset, surpassing other similar- sized models such as MobileNetV3, MixNet, and FairDARTS. The detailed results are shown in Appendix (Table 11).
4.3 ORTHOGONAL COMBINATION WITH OTHER VARIANTS
Our method can be flexibly adapted to combine with prior work for further improvements. Here we investigate the joint outcome with two methods: P-DARTS and PC-DARTS.
Progressive DARTS (P-DARTS). P-DARTS (Chen et al., 2019b) proposes a progressive approach to search gradu- ally with deeper depths while pruning out the uncompeti- tive paths. Additionally, it makes use of some handcrafted criteria to address the collapse (the progressive idea itself cannot deal with it), for instance, they impose two strong priors by regularizing the number of skip connections M as 2 as well as dropout. To be fair, we remove such a carefully handcrafted trick and run P-DARTS for several
We remove the strong con- straints on the number of skip connec- tions as 2 and dropout (priors) for P- DARTS and compare its performance w/ and w/o DARTS-
Table 4:
Method Setting
Acc (%)
P-DARTS w/o priors 96.48.1..5
P-DARTS- w/o priors 97.28.10.04

---

Published as a conference paper at ICLR 2021
Table 7: Comparison in various search spaces. We report the lowest error rate of found archi- tectures. 1; under Chen & Hsich (2020)'s training settings where all models have 20 layers and 36 initial channels (the best is shown in boldface). t: under Zela et al. (2020)'s settings where CIFAR-100 models have 8 layers and 16 initial channels (The best is in boldface and underlined).
R-DARTS'
DARTS'
SDARTS
Benchmark DARTS
Ourst
PC-DARTS'
Ours
DP
ES
ADA
RS
ADV
S1
3.84
3.11
2.78
3.01
3.10
2.68
3.11
2.78
2.73
2.68
S2
4.85
3.48
3.31
3.26
3.35
2.63
3.02
2.75
2.65
2.63
C10
S3
3.34
2.93
2.51
2.74
2.59
2.42
2.51
2.53
2.49
2.42
S4
7.20
3.58
3.56
3.71
4.84
2.86
3.02
2.93
2.87
2.86
S1
29.46
25.93
24.25
28.37
24.03
22.41
18.87
17.02
16.88
16.92
S2
26.05
22.30
22.24
23.25
23.52
21.61
18.23
17.56
17.24
16.14
C100
S3
28.90
22.36
23.99
23.73
23.37
21.13
18.05
17.73
17.12
15.86
S4
22.85
22.18
21.94
21.26
23.20
21.55
17.16
17.17
15.46
17.48
times. As a natural control group, we also combine DARTS- with P-DARTS. We run both exper- iments for 3 times on CIFAR-10 10 dataset in Table 4. Without the strong priors, P-DARTS severely suffers from the collapse, where the inferred models contain an excessive number of skip connec- tions. Specifically, it has very high test error (3.42% average), even worse than DARTS. How- ever, P-DARTS can benefit greatly from the combination with DARTS- The improved version (we call P-DARTS-) obtains much higher top-1 1 accuracy (+0.8 %) CIFAR-10 10 than its baseline.
Memory Friendly DARTS (PC-DARTS). To alle- viate the large memory overhead from the whole su- pernet, PC-DARTS (Xu et al., 2020) selects the par- tial channel for searching. The proportion hyper- parameter K needs careful calibration to achieve a good result for specific tasks. As byproduct, the search time is also reduced to 0.1 GPU days (K=4). We use their released code and run repeated experi- ments across different seeds under the same settings.
Table 5: Comparison of PC-DARTS remov- ing the strong prior (i.e. channel shuffle) and combining DARTS- The results are from independent runs on CIFAR-10. The GPU memory cost is on a batch size of 256.
Method
Setting Ace (%) Memory Cost
PC- PC-DARTS K = 2 97.0940.14 19.9G 3.75h
PC-DARTS- K = 2 97.35+00.02 20.8G 3.41h
To accurately evaluate the role of our method, we choose K=2 (a bad configuration in the original paper). We make comparisons between the original PC-DARTS and its combination with ours (named PC-DARTS-) Table 5. PC-DARTS- can marginally boost the CIFAR-10 top-1 accuracy (+0.26% on average). The result also confirms that our method can make PC-DARTS less sensitive to its hyper-parameter K while keeping its advantage of less memory cost and run time.
4.4 ABLATION STUDY
Robustness to Decay Strategy. Our method is insensitive to the type of decay policy on B. We design extra two strategies comparisons: cosine and step decay. They both have the similar performance. Specifically, when 3 is scheduled to zero by the cosine strategy, the average accuracy of four searched CIFAR-10 10 models in S3 is 9.009, the best is 97.47%. The step decay at epoch 45 obtains 97.30% top-1 accuracy on average in the same search space.
Robustness Comparison on C10 and C100 in S0-S4. To verify the robustness, it is required to search several times to report the average performance of derived models (Yu al., 2020; Antoine et al., 2020). As shown in Table 1, Table 9 and Table 7, our method outperforms the recent SOTAS across several spaces and datasets. Note that SDARTS-ADY utilizes adversarial training and requires 3x more search time than ours. Especially, we find good model in Sy on CIFAR-100 with the lowest t test error 15.86% The architectures of these models can be found in the appendix.
Table 6: Searching performance on CIFAR-I0 in S3 w.r.t the initial lin- ear decay rate 3o. Each setting is run for three times.
Bo
Error (%)
1
2.65
0.7
2.76=0.11
0.4
3.04
0.1
0.11
0
1.5
Sensitivity Analysis of 3 The power of the auxiliary skip connection branch can be discounted by setting lower initial B. We now evaluate how sensitive
7

---

Published as a conference paper at ICLR 2021
approach is to the value of B. It's trivial to that our approach degrades to DARTS when 3 = We compare results when searching with 3 E 0...7,0.44001... in Table 6, which show that bigger in advantageous to obtain better networks.
The choice of auxiliary branch Apart from the default skip connection serving as the auxiliary branch, we show that it is also effective replace it with learnable 1 convolution projection, which is initialized with an identity tensor. The average accuracy of three searched CIFAR-10 models in S3 97.25%%10.00 here works in a similar way as the proposed skip connection. This proves the necessity of the auxiliary branch.
Akin the ablation in He et al. (2016), the projection convolution
Performance with longer epochs. It is claimed by Bi et al. (2019) that much longer epochs lead better convergence of the supernet. supposedly beneficial for inferring the final models. However, many of DARTS variants fail since their final models are full of skip connections. We thus evaluate how our method behaves in such a situation. Specifically, we extend the standard 50 epochs 200 and we search 3 independent times each for SO, S2 and S3, Due to the longer epochs, we slightly change our decay strategy, we keep 3 all the way until for last 50 epochs we decay 3 to O. Other hyper-parameters are kept unchanged. The results are shown in Table 8 and the found genotypes are listed in Figure 17, 18, 19, 20 and 21. It indicates that DARTS- doesn't 't suffer from longer epochs since it has reasonable values for #P compared with those (#P = 0) investigated by Bi et al. (2019). Notice S2 and S3 are harder cases where DARTS suffers more severely from the collapse than SO. As result, DARTS- spaces. Noticeably, it is still unclear whether longer epochs can truly boost searching performance. Although achieve new state-of-the-ar result in S2 where the best model has 2.50% error rate (previously 2.63%), ), it still has worse average performance (2.71 11%) in SO than the models searched with 50 epochs (..59+0.08%), and the best model S3 (2.53%) also weaker than before (2.42%).
150,
successfully survive longer epochs even in challenging search
Table 8: Searching performance CIFAR-10 in SO, S2 and S3 using longer epochs. Following Bi et al. (2019), # means the number parametric operators in the normal cell. Averaged on 3 runs of search.
Epoch
SO
$2
S3
#
Params (M)
Error (%)
#P
Params (M)
Error (%)
#
Params (M) Error (%)
150
1.1 3.3 2.74 6..0 3.9+:0.3 2.50-0.11660011.0 3.6 2.05 4.3 2.66
200
7.6 3.2 2. 8.0 4.3 2.65=0.21
Besides, compared with first-order DARTS with a cost of 0.4 GPU days in SO, Amended-DART'S (Bi et al., 2019), particularly designed to survive longer epochs, reports 1.7 GPU days with pruned edges in SO. Our approach has the same cost as first-order DARTS, which is more efficient.
ANALYSIS AND DISCUSSIONS
5.1 FAILURE OF HESSIAN EIGENVALUFE
The maximal Hessian eigenvalue calculated from the validation loss w.r.t a is regarded as an indi- cator of performance collapse (Zela al., 2020; Chen & Hsieh, 2020). Surprisingly, our method develops growing eigenvalue in the majority of configurations, which conflicts with the previous observations. We visualize these statistics across different search space and datasets in Figure (A.2.2). Although cigenvalues increase almost monotonously and reach a relatively large value in the end, the final models still have good performance that matches with state-of-the of-the-ar art ones (see Ta- ble 9). These models can be mistakenly deemed as bad ones never visited according to the eigen- value criteria. Our observations disclose one fatal drawback of these indicator- based approaches: they are prone to rejecting good models. Further analysis can be found in A.2.2.
8

---

Published as a conference paper at ICLR 2021
5.2 VALIDATION ACCURACY LANDSCAPE
Recent works, R-DARTS (Zela et al., 2020) and SDARTS (Chen & Hsieh, 2020) point that the ar- chitectural weights expected converge optimal point where accuracy is insensitive per- turbations to obtain a stable architecture after the discretization process, i.e., the convergence point should have a smooth landscape. SDARTS proposes perturbation-based regularization, which further stabilizes the searching process of DARTS. However, the perturbation regularization dis- turbs the training procedure and thus misleads the update of architectural weights. Different from SDARTS that explicitly smooths landscape by perturbation, DARTS- can implicitly do the same without directly perturbing architectural weights.
(a) DARTS
DARTS-
(c) DARTS
(d) DARTS
Figure 3: Comparison of the validation accuracy landscape of (a) DARTS and (b) DARTS- w.r.t. on CIFAR-10 S3. Their contour maps are shown respectively (c) and (d), where we set the step of contour map 0.1. The accuracy of the derived models are 94.84% (a,c) and 97.58% (b,d), while the maximum Hessian eigenvalues are similarly high (0.52 and 0.42)
To analyze the efficacy of DARTS-, we plot the validation accuracy landscape w.r.t architectural weights o, and find that auxiliary connection smooths the landscape and thus stabilizes the search- ing stage. Specifically, we choose two random directions and apply normalized perturbation on a (following Li et al. 2018a). As shown in Figure 3, DARTS is less sensitive to the perturbation than DARTS, and the contour map of DARTS- descends more gently.
6 CONCLUSION
We propose a simple and effective approach named DARTS- address the performance collapse in differentiable architecture search. Its core idea is to make use of an auxiliary skip connection branch take over the gradient advantage role from the candidate skip connection operation. This can create fair competition where the bi-level optimization process can casily differentiate good oper- ations from the bad. As result, the search process is more stable and the collapse seldom happens across various search spaces and different datasets. Under strictly controlled settings, it steadily outperforms recent state-of-the-arr RobustDARTS (Zela et al., 2020) with 3x fewer search cost. Moreover, method disapproves of various handcrafted regularization tricks. Last but not least, can be used stand-alone or in cooperation with various orthogonal improvements necessary.
This paper conveys two important messages for future research. On the one hand, the Hessian eigenvalue indicator for performance collapse (Zela et al., 2020; Chen & Hsich, 2020) is not ideal because it has risk of rejecting good models. On the other hand, handcraft regularization tricks (Chen et al., 2019h seem more critical to search a good model instead the proposed methods. Then what's the solution? In principle, it's difficult find perfect indicator of the collapse. Our approach shows the potential to control the search process and doesn't 't impose limitations or priors on the final model. We hope more attention be paid this direction.
7 ACKNOWLEDGEMENT
This research was supported by Meituan.

---

Published as a conference paper at ICLR 2021
REFERENCES
Yang Antoine, Esperança Pedro M., and Carlucci Fabio M. NAS Evaluation Frustratingly Hard. In ICIR, 2020.
Kaifeng Bi, Changping Hu, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Stabilizing darts with amended gradient estimation on architectural parameters. arXiv preprint arXiv:1910.1183. 2019.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In ICLR, 2019.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. MMDetection: Open mmlab detection toolbox and benchmark. 1906:0990.555 2019a.
arXiv preprint
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In JCML, 2020.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive Differentiable Architecture Search: Bridg- ing the Depth Gap between Search and Evaluation. In ICCV, 2019b.
Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. Scarletnas: Bridging the gap between scalability and fairness in neural architecture search. arXiv preprint arXiv: 06022, 2019a.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. arXiv preprint. crXiv:1907.01845 2019b.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. MoGA: Searching Beyond MobileNerV3. In ICASSP, pp. 4042-4046, 2020a.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. In ECCV, 2020b.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning Augmentation Policies from Data. In CVPR, 2019.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3681-3690, 2019a.
Xuanyi Dong and Yi Yang. Searching for a Robust Neural Architecture in Four GPU Hours. In CVPR, pp. 1761-1770, 2019b.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020. URL https: Huyrruumiid--uxxyxxxaexx
/ //openreview.....//m/../.e.e...e
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe V Le. NAS-FPN: Learning scalable feature pyramid archi- tecture for object detection. CVPR, pp. 7036-7045, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pp. 770-778, 2016.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chit Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for MobileNetV3. In ICCV, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-Excitane Exciitatio Networks. In CVPR, pp. 7132-7141, 2018.
Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Müller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In CVPR, 2020.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in Neural Information Processing Systems, pp. 1389-6399, 2018a.
10

---

Published as a conference paper at ICLR 2021
Juncheng Li, Faming Fang, Kangfu Mei, and Guixu Zhang. Multi-scale residual network for image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 517-532, 2018b.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. DARTS+: Improved Differentiable Architecture Search with Early Stopping, arXiv preprint wxXi::999.00003, 2019.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. In ICCV, pp. 2980-2988, 2017.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive Neural Architecture Search. In ECCV, pp. 19-34, 2018.
Chenxi Liu, Liang-Chis Chigh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, pp. 82-92, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In ICLR, 2019b.
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang. Alan Yuille, and Yang Jianchao. AtomNAS: Fine-Grained End-to-End Neural Architecture Search. In ICLR, 2020.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient Neural Architecture Search via Parameter Sharing. In ICML, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, volume 33, pp. 4780-4789, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object
detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bileNetV2: Inverted Residuals and Linear Bottlenecks. In CVPR, pp. 4510 4510-4520, 2018.
David So, Quoc Le, and Chen Liang. The evolved transformer. In ICMI., pp. 5877-5886, 2019.
Dimitrios Stamoulis, Ruizhou Ding, Wang. Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-Path NAS: Designing Hardware-Efficceet ConvNets in less than 4 Hours. ECML PKDD, 2019.
Ying Tai, Jian Yang, and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3147- 3155, 2017.
Mingxing Tan and Quoe V Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML, 2019.
Mingxing Tan and Quoc V. Le. MixConv: Mixed Depthwise Convolutional Kernels. BMVC, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform- Aware Neural Architecture Search for Mobile. In CVPR, 2019.
Yancong Wei, Qiangqiang Yuan, Huanfeng Shen, and Liangpei Zhang. Boosting the accuracy of multispectral image pansharpening by learning deep residual network. IEEE Geoscience and Remote Sensing Letters, 1795-19999 2017.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. In CVPR, 2019.

---

Published as a conference paper at ICLR 2021
Sirui Xie, Hehui Zheng. Chunxiao Liu, and Liang Lin. SNAS: Stochastic Neural Architecture Search. In ICLR,, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong, PC- DARTS: Partial Channel Connections for Memory-Efficient Architecture Search. In ICLR, 2020. URL http::///ppeeeeeewwwe net / forum? id=BJ1s634tPP
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In ICLR, 2020. URL httph::/ fost/forum?id-i11 1oo2:11o2d1Fwww..
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter.
Understanding and robustifying differentiable architecture search. In ICLR, 2020. URL https:
/popnneeeieww... net/forum?iddit d=w.mmiiiiiillmmmm gDNykkDS..
Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Theory-inspired path-regularized differential network architecture search. arXiv preprint arXiv:2006. 16537, 2020.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning Transferable Architec-
tures for Scalable Image Recognition. In CVPR, volume 2, 2018.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 PRELIMINARY ABOUT DARTS
In differentiable architecture search (Liu et al., 2019b), a cell-based search space in the form of Di- rected Acyclic Graph (DAG) is constructed. The DAG has two input nodes from the previous layers, four intermediate nodes, and one output node. There are several paralleling operators (denoted as O) between each two nodes (say i, f), whose output r(اا) given an input x defined as,
exp(a!"
σ (z) (x)= Σ DEO Eoreo exp(a (10)) o(x)
is essentially applying softmax over all operators where each operator is assigned with an archi- tectural weight a. A supernet is built two kinds of such cells, so-called normal cells and reduction cells (for down-sampling). The architectural search is then characterized as a bi-level optimization:
min ( (a),
s.t. (aa)) arg min Lerain (10, o:)
This indicates that the training of such a cell-based supernet should be interleaved, where at each step the network weights and architectural weights are updated iteratively. The final model is determined by simply choosing operations with the largest architectural weights.
A.2 EXPERIMENT
A.2.1 TRAINING DETAILS
CIFAR-10 and CIFAR-100. Table 9 gives the averaged performance in reduced search spaces S1-S4, as well with maximum eigenvalues. Table 10 reports the CIFAR-100 results in SO.
Table 9: Comparison of searched CNN architectures in four reduced search spaces S1-S4 (Zela al., 2020) on CIFAR-10 and CIFAR- 100. We report the meantstt of test error over 3 found architectures retrained from scratch, alongside with eigenvalue (EV) that corresponds to the best
validation accuracy. We follow the same settings as Zela et (2020).
Benchmark
DARTS
DARTS-ES
DARTS-ADA
ours
ours (EV)
S1
4.66110.71
3.05:10.07
3.03110.8
2.76:10.07
0.37:0.19
S2
4.4210.40
3.4110.114
3.59-1.31
2.79-10.04
0,41
C10
S3
4.1210.85
3.71111.14
2.99-0.34
2.65:10.04
0.31
S4
6.95100118
4.1710.21
3.89-0.67
2.91
0.30
S1
29.93.10.41
28.9010.81
24.94.10.81
23.26.10.59
0.60-10.15
S2
28.75.10.92
1..68
26.88
22.31
0.5410..15
C100
S3
29.01
26.99-1.79
24.55+00.3
21.47+0.40
0.40=0.03
S4
2..77
23.90=2.01
23.66=20.90
21.75:10.26
0.93=0..5
ImageNet classification. For training on ImageNet, we use the same setting as MnasNet (Tan al., 2019). To be comparable with EfficientNet (Tan & Le. 2019), also use qqueezeaand-xxciaaion (Hu et al., 2018). Furthermore, we don't include methods trained using large model distillation, because it can boost final validation accuracy marginally. To be fair, we don't 't use the efficient head in MobileNetV3 (Howard et al., 2019) although it can reduce FLOPs marginally.
COCO object detection. All models are trained and evaluated on MS COCO dataset for 12 epochs with a batch size of 16. The initial learning rate 0.01 and reduced by 0.1 at epoch and 11.
A.2.2 FURTHER DISCUSSIONS ON FAILURE OF EIGEN VALUE
To further explore the relationship between the searching performance and Hessian Eigenvalue, we plot the performance trajectory of the searched models in Figure (b). Specifically, sample
13

---

Published as a conference paper at ICLR 2021
Table 10: Comparison of searched models on CIFAR-100. o Reported by Dong Yang (2019b),
*: Reported by Zela et al. (2020), R:Rerun their code.
Models
Params Error
Cost
(M)
(%)
GPU Days
ResNet (2016)
1.7
22.10
AmocbaNet (2019)
3.1
18.93
3150
PNAS (2018)
3.2
19.53
150
ENAS (2018)
4.6
19.43
0.45
DARTS (2019b)
20.58+L0.44*
0.4
GDAS (2019h)
3.4
18.38
0.2
P-DARTS (2019h)
3.6
17.49
0.3
R-DARTS (2020)
18.01
1.6
DARTS- (avg.)
3.3
17.51+00.25
0.4
DARTS- (best) 3.4
17.16
0.4
Table 11: Transfer results on COCO datasets of various drop-in backbones.
Backbones
Params (M)
Acc
AP
APsa
AP
APs
APM
AP2
MobileNetV2 (2018)
3.4
72.0
28.3
46.7
29.3
14.8
30.7
38.1
SingPath NAS (2019)
4.3
75.0
30.7
49.8
32.2
15.4
33.9
41.6
MnasNet-AA2 (2019)
4.8
75.6
30.5
50.2
32.0
16.6
34.1
41.1
MobileNetV3 (2019)
5.4
75.2
29.9
49.3
30.8
14.9
33.3
41.1
MixNet-M (2019)
5.0
77.0
31.3
51.7
32.4
17.0
35.0
41.9
FairDARTS-C (2020b)
5.3
77.2
31.9
51.9
33.0
17.4
35.3
43.0
DARTS-A (Ours)
5.5
77.8
32.5
52.8
34.1
18.0
36.1
43.4
models every 10 epochs and train these models from scratch using the setting as above (Fig- ure 5). The performance of the inferred models continues growing, where the accuracy is boosted from 96.5% 97.4%. This affirms the validity of searching using our method. In contrast, the carly-stopping strategies based on cigenvalues (Zela et al., 2020) would fail in this setting, We ar- gue that the proposed auxiliary skip branch can regularize the overfitting of the supernet, leaving the architectural weights to represent the ability of candidate operations. This experiment poses as a counterexample R-DARTS, where good models can appear albeit Hessian eigenvalues change fast. It again denies the need for costly indicator.
(a) Trajectory of eigenvalues in SO-S4 (left to right) on CIFAR-10
(b) Sampled models' performance
Bonh
Eprs
Trajectory of eigenvalues in S0-S4 (left right) CIFAR-100
Figure 4: The evolution of maximal eigenvalues of DARTS- when searching in different search spaces SO-S4 on CIFAR-10 (a) and CIFAR-100 (c). We run each experiment times on different seeds. (b) DARTS-'s growing Hessian eigenvalues don't induce poor performance. Among the sampled five models, the one corresponding to the highest eigenvalue has the best performance. This example done in SO CIFAR-I0.
14

---

Published as a conference paper at ICLR 2021
34-
ceddiia 10.0 spec.
521
540
300
393
930
madkiiat 23-4 spech
mak.iat 3344 sped.
mmk.-.t qped.
meaio speal
Figure 5: Training five models sampled every 10 epochs during DARTS- searching process. See Fig 4 (b) for the corresponding eigenvalues.
A.2.3 MORE ABLATION STUDIES
To supplement the sensitivity analysis in Section 4.4, Figure 6 shows the training loss curve when initializing with different values.
CIFAR-10, S3
B-1.0
8-0.7
8-0.4
A-0.1
Lpaid 0
8-0.
0.5
spach
Figure 6: The training loss curve of the over-parameterized network on CIFAR- in S3 with differ- ent initial Sy.
A.3 LOSS LANDSCAPE
To show the consistent smoothing capacity of DARTS-, draw more loss landscapes in SO and S3 (on several seeds) w.r.f architectural parameters and their contours in Figure and 8. Generally, DARTS-'s slopes more inflated if consider them camping tents, which suggest better convergence of the over-parameterized network.
A.4 LIST OF EXPERIMENTS
We summarize all the conducted experiments with their related figures and tables in Table 12.
B FIGURES OF GENOTYPES
15

---

Published as a conference paper at ICLR 2021
(a) DARTS
(b) DARTS
(c) DARTS
(d) DARTS
(0) DARTS-
(I) DARTS-
DARTS
(h) DARTS
Figure 7: More visualization of validation accuracy landscapes of DARTS (a.b) and DARTS- (c.f) w.r.t. the architectural weights a on CIFAR- 10 in SO. Their contour maps are shown respectively in (c.d) and (g.h). The step of contour map is 0.1. The inferred models by DARTS- have higher accuracies (97.50%, 97.49%) than DARTS (97.19%, 97.20%)
(a) DARTS
(b) DARTS
(c) DARTS
(d) DARTS
DARIS-
DARTS-
(g) DARTS
(h) DARTS-
Figure 8: More visualization of validation accuracy landscapes of DARTS (a,b) and DARTS- (e.f) w.r.t. the architectural weights a CIFAR- in S3. Their contour maps are shown respectively in (c.d) and (g.h). The step of contour map is 0.
Table 12: List of experiments conducted in this paper
Method
Search Space
Dataset
Figures
Tables
Main text
Supp.
Main text
Supp.
DARTS
SO
CIFAR-10
DARTS
$3
CIFAR-10
3
DARTS-
SO
CIFAR-10 10
4,5,9,7
1,2
DARTS-
S1-S4
CIFAR-10 10
3
4,6,9,8
7,9
6
DARTS-
S5
ImageNet
5.10
DARTS-
COCO
DARTS-
S6
CIFAR-10
3
DARTS-
S1-S4
CIFAR-100
4,11
7
DARTS-
S0
CIFAR-100
4.11
1.10
P-DARTS M=2
CIFAR-I0
12
P-DARTS w/ auxiliary skip
SO
CIFAR-10
13
4
PC-DARTS w/o channel shuffling
SO
CIFAR-10 10
14
5
PC-DARTS w/ auxiliary skip
SO
CIFAR-10 10
15
5
16

---

Published as a conference paper at ICLR 2021
sep3
stip
Mp3
c+
NIDS
slip
skip
up3
(a) SO
(d) $3
(c) S4
Figure The best found normal cell and reduction cell in search spaces S0-$4 on CIFAR-14 dataset.
234-2200
1124112-32
112-112-14
30-23-00
14:24000
7:740
7-74120 200
MRE6 K3
MRE6 K3
MBEE K3
MBEE K3
MBKS ks
MBRS JC
MBEE KC
MBE6, K3
MBE6, KT
MBF6, K3
MBF6 KT
MBF6, KT
GPool FC2
Skem
PCI
Figure 10: Architecture of DARTS-A searched on ImageNet dataset.
17

---

Published as a conference paper at ICLR 2021
2013
scpS
maxs
skip
skip
seps
skip
kkip
SO
(b) S1
(c) S2
PORS
DODO
(d) S3
(c) S4
The best found normal cell and reduction cell in search spaces S0-S4 on CIFAR-100 dataset.
Figure 11:
dcp
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(1)
Reduction
Figure 12: Found normal cells and reduction cells by P-DARTS (Chen al., 2019b) without prior (M=2) in the DARTS standard search space on CIFAR-10 10 dataset.
18

---

Published as a conference paper at ICLR 2021
(a) Normal
(b) Reduction
Normal
Reduction
XBC
(c) Normal
(f) Reduction
Figure 13: Found normal cells and reduction cells by P-DARTS (Chen et al., 2019b) with the proposed auxiliary skip connections in the DARTS' standard search space on CIFAR- 10 dataset.
Scp3
TEO
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(I) Reduction
Figure 14: Found normal cells and reduction cells by PC-DARTS (Xu et al., 2020) without channel shuffling in the DARTS' standard search space on CIFAR-10 dataset.
19

---

Published as a conference paper at ICLR 2021
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(1) Reduction
Figure 15: Found normal cells and reduction cells by PC-DARTS (Xu et al., 2020) with the pro-
posed auxiliary skip connections in the DARTS' standard search space on CIFAR-10 dataset.
(a) Normal
(b) Reduction
Normal
Reduction
(e) Normal
(f)
Reduction
Figure 16: Keep Sakip 1 throughout the DARTS- searching in the DARTS' standard search space
on CIFAR-10 dataset.
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
Figure 17: Best cells found when decaying Bakip in the last 50 epochs during the DARTS- searching for 150 and 200 epochs respectively in the DARTS search space on CIFAR-10.
20

---

Published as a conference paper at ICLR 2021
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(f) Reduction
Figure 18: Decaying Bakip in the last 50 epochs during the DARTS- searching for 150 epochs in S2 on CIFAR-10 dataset.
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(f) Reduction
Figure 19: Decaying Pokip in the last 50 epochs during the DARTS- searchin for 200 epochs in S2 on CIFAR-10 datasct.
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(f) Reduction
Figure 20: Decaying Bakip in the last 50 epochs during the DARTS- searching for 150 epochs in S3 on CIFAR-10 dataset.
21

---

Published as a conference paper at ICLR 2021
NEL
(a) Normal
(b) Reduction
(c) Normal
(d) Reduction
(e) Normal
(I) Reduction
Figure 21: Decaying Sakip in the last 50 epochs during the DARTS- searchin for 200 epochs in S3 on CIFAR-10 dataset.
22


Reviews:
[{"review_id": "et15Bdbr-qa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on improving the robustness of differentiable architecture search models. I am not an expert in the topic, so please see my review as an outsider's perspective. Although the problem is clearly formulated and motivated, I don't really understand the actual contribution of the paper. There is no intuitive explanation of the approach and why it should work. I hope the other reviewers are expert in the topic and can comment on this aspect, but as an outsider, I think the paper is poorly written (the English is clear, it's just the way they explain the solution and contribution) and the contributions are not clear.\n\nI tried to look at the prior work (mostly those mentioned in the paper and frequently cited) and based on this quick research, it seems that the approach is novel. However, again, I do believe that at least an expert view on this aspect is necessary. I am not sure if I know all the prior work.\n\nThe experimental results are encouraging. They show robustness across datasets. I think this is a strong paper from an empirical perspective. \n\nSince I do not fully understand the contributions and the authors didn't convince me on why this is a novel approach, I'd vote for marginally above acceptance threshold (mainly because of strong evaluation). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sxscak9yIiU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting method to alleviate the mode collapse of DARTS (all operations degenerate to skip-connect). This is done by simply adding a skip-connect operation to complement the output of the cell function and making the coefficient of the auxiliary operation decay with time. The method is tested on a few benchmarks and settings.\n\nOverall, I think the idea is neat and interesting, but more experiments need to be done to *support the claim of this paper*.\n\n1. The statement that \"skip-connect has two roles, one for auxiliary connection and one for candidate operation\" is interesting, but not well justified. According to the authors' claim, the skip-connect operator start to dominate the cell structure after a long training procedure *because* the role of auxiliary connection becomes more and more important along with the training procedure. This is not supported by either [Zela, ICLR'20] or [Bi, 1910.11831], both of which claimed that the dominance of skip-connect comes from the incorrectly computed gradients of DARTS. *It is highly suggested to discuss this point.* Moreover, to justify the statement, an experiment is easily done: keep the auxiliary skip-connect throughout the search procedure (\\beta==1), and observe if mode collapse still happens. If not, please provide the results and the guess will become more convincing.\n\n2. The claim of \"preventing mode collapse\" is not well justified using experiments. For example, if the search procedure (on CIFAR10) is extended to 200 epochs or longer, is it possible to avoid mode collapse using the proposed method? It would be interesting to list a few results with varying lengths of the training procedure.\n\n3. Some experimental results need explanation.\n\n(1) In ImageNet experiments, the authors reported a 76.2% accuracy with 467M parameters. This really surprised me, before I checked the searched architecture and noticed that this architecture was obtained from the MobileNet space. Provided the name of DARTS-, this is VERY confusing, seemingly to guide the readers to believe that this is obtained in the DARTS space. I urge the authors to try their best to avoid such misunderstanding.\n\n(2) (minor) In the ablation study, the comparison against P-DARTS and PC-DARTS is weird. Even without the constraint of 2 skip-connect operations, P-DARTS does not report 96.48% on CIFAR10, even lower than the random search baseline.\n\n(3) (minor) Since a \"bad\" hyper-parameter of PC-DARTS (K=2) is used, I am wondering how the results are like if K=4 is used. BTW, the PC-DARTS paper reported 97.3% accuracy using K=2, but this paper reports 97.1%.\n\n\nI am now neutral on this paper, slightly leaning towards rejection. Provided more complete results, I will consider changing my score.\n\nAFTER READING THE REBUTTAL, I changed my score from 5 to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "72v29HFFMPf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to improve the robustness of DARTS, and proposes to add an auxiliary skip connection branch to the “mixOp” in cells. The authors also analyze the effect of auxiliary branch on residual block from the view of gradient flow. Additionally, this paper refers to the theory of the work [1] and demonstrate that the auxiliary branch is able to reduce the dependence of the convergence of network weight on $\\beta_{skip}$ so as to disentangle the “two-fold role” of skip connection: “as an auxiliary connection to stabilize the supernet training, and as a candidate operation to build the final network”. Moreover, extensive experiments on multiple search spaces and datasets are conducted, showing the effectiveness of this method. They also illustrate the accuracy landscape w.r.t the architecture parameters to demonstrate that the auxiliary branch is able to smoothen the landscape, making DARTS- less sensitive to the perturbation than DARTS\n\nI have two questions: \n1). I notice that the authors adopt decay strategy on the weight of auxiliary branch ($\\beta$), and claim that their method is insensitive to the type of decay policy. I think the authors find an interesting phenomenon of skip connection, and I wonder how $\\beta$ affects the searching procedure? And can the effect of auxiliary branch holds or how long can the effect holds after $\\beta$ decays to 0. Specifically, suppose we search for 50 epochs with auxiliary branch, then we remove the auxiliary branch and search for another 50 epochs, will DARTS collapse?\n2). I agree with the authors’ point that skip connection “plays two-fold roles”. However, I wonder with the auxiliary skip connection suppress the architecture parameter of skip connection ($\\beta_{skip}$), so that no skip connections are chosen in the final model?\n\nIn general, I think this paper propose a simple but efficient method to alleviate the performance collapse of DARTS, the strengths and weaknesses are listed as follows:\n\n\nStrengths:\n1). The authors propose a simple but efficient indicator-free method to prevent skip connection from dominating the superNet. They also demonstrate the effectiveness of auxiliary branch from the view of gradient flow and the convergence of network weight. \n2). Extensive experiments on multiple search spaces and datasets show the effectiveness of the method.\n3). The method can combine with DARTS variants to further improve the performance.\n\n\nWeaknesses:\n1). I am sort of concerned that the auxiliary skip connection may suppress the weight of original skip connection. And I hope the authors will have further analysis.\n2). I wonder what is the meaning of “DARTS-”since the method actually add an auxiliary skip connection stead of removing any connections or operations. I strongly suggest the authors change the name into “regDARTS” (regularized), “gradDARTS” (graduated) etc.\n\n\n[1] Zhou, Pan, et al. \"Theory-inspired path-regularized differential network architecture search.\" arXiv preprint arXiv:2006.16537 (2020).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4vyb_Jq8POh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper reveals two roles of skip connection to prevent model collapse: stabilize the supernet training, and as a candidate operation to build the final network. Intuitively, the skip connection playing the first role should only be there during the training phase. Therefore, the authors propose to add an auxiliary skip connection to play that role. This auxiliary skip connection is decayed gradually during training. The paper provides an interesting theoretical analysis that this can help prevent the gradient vanishing problem. \nCompared to other approaches, the proposed one has the advantage that it does not rely on any heuristic indicator. Indeed, it is found that the existing indicator based on Hessian eigenvalue can discard good models.\nIn the experiments, this method - DART-, is compared with DART and several other approaches, and show that the proposed method can outperform the others, however, by small margins. \n\npros:\nThe assumption that skip connections play two different roles is very interesting and inspiring. It is indeed the case that such skip connections not only can stabilize the training, but are part of the model. The idea to separate the two roles is well motivated.\nThe paper provides some interesting theoretical analysis to show the potential impact of the auxiliary skip connection on the gradient vanishing problem.\nThe experiments are extensive, using several datasets and comparing several existing approaches. The experimental results provide some evidence that the idea works.\n\ncons:\nWhile the basic idea is well motivated, one could question about the specific architecture to add the auxiliary skip connection. The latter is intended to play the role of stabilizing the training. It is unclear why this specific architecture is appropriate for it. The paper does not provide a strong explanation for it.\nWhile DART- outperforms many of the baselines, it is only marginally better than some recent methods (namely P-DART). For example, in Table 2, DART- is equivalent to P-DART on CIFAR-10.  The experimental evident is moderately strong to show that DART- is better than the existing methods.\n\nRecommendation to the authors: It would be useful to justify the choice of the architecture of DART-, namely to explain why the auxiliary skip connection added can truly play the role of stabilizer. Would it be possible to use a different architecture as well?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an interesting method to improve the robustness of DARTS and hence to alleviate the mode collapse. The idea consists of adding an auxiliary skip connection branch that complements the output of the cell function together with a depth analysis about the effect of the auxiliary branch. The proposed approach is validated on a few benchmarks showing the effectiveness of the proposed approach. All reviewers agreed that the idea is simple, efficient and interesting. author response satisfactorily addressed most of the points raised by the reviewers, and most of them increased their original score accepting the paper. Therefore, I recommend acceptance.
Decision: Accept (Poster)