Decoupled Differentiable Neural Architecture Search:
Memory-Efficient Differentiable NAS via Disentangled Search Space

Libin Hou 1 Linyuan Wang 1 Bin Yan 1

Abstract

average accuracy of models with increasing search space
sizes. However, the comprehensive nature of the supernet,
encompassing all connections and operations in the search
space, poses a challenge of excessive GPU memory
consumption. There is an urgent need for larger-scalemodels but the shortening of the search process does
not necessarily mean an equivalent reduction in GPU
memory usage [14]. Therefore, designing a new method toeffectively reduce GPU memory usage is crucial to fully
exploit the utility of Differentiable NAS.
In this paper, we revisit the gradient process of searchand point out that the intermediate gradient of cachingactivation values in previous methods becomes a significant
part of GPU memory footprint. Therefore, Decoupled
Differentiable Neural Architecture Search (D2NAS) is
proposed to replace the above design of entanglement.
The Disentangled Search Space containing Branch Cellis designed, see Fig. 1 for details. Only Branch cells are
updated during the search to avoid a large accumulationof gradients of Normal cells. Moreover, we use GumbelSoftmax to make the selection of candidate operations morestable than previous methods [10], [11].

Differentiable Neural Architecture Search (NAS)is a popular paradigm, but scaling this approach
to models with larger parameters is severely hampered by the fact that the entire supernet resides
in GPU memory. In this paper, we rethink thegradient propagation processs of Differentiable
NAS and propose Decoupled Differentiable Neural Architecture Search (D2NAS). In our method,
the branch structure is designed to decouple theweight update of the trainable parameters fromthe backbone network, and the candidate operation selection is redesigned with Gumbel-Softmax
to make the overall differentiable process morestable. Experiments show that D2NAS achieves
both performance and stability, with 67% memory cost compared to the best other differentiable
methods.

1. Introduction
Neural Architecture Search (NAS) [1] has made rapid
progress in neural network design for vision tasks suchas classification [2], [3], object detection [4] and segmentation [5]. Early approaches based on evaluating a large
number of candidate models required an unaffordable
cost [6], [7]. Inspired by the weight sharing mechanism,various low-cost approaches have emerged [8]. DARTS [9]introduces architecture weights to indicate the importanceof each operation and employs first-order and second-orderapproximations to update operation parameters andarchitecture weights through stochastic gradient descent.
DARTS dominates with its fast speed and has a lot of
follow-up research [3], [10]–[13].
The scalability is essential since larger search spaces holdmore high quality models, as depicted by the increasing

NC1

NC1
BC1

RC1

×2

NC2
BC2

NC2

NC3

...

...

...

BCm-2
RCm-1

×2

NCm
(a) Vanilla Search Space

NCm-1

NCm
(b) Disentangled Search Space

Figure 1. Illustration of the differences of neural architecture between D2NAS and previous approaches. (a) Vanilla Search Space
from [9] is stacked with Normal Cell (NC) and Reduction Cell
(RC). The whole architecture bears the burden of all gradient propagation while searching. (b) Our proposed Disentangled Search
Space is composed of Normal Cell (NC) and Branch Cell (BC).
Only the architecture parameters of BC are updated during thesearch.

1Henan Key Laboratory of Imaging and Intelligence Processing, Zhengzhou, China. Correspondence to: Libin Hou <lbhou98@163.com>.
Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1D2NAS

2. Related work

parameters α,
minα Lval (w∗ (α), α)s.t. w∗ (α) = arg min Ltrain (w, α∗ ) .

In order to reduce GPU memory cost, some previous studies
modified the forward process of the supernet.
DARTS-like Methods. P-DARTS [15] proposes the search
space approximation strategy, which reduces the number ofcandidate operations accordingly whenever the number oflayers increases. Reducing the number of candidate operations frees up the GPU memory footprint, but also affects
the accuracy of the search. PC-DARTS [11] employs partial
connections instead of a full supernet.
Single-path Methods. Certain research works have suggested consolidating all parameterized operations into a
unified convolution, with a comparable hyperkernel strategy
applied in single-path NAS studies. ProxylessNAS [10]samples two operations per edge during the search, whichenables agent-free search on large datasets. Single-path supernet like SPOS [16] and FairNAS [17] only sample one
path per iteration, however, both require additional searchphases to select the final model. Single-path differentiable
methods like GDAS [18] sample the subgraph of the Directed acyclic graphs(DAG) in each iteration, which is by
far the most efficient method.
However, the iteration of the differentiable process itself
consumes a lot of GPU memory, and this problem is not
mitigated, which has been neglected in previous studies. In
our method D2NAS, the differentiable search procedure is
optimized from the decoupled space.

(2)
(3)

w

The gradients back propagated from the loss L to Wi and bi
are
∂L 0
∂L
=
σ oi ,∂Wi∂oi+1 i
∂L
∂L 0
=
σ ,
∂bi
∂oi+1 i(4)where σi0 is the abbreviation of ∂oi+1 /∂zi+1 . Furthermore,
the term ∂L/∂oi+1 can be recursively expressed as∂L ∂oi+2 ∂zi+2
∂L 0
∂L
=
=
σ Wi+1 . (5)∂oi+1∂oi+2 ∂zi+2 ∂oi+1∂oi+2 i+1
To correctly calculate the gradients, except for parametersfrom the model (in this case, Wi and bi ), all corresponding{σi0 } in the chain rule have to be cached during search,which dominates the GPU memory usage.

4. Method
The gradient update is consistent with the Eq. 2 in Disentangled Search Space, but the gradient merging of Normal
Cell and Branch Cell is prone to produce gradient collapse
in the computational accumulation. In order to improve thestability of the method, instead of Eq. 1, we propose to use
Gumbel softmax for the selection of candidate operations,which is mainly described below:
To disentangle the search for topology and operations oneach edge, we use an indicator Bi,j ∈ {0, 1} to denote
whether edge ei,j is selected, and Aoi,j ∈ {0, 1} for whether
operation o on edge ei,j is selected. Sampling architecture
z with M connections can be decomposed into two parts:
sample M edges first, and their operations second.

3. Preliminary
The Differentiable NAS framework is defined as follows.
Given a specific task, such as a classification task, withtraining set Dtrain and validation set Dval , a unit can be
represented as a Directed Acyclic Graph (DAG) with N
nodes, each node representing a potential feature. Eachdirected edge (i, j) is associated with an operation, o(i, j) ∈O, where O is a set of candidate operations. DARTS [9]relaxes the learnable architecture parameters α continuously
to blend the outputs of operations as


(i,j)
X exp αk o(x).
ō(i,j) (x) =
(1)
P
(i,j)
exp αk0
k∈O

Sampling for edges. Topology inconsistency exists insingle-path based methods [19], [20], as all 14 edges ina cell are selected in the search stage but the final architecture only has 8 edges. To address this issue, we propose tosample the same number of edges in search.

k0 ∈O

Each intermediate node should connect with exact two predecessors, satisfying the constraint of DARTS. Formally,
we use Bi,j to indicate whether the edge ei,j between nodexi and xj is sampled, and we enforce,
X
Bi,j = 2, ∀j.
(6)

Candidate architecture is an N layer convolution neural network y = fN (fN −1 (...f1 (x))), where layer i has a weightmatrix Wi and a bias term bi . We denote oi+1 , zi+1 as
the output and pre-activation of layer i, respectively. Then,oi+1 = σ(zi+1 ) = σ(Wi oi + bi ), where σ is the activation
function.
At the search stage, NAS can be modeled as a bi-level optimization problem that minimizes the loss L by alternately
updating the operation weights w (the parameters of thecandidate operations on each edge) and the architecturei<jSampling for operations. We use Aoi,j to denote whether
the operator o is sampled on the edge ei,j , and we adopt
2D2NAS

Gumbel-Softmax technique to sample operations, where
o
o
=
gi,j
is sampled from Gumbel(0, 1) distribution and gi,joo− log(− log(i,j )), i,j obeys uniform distribution, and
o
α̃i,j
=

P

exp(αo
i,j )
o0 )
exp(α
0
o ∈O
i,j

space S0, inferred models are built by stacking 20 cells with36 initial channels and trained under the same conditions
as outlined in [9], [15]. For S1-S4, we adhere strictly tothe settings established in R-DARTS [23] to ensure a faircomparison.

is the normalized architecturalweights:
Ai,j = Oone hotoo
arg max(log α̃i,j+ gi,j)o∈O5.2. Results,(7)

Performance in S0 on CIFAR. We follow DARTS [9] andsearch on the CIFAR-10. Table 1 shows that achieves stateof-the-art performance with only 0.3 GPU-days. D2NAS
has an average of 2.58±0.07% error rate, which is slightly
higher than up-to-date SOTAs such as SDARTS-ADV [25].
However, D2NAS is more than 4× faster. Compared with RDARTS [23], D2NAS robustly outperforms it with 5× fewer
search costs. Our best model achieves 97.52% accuracy
with 3.6M parameters.

To make the objective function differentiable to architecturalweights α, we relax the discrete distribution to a continuous
one by Gumbel-Softmax:ooexp (log α̃i,j)/τ+ gi,joÃi,j = P|O|

,
o0
o0
o0 =1 exp (log α̃i,j + gi,j )/τ
Ai,j = Oone hot arg max Ãoi,j ,(8)o∈OModelswhere the temperature τ gradually decreases in the search
process.

5. Experiment5.1. Settings
Search space. The search space of DARTS is denoted as
S0, which includes a stack of duplicate normal cells andreduction cells. Each cell is represented by a DAG with fourintermediate nodes. The candidate operations between every
two nodes are {maxpool, avgpool, skip connect, sep conv
3×3 and 5×5, dil conv 3×3 and 5×5}.We search and evaluate on CIFAR-10 [21] and ImageNet [22] under S0 search
space, respectively.
We also perform experiments on the four reduced search
spaces, S1-S4, introduced by R-DARTS [23] to assess thestability of our approach. S1 represents a pre-optimized
search space where each edge in the supernet is associatedwith a predefined set of candidate operations. In the remaining three search spaces, the candidate operations on eachedge remain consistent. Consistent with R-DARTS [23], thesearch and evaluation are conducted within these four search
spaces using CIFAR-10, CIFAR-100 [21], and SVHN [24]datasets.

Params FLOPs
(M)
(M)

Error
(%)

Cost
SP
(GPU Days)

DARTS-V1 [9]3.3
528 3.38±0.23
P-DARTS [15]‡3.3±0.2 540±34 2.81±0.14
PC-DARTS [26]‡ 3.6±0.5 592±90 2.89±0.22
PR-DARTS [27]‡3.42.68±0.10
R-DARTS [23]2.95±0.21
SDARTS-ADV [25] 3.3
2.61±0.02
3.7
2.54±0.07
ZARTS [28]Few-shot NAS [29]3.82.31±0.08
3.4
2.93
GDAS [19]SNAS [20]2.82.85±0.02

0.4
0.3
0.1
0.2
1.6
1.3
1.0
1.35
0.2
1.5

D2NAS (best)
D2NAS (avg.)

0.3
0.3

3.5
593
2.29
3.7±0.4 595±25 2.31±0.08

×
×
×
×
×
×
×
×
X
X

Table 1. Averaged performance among 4 independent runs ofsearch on CIFAR-10. ‡ : reproduced result using their released
code since they didn’t report the average performance. † : FLOPs
are calculated by their released architecture. SP: single-path based
methodWe also search on CIFAR-100 and show the results in Table 2. D2NAS surpasses all the methods and achieves stateof-the-art with only 0.3 GPU-days search cost.
Performance in S0 on ImageNet. First, we transfer thearchitecture searched on CIFAR-10 to ImageNet following
the common practice [9], [15], [17]. Same as [17], we trainmodels for 250 epochs with a batch size of 1024 by SGD
optimizer with a momentum of 0.9 and an initial lr of 0.5
base learning rate. We also utilize an auxiliary classifier
strategy. The results are shown in Table 3, where D2NAS
achieves 75.3% top-1 accuracy.
Second, as D2NAS features low memory cost and greatrobustness, we directly search on ImageNet as well. We
randomly sample 10% images to train operation parameters
and another 10% to train architectural weights. A supernet
is constructed by stacking 8 cells with 16 initial channels.

Search Settings Similar to DARTS, the supernet comprises
8 cells with 16 initial channels. The search is conducted
over 50 epochs with a sampling number set to K = 7. The
operation parameters are optimized using the SGD optimizer
with a momentum of 0.9 and an initial learning rate of 0.05.
As for the architectural weights, the Adam optimizer is
employed with an initial learning rate of 3 × 10−4 .
Evaluation Settings The evaluation settings mirror thoseof DARTS [9], where the inferred model is trained for 600
epochs using SGD with a batch size of 96. In the search3

D2NAS

ModelsParams
(M)

Error
(%)

Cost
(GPU Days)

ModelsResNet [30]
AmoebaNet [31]PNAS [32]
ENAS [33]
DARTS [9]GDAS [19]P-DARTS [15]R-DARTS [23]
ZARTS [28]

1.7
3.1
3.2
4.6
3.4
3.6
4.1

22.10
18.93
19.53
19.43
20.58±0.44?
18.38
17.49‡
18.01±0.26
16.29±0.53

3150
150
0.45
0.4
0.2
0.3
1.6
1.0

D2NAS (best)
D2NAS (avg.)

3.2
3.4±0.5

17.39
17.52±0.13

0.3
0.3

AmoebaNet-A [31]
NASNet-A [34]
PNAS [35]DARTS [9]P-DARTS [15]FairDARTS-B [17]SNAS [20]PC-DARTS [26]GDAS [19]D2NAS (ours)

555
564
588
574
577
541
522
586
581
576

DARTS, P-DARTS
FairNAS-C [17]SPOS [16]ProxylessNAS [10]FBNet-C [36]PC-DARTS [26]GDAS [19]D2NAS (ours)

OOM‡ when batch-size ≥ 32
321
4.4
74.7
12
323
3.5
74.4
12
465
7.1
75.1
8.3
375
5.5
74.9
9
597
5.3
75.4
3.8
405
3.6
72.5
0.8
592
4.8
75.8
0.4

Table 2. Comparison of searched models on CIFAR-100.  : Reported by [19], ? : Reported by [23], ‡ :Rerun their code.

We search for 30 epochs with K = 3. The batch size is set
as 256. Our search cost is reduced to 0.4 GPU days on a single Tesla V100. We fully train the discovered model for 250
epochs with the same evaluation settings as above. Results
are illustrated in Tabel 3, showing that D2NAS achieves
75.8% top-1 accuracy. To make fair comparisons, we reproduce GDAS [19] under the same settings (90 epochs).
However, the network is dominated by skip connections andonly achieves 72.5% top-1 accuracy.

FLOPs Params Top-1
Cost
(M)
(M)
(%) (GPU days)
5.1
5.3
5.1
4.7
5.1
4.8
4.3
5.3
5.3
5.2

74.5
74.0
74.2
73.3
75.3
75.1
72.7
74.9
74.0
75.3

3150
2000
225
0.4
0.3
0.4
1.5
0.1
0.2
0.3

Table 3. Performance on ImageNet. The first block indicates themodels transferred from CIFAR-10; The second block indicatesthat the models are directly searched on ImageNet.
‡
OOM: Out of Memory while running codes on GPU.
MethodDARTS

Memory (G)

9.4

GDAS
3.1

PC-DARTS
M=4

M=2

3.7

5.7

D2NAS
2.1

Table 4. GPU Memory cost comparison. We measured the costbased on a batch size of 64, where the supernet has 16 initialchannels, and 8 layers.

Robustness Evaluation. We follow the recommended bestpractices for NAS by [23], [25] to report the mean andvariance across several times of parallel searching withvarious random seeds, through which the robustness of amethod can be measured.
We follow R-DARTS [23] to evaluate the performance andacross 3 datasets in S1-S4 search spaces, see Table 5(Appendix). Our methods robustly outperform other methods
with a clear margin across all these benchmarks.

6. Conclusion
This study introduces a Memory-Efficient method with Disentangled Search Space in Differentiable Neural Architecture Search (NAS). We show that the decoupled search
space in the differentiable search framework can searchgood performance models while saving memory resources.
Experimental results on CIFAR10, CIFAR-100, SVHN and
ImageNet datasets reveal that D2NAS approach achieves
competitive performance compared to other differentiable
methods, with only about 67% of their GPU memory usage
and more stability.

Memory Analysis. Table 4 compares GPU memory costin S0 search space on CIFAR-10. D2NAS has the lowest
memory cost thanks to our disentanglement of the search for
topology. Unlike GDAS that preserves multiple edges foreach node, we strictly sample 2 edges for each node leading
to 26% memory reduction compared to GDAS.
PC-DARTS [26] uses partial channels during the search
stage to reduce GPU memory cost, in which the partial
ratio is controlled by a hyperparameter M . But M requires
careful calibration for different tasks. In contrast, D2NAS
doesn’t require calibrating such an extra hyperparameterand is more memory-efficient.

References
[1]

[2]

4

B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learningtransferable architectures for scalable image recognition,”in CVPR, 2018.
M. Tan and Q. Le, “Efficientnet: Rethinking model scaling
for convolutional neural networks,” in Proceedings of theD2NAS

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

International Conference on Machine Learning (ICML),
PMLR, 2019, pp. 6105–6114.
X. Chu, X. Wang, B. Zhang, S. Lu, X. Wei, and J. Yan,
“Darts-: Robustly stepping out of performance collapse
without indicators,” en, in Proceedings of the International
Conference on Learning Representations (ICLR), 2021.
G. Ghiasi, T.-Y. Lin, and Q. V. Le, “Nas-fpn: Learning
scalable feature pyramid architecture for object detection,”in The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2019.
C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L.
Yuille, and L. Fei-Fei, “Auto-deeplab: Hierarchical neuralarchitecture search for semantic image segmentation,” inProceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2019.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in Proceedings of the International
Conference on Learning Representations (ICLR), 2017.
[Online]. Available: https : / / arxiv . org / pdf /1611.01578.pdf.
M. Tan, “Mnasnet: Platform-aware neural architecture
search for mobile,” en, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
G. Bender, P.-J. Kindermans, B. Zoph, V. Vasudevan, and Q.
Le, “Understanding and simplifying one-shot architecture
search,” in Proceedings of the International Conference on
Machine Learning (ICML), 2018.
H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiablearchitecture search,” en, in Proceedings of the International
Conference on Learning Representations (ICLR), 2019.
H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” Proceedingsof the International Conference on Learning Representations (ICLR), 2019.
Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H.
Xiong, “Pc-darts: Partial channel connections for memoryefficient architecture search,” in International Conference
on Learning Representations, 2019.
H. Liang, Darts+: Improved differentiable architecture search with early stopping, en, arXiv preprint
arXiv:1909.06035, 2019.
P. Ye, B. Li, Y. Li, T. Chen, J. Fan, and W. Ouyang, “Betadarts: Beta-decay regularization for differentiable architecture search,” en, in 2022 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), 2022,
10864–10873,
R. Wang, M. Cheng, X. Chen, X. Tang, and C.-J. Hsieh,“Rethinking architecture selection in differentiable nas,” inProceedings of the International Conference on Learning
Representations (ICLR), 2021.
X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap
between search and evaluation,” in Proceedings of theIEEE/CVF International Conference on Computer Vision,
2019, pp. 1294–1303.
Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J.
Sun, “Single path one-shot neural architecture search withuniform sampling,” in European Conference on Computer
Vision, Springer, 2020, pp. 544–560.

[18]

[19]

[20]

[21]
[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

5

X. Chu, T. Zhou, B. Zhang, and J. Li, “Fair darts: Eliminating unfair advantages in differentiable architecture search,”
in European conference on computer vision, Springer,
2020, pp. 465–480.
X. Dong and Y. Yang, “Searching for a robust neural architecture in four gpu hours,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition(CVPR), 2019, pp. 1761–1770.
——, “Searching for a robust neural architecture in four
gpu hours,” in Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), 2019.
S. Xie, H. Zheng, C. Liu, and L. Lin, “Snas: Stochastic neural architecture search,” in Proceedings of the International
Conference on Learning Representations (ICLR), 2018.
Tech. Rep.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei, “Imagenet: A large-scale hierarchical image database,”in 2009 IEEE conference on computer vision and patternrecognition, Ieee, 2009, pp. 248–255.
A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, andF. Hutter, “Understanding and robustifying differentiablearchitecture search,” in Proceedings of the International
Conference on Learning Representations (ICLR), 2020.
Y. Netzer and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in Proceedings of
the Annual Conference on Neural Information Processing
Systems (NeurIPS), 2011.
X. Chen and C.-J. Hsieh, “Stabilizing differentiable architecture search via perturbation-based regularization,” inProceedings of the International Conference on Machine
Learning (ICML), PMLR, 2020, pp. 1554–1565.
Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H.
Xiong, “Pc-darts: Partial channel connections for memoryefficient architecture search,” in Proceedings of the International Conference on Learning Representations (ICLR),
2019.
X. C. Zhou Pan, “Theory-inspired path-regularized differential network architecture search,” in Proceedings of
the Annual Conference on Neural Information Processing
Systems (NeurIPS), 2020.
X. Wang, W. Guo, J. Su, X. Yang, and J. Yan, “Zarts: On
zero-order optimization for neural architecture search,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22, ¡confloc¿, ¡city¿New Orleans¡/city¿, ¡state¿LA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿: Curran Associates Inc.,
2024, ISBN: 9781713871088.
Y. Zhao, L. Wang, Y. Tian, R. Fonseca, and T. Guo, “Fewshot neural architecture search,” in Proceedings of the International Conference on Machine Learning (ICML), PMLR,
2021, pp. 12 707–12 718.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE
conference on computer vision and pattern recognition,
2016, pp. 770–778.
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” inProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.

D2NAS
[32]

[33]

[34]

[35]

[36]

C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J.
Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in Proceedings of theEuropean Conference on Computer Vision (ECCV), 2018,
pp. 19–34.
H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” inProceedings of the International Conference on Machine
Learning (ICML), 2018.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in Proceedings of the International
Conference on Learning Representations (ICLR), 2017.
C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li,
L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in ECCV, 2018.
B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian,
P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-awareefficient convnet design via differentiable neural architecture search,” in The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2019.

6

D2NAS

A. Appendix
A.1. The comparative experiment for the stability of D2NAS
Benchmark DARTS†

ES†

ADA†

Error (%)

Error (%)

Error (%)

Error (%)

3.05±0.07
3.41±0.14
3.71±1.14
4.17±0.21

3.03±0.08
3.59±0.31
2.99±0.34
3.89±0.67

2.89±0.09 3.8±0.4 2.66±0.05 1.3±0.4
3.89±0.17 6.0±0.7 3.12±0.13 2.0±0.0
3.04±0.10 6.5±0.5 2.65±0.01 2.0±0.0
3.34±0.10 0.0±0.0 3.20±0.14 0.0±0.0

GDAS

D2NAS
Num

Error(%)

Num

C10

S1
S2
S3
S4

4.66±0.71
4.42±0.40
4.12±0.85
6.95±0.18

C100

S1
S2
S3
S4

29.93±0.41 28.90±0.81 24.94±0.81 24.49±0.08 4.0±0.0 22.70±0.67 2.3±0.4
28.75±0.92 24.68±1.43 26.88±1.11 24.57±0.47 6.3±0.4 22.87±0.74 3.5±0.5
29.01±0.24 26.99±1.79 24.55±0.63 22.86±0.17 3.0±0.7 22.41±0.32 2.5±0.524.77±1.51 23.90±2.01 23.66±0.90 24.14±0.89 2.3±1.1 20.90±0.43 0.0±0.0

S1
S2
SVHN
S3
S4

9.88±5.50
3.69±0.12
4.00±1.01
2.90±0.02

2.80±0.09
2.68±0.18
2.78±0.29
2.55±0.15

2.59±0.07
2.79±0.22
2.58±0.07
2.52±0.06

2.48±0.04 2.8±0.4 2.32±0.06 0.8±0.4
3.05±0.02 7.8±0.4 2.43±0.06 1.0±0.0
3.62±0.36 7.5±0.5 2.56±0.05 1.5±0.5
2.51±0.06 1.5±1.5 2.32±0.01 0.0±0.0

Table 5. Comparison in 4 search spaces and 3 datasets. For each algorithm, we independently search for 3 times under the settings in
R-DARTS [23] and report the averaged performance. ‘EA’ and ‘ADA’ are two methods proposed by R-DARTS. ‘Num’: To reveal thecollapse issue, we also report the average number of parameterless operations in the discovered normal cell for GDAS and D2NAS †:Results are obtained from R-DARTS.

7