DrNAS: Dirichlet Neural Architecture Search
Authors: Xiangning Chen
Ruochen Wang
Minhao Cheng
Xiaocheng Tang
Cho-Jui Hsieh
Venue: ICLR 2021 Poster (2021)

Abstract: This paper proposes a novel differentiable architecture search method by formulating it into a distribution learning problem. We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based optimizer in an end-to-end manner. This formulation improves the generalization ability and induces stochasticity that naturally encourages exploration in the search space. Furthermore, to alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme that enables searching directly on large-scale tasks, eliminating the gap between search and evaluation phases. Extensive experiments demonstrate the effectiveness of our method. Specifically, we obtain a test error of 2.46\% for CIFAR-10, 23.7\% for ImageNet under the mobile setting. On NAS-Bench-201, we also achieve state-of-the-art results on all three datasets and provide insights for the effective design of neural architecture search algorithms.

Published as a conference paper at ICLR 2021
DRNAS:
DIRICHLET NEURAL ARCHITECTURE SEARCH
Xiangning Chen' Ruochen Wang'~ Minhao Cheng
Xiaocheng Tang² Cho-Jui Hsieh
Department of Computer Science, UCLA, DiDi AI Labs {xi (xiangning, chohsiehJecs.co ucla. edu (ruocwang, mhchengJ@ucla.edt
xi aacchengtanggddidgglllo.l.ccc
ABSTRACT
This paper proposes a novel differentiable architecture search method by formu- lating it into a distribution learning problem. We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-basee optimizer in an end-to-end manner. This formulation improves the generalization ability and induces stochasticity that natu- rally encourages exploration in the search space. Furthermore, to alleviate the large memory consumption of differentiable NAS, propose simple yet effective progressive learning scheme that enables searching directly on large-scale tasks, eliminating the gap between search and evaluation phases. Extensive experiments demonstrate the effectiveness of our method. Specifically, we obtain test error of 2.46% for CIFAR-I0, 23.7% for ImageNet under the mobile setting. On NAS- Bench-201, we also achieve state-of-the-arr results on all three datasets and provide insights for the effective design of neural architecture search algorithms.
INTRODUCTION
Recently, Ncural Architecture Search (NAS) has attracted lots of attentions for its potential to democratize deep learning. For a practical end-to-end deep learning platform, NAS plays a crucial role in discovering task-specific architecture depending on users' configurations (e.g., dataset, evaluation metric, etc.). Pioneers in this field develop prototypes based on reinforcement learning (Zoph Le)
2017), evolutionary algorithms (Real et al. 2019) and Bayesian optimization (Liu et al.
2018). These
works usually incur large computation overheads, which make them impractical to use. More recent algorithms significantly reduce the search cost including one- one-shot methods (Pham et al.) [018 Bender al. 2018), continuous relaxation of the space (Liu al. 2019) and network morphisms (Cai et al. 2018). In particular, [iu et (2019) proposes differentiable NAS framework DARTS, converting the categorical operation selection problem into learning a continuous architecture mixing weight. They formulate bi-level optimization objective, allowing the architecture search to be efficiently performed by a gradient-based optimizer.
While current differentiable NAS methods achieve encouraging results, they still have shortcomings that hinder their real-world applications. Firstly, several works have cast doubt on the stability and generalization of these differentiable NAS methods (Chen & Hsich 2020; Zela al. (020a). They discover that directly optimizing the architecture mixing weight is prone to overfitting the validation set and often leads distorted structures, e.g., searched architectures dominated by parameter-free operations. Secondly, there exist disparities between the search and evaluation phases, where proxy tasks are usually employed during search with smaller datasets or shallower and narrower networks, due to the large memory consumption of differentiable NAS.
In this paper, we propose an effective approach that addresses the aforementioned shortcomings named Dirichlet Neural Architecture Search (DrNAS). Inspired by the fact that directly optimizing the architecture mixing weight is equivalent to performing point estimation (MLE/MAP) from probabilistic perspective, we formulate the differentiable NAS as a distribution learning problem
"Equal Contribution.

---

Published as a conference paper at ICLR 2021
instead, which naturally induces stochasticity and encourages exploration. Making use of the probability simplex property of the Dirichlet samples, DrNAS models the architecture mixing weight random variables sampled from parameterized Dirichlet distribution. Optimizing the Dirichlet objective can thus be done efficiently in an end-to-end fashion, by employing the pathwise derivative estimators to compute the gradient of the distribution (Martin Jankowiak 2018). A straightforward optimization, however, turns out to be problematic due to the uncontrolled variance of the Dirichlet, i.e., too much variance leads to training instability and too little variance suffers from insufficient exploration. In light of that, we apply an additional distance regularizer directly on the Dirichlet concentration parameter strike a balance between the exploration and the exploitation. We further derive a theoretical bound showing that the constrained distributional objective promotes stability and generalization of architecture search by implicitly controlling the Hessian of the validation error.
Furthermore, to enable direct search on large-scale tasks, we propose progressive learning scheme, eliminating the gap between the search and evaluation phases. Based on partial channel connection (Xu et al. 2020), we maintain a task-specific super-network of the same depth and number of channels as the evaluation phase throughout searching. To prevent loss of information and instability induced by partial connection,
divide the search phase into multiple stages and progressively increase the channel fraction via network transformation (Chen et al. (016). Meanwhile, we prune the operation space according the learnt distribution to maintain the memory efficiency.
We conduct extensive experiments on different datasets and search spaces to demonstrate DrNAS's effectiveness. Based on the DARTS search space (Liu al. 2019), we achieve average error rate of 2.46% on CIFAR-10, which ranks top amongst NAS methods. Furthermore, DrNAS achieves superior performance on large-scale tasks such ImageNet. It obtains a top- error of 23.7%/7.1 1%, surpassing the previous state-of-the-art (24.0% 47.3%%%%% 7.3% under the mobile setting, On NAS-Benche 201 (Dong & Yang, 2020). we also set new state- of art results on all three datasets with low variance. Our code is available at htttp:::/ /ttt:///ghhhb com/ xi http//iaannnngggghhaga www.:piiiinocceee/aaaaaaa.
2 THE PROPOSED APPROACH
In this section, first briefly review differentiable NAS setups and generalize the formulation to motivate distribution learning. We then layout our proposed DrNAS and describe its optimization in section 2.2 In section 2.3 we provide a generalization result by showing that method implicitly regularizes the Hessian norm over the architecture parameter. The progressive architecture learning method that enables direct search is then described in section 2.4
2.1 PRELIMINARIES: DIFFERENTIABLE ARCHITECTURE SEARCH
Cell-Based Search Space The cell-based search space is constructed by replications of normal and reduction cells (Zoph et al. 2018 Liu et al. 2019). A normal cell keeps the spatial resolution while a reduction cell halves it but doubles the number of channels. Every cell is represented by DAG with N nodes and E edges, where every node represents a latent representation x' and every edge (i,j) is associated with an operations o(f-)) (e.g., max pooling or convolution) selected from a predefined candidate space O. The output of a node is a summation of all input flows, i.e., x² = -Ei, oft-)) (x'), and a concatenation of intermediate node outputs, i.e., co2col(x2 .... NN- composes the cell output, where the first two input nodes x and x² are fixed to be the outputs previous two cells.
Bradient-Based Search via Continuous Relaxation To enable gradient-based optimization, ct al. (2019) apply continuous relaxation to the discrete space. Concretely, the information passed from node to node j is computed by weighted sum of all operations alone the edge, forming a mixed-operation ((t.5) (xx Eneo (f o(x). The operation mixing weight ((1,2) is defined over the probability simplex and its magnitude represents the strength of each operation. Therefore, the architecture search can be cast as selecting the operation associated with the highest mixing weight for each edge. To prevent abuse of terminology, we refer weight, and concentration parameter / in DrNAS as the architecture parameter throughout the paper.
the architecture/operation mixing
Bilevel- Dalevel-Optimizaation with Simplex Constraints With continuous relaxation, the network weight and operation mixing weight 0 can be jointly optimized by solving constraint bi-level optimization
2

---

Published as a conference paper at ICLR 2021
problem:
10 Σ - = v (1)
min (eat(w",0 .) s.t. w* = arg min (irain f),
where the simplex constraint Eo 0ft0)
be either solved explicitly via Lagrangian function (Li et al., 2020), or eliminated by substitution method (e.g., 5oflmaz(()) E
R R E (Liu et al. 2019). In the next section we describe how this generalized formulation motivates our method.
2.2 DIFFERENTIABLE ARCHITECTURE SEARCH AS DISTRIBUTION LEARNING
Learning a Distribution over Operation Mixing Weight Previous differentiable architecture search methods view the operation mixing weight 0 as learnable parameters that can be directly optimized (Liu et al., 2019 Xu et al.) 2020 Li et al. 2020). This has been shown to cause 0 to overfit the validation set and thus induce large generalization error (Zela et al. 2020a b Chen & Hsieh 2020). We recognize that this treatment equivalent to performing point estimation (e.g.. MLE/MAP) of ( in probabilistic view, which is inherently prone overfitting (Bishop 2016 Gelman ct al. 2004). Furthermore, directly optimizing 0 lacks sufficient exploration in the search space, and thus cause the search algorithm commit suboptimal paths in the DAG that converges faster the beginning but plateaus quickly (Shu et al.) 2020).
Based these insights, formulate the differentiable architecture search as a distribution learning problem. The operation mixing weight 0 is treated as random variables sampled from a learnable distribution. Formally, let ())]3) denote the distribution of 0 parameterized by B. The bi-level objective is then given by:
min g (w" 0)] A)(3,3) s.t. w" = arg min Lirain (w, 0). B
where d(.:.) .) is a distance function. Since lies on the probability simplex, we select Dirichlet distri- bution to model its behavior, i.e., q( () Dir(3), where 3 represents the Dirichlet concentration parameter. Dirichlet distribution is a widely used distribution over the probability simplex (Joo et al.)
2019 David M. Blei 2003 Lee et al. 2020; Kessler et al. 2019). and it enjoys nice properties that enables gradient-based training (Martin Jankowiak] 2018).
The concentration parameter 3 controls the sampling behavior of Dirichlet distribution and is crucial in balancing exploration and exploitation during the search phase. Let 3, denote the concentration parameter assign operation When B, < 1 for most sparse samples with high variance, reducing the training stability; when 3. >>> for most o = ~ [O], the samples will be dense with low variance, leading insufficient exploration. Therefore, we add penalty term in the objective (2) to regularize the distance between /3 and the anchor 3 = which corresponds to the symmetric Dirichlet. In section 2.3 we also derive theoretical bound showing that our formulation additionally promotes stability and generalization the architecture search by implicitly regularizing the Hessian of validation loss w.r.t. architecture parameters.
|0], Dirichlet tends produce
Learning Dirichlet Parameters via Pathwise Derivative Estimator Optimizing objective with gradient-based methods requires propppropagttion through stochastic nodes of Dirichlet samples. The commonly used reparameterization trick does not apply to Dirichlet distribution, therefore we approximate the gradient of Dirichlet samples via pathwise derivative estimators (Martin Jankowiak/ 2018)
Reta 03, d3, f (0,13,, Brot - 3,) (11-01))
d8;
(0,18,, Brot - 18))
6ig
0i) 0y
1,.,
(3)
where FBeeaa and feta denote the CDF and PDF of beta distribution respectively, 6,y is the indi- cator function, and Stot is the sum of concentrations. FBeta is the iregularised incomplete beta function, for which its gradient can be computed by simple numerical approximation. We refer to (Martin Jankowiak] 2018) for the complete derivations.
3

---

Published as a conference paper at ICLR 2021
Joint Optimization of Model Weight and Architecture Parameter With pathwise derivative estimator, the model weight w and concentration 3 can be jointly optimized with gradient descent. Concretely, draw sample 0 ~ Dir(3) for every forward pass. and the gradients can be obtained easily through backpropagation. Following DARTS (Liu et al. 2019), we approximate w" in the lower level objective of equation with step of gradient descent, and run alternative updates between w" and B.
Selecting the Best Architecture At the end of the search phase, a learnt distribution of operation mixing weight is obtained. We then select the best operation for each edge by the most likely operation in expectation:
(b.)) arg max E OEO 200 [o
(4)
In the Dirichlet case, the expectation term is simply the Dirichlet mean Ee' B
Note that under
distribution learning framework, we are able to sample a wide range of architectures from the learnt distribution. This property alone has many potentials. For example, in practical settings where both accuracy and latency are concerned, the learnt distribution can be used to find architectures under resource restrictions post search phase. We leave these extensions to future work.
2.3
THE IMPLICIT REGULARIZATION ON HESSIAN
It has been observed that the generalization of differentiable NAS is highly related the domi- nant eigenvalue of the Hessian of validation loss w.r.t. architecture parameter. Several recent works report that the large dominant eigenvalue of V Coel 0) in DARTS results in poor generalization the Lagrangian function
performance (Zela al. 2020a Chen & Hsieh 2020). Our objective ?
of the following constraint objective:
min (0(0) [Crai("", ,)] s.t. w" arg min Lirain (w, (w,0), d(B,3)
(5)
3
Here we derive an approximated lower bound based on 5 which demonstrates that our method implicitly controls this Hessian matrix.
Let d(3,,) = 1|B --11 < 6 and 3 = 1 in the bi-level formulation 5 Let p denote the mean under the Laplacian approximation of Dirichlet. If V Evat (w", p)) is Positive Semi- Semi-definite, the upper-level objective can be approximated bounded by:
Proposition 1
Egg (Coat (w, 2 Enni(us (w="",,,0 ,P) + 2 $10
2
+
1
)r (V (ma (6)
10
0
1
with:
Softmar((4)), Beal(w" (1) : Ho = log Bo
Σ log Bor, 0 o 1....,
101
This proposition is driven by the Laplacian approximation to the Dirichlet distribution (acKay 1998 Akash Srivastava 2017). The lower bound (6) indicates that minimizing the expected validation loss controls the trace norm of the Hessian matrix. Empirically, we observe that DrNAS always maintains the dominant eigenvalue of Hessian at low level (Appendix A.4). The detailed proof can be found in Appendix A.
2.4 PROGRESSIVE ARCHITECTURE LEARNING
The GPU memory consumption of differentiable NAS methods grows linearly with the size of operation candidate space. Therefore, they usually use easier proxy task such as training with a smaller dataset, or searching with fewer layers and number of channels (Cai et al. 2019). For instance, the architecture search is performed 2019). But during evaluation, the network has 20 cells and 36 initial channels. Such gap makes hard to derive optimal architecture for the target task (Cai et al. 2019
cells and 16 initial channels in DARTS (Liu et al.)
4

---

Published as a conference paper at ICLR 2021
PC-DARTS (Xu et al. 2020) proposes a partial channel connection to reduce the memory overheads of differentiable NAS, where they only send à random subset of channels to the mixed-operann cixeedoperation while directly bypassing the rest channels in a shortcut. However, their method causes loss of information and makes the selection of operation unstable since the sampled subsets may vary widely across iterations. This drawback is amplified when combining with the proposed method since we learn the architecture distribution from Dirichlet samples, which already injects certain stochasticity. As shown in Table [ when directly applying partial channel connection with distribution learning, the test accuracy of the searched architecture decreases over 3% and 18% on CIFAR-10 10 and CIFAR-100 respectively if we send only 1/8 channels the mixed mixed-operation.
To alleviate such information loss and instability problem while being memory -efniorefffiiient, we propose
a progressive learning scheme which gradually increases the fraction of channels that are forwarded to the mixed-operation and meanwhile prunes the operation space based on the learnt distribution. We split the search process into consecutive stages and construct a task-speeific super-network with the same depth and number of channels as the evaluation phase at the initial stage. Then after each stage, we increase the partial channel fraction, which means that the super-network in the next stage will be wider, i.e., have more convolution channels, and in turn preserve more information. This is achieved by enlarging every convolution weight with a random mapping function similar to Net2Net (Chen al. 2016). The mapping function y {1, 11,2....97 o (1,2,....,,) nn] with q > n is defined
{ random sample from {
j S n j > n
g(i)
To widen layer 1, we replace its convolution weight w(() ROOxxInxHX with new weight U() w
U o, g(0))
(8)
where Out, In, H, W denote the number of output and input channels, filter height and width respectively. Intuitively, as defined in g. Unlike Net2Net, information flow on each edge has the same scale no matter the partial fraction is. After widening the super-network, we reduce the operation space by pruning out less important operations according to the Dirichlet concentration parameter /3 learnt from the previous stage, maintaining a consistent memory consumption. As illustrated in Table [T] scheme effectively discovers high accuracy architectures and retains low GPU memory overhead,
copy w directly into U and fulfill the rest part by choosing randomly
do not divide U by replication factor here because the
the proposed progressive architecture learning
3 DISCUSSIONS AND RELATIONSHIP TO PRIOR WORK
Early methods in NAS usually include full training and evaluation procedure every iteration as the inner loop to guide the consecutive search (Zoph & Le 2017| Zoph jet al., 2018: Real et al., 2019). Consequently, their com- putational overheads are beyond acceptance for practical usage, especially on large-scale tasks.
Table I: Test accuracy of the derived architectures when searching NAS- Bench-201 with different partial channel fraction. where 1/K channels are sent to the mixed-operation.
CIFAR-10
Differentiable NAS Recently, many works pro- posed to improve the efficiency of NAS (Pham et al. 2018
Test Accuracy (%)
GPU Memory (MB)
K
Cai al.
2018 Liu et al., 2019; Bender et al. 2018 Yao et al. 2020b a Mei et al. 2020). Amongst them,
94.36 = 0.00
2437
93.49 = 0.28
1583
92.85 + 0.35
1159
DARTS (Liu al. 2019) proposes a differentiable NAS
91.06 = 0.00
949
framework, which introduces a continuous architecture
Ours
94.36 + 0.00
949
parameter that relaxes the discrete search space. Despite being efficient, DARTS only optimizes a single point on the simplex every search epoch, which has no guarantee to generalize well after the discretization during evaluation. So its stability and generalization have been widely chal- lenged (Li & Talwalkar 2019; Zela et al.) 2020a Chen & Hsich, 2020; Wang et al. (021). Following DARTS,
CIFAR-100 100
Test Accuracy (%)
GPU Memory
K
(MB)
73.51 = 0.00
2439
2
68.48 + 0.11
1583
66.68 - 3.22
1161
55.11 13.78
949
Ours
73.51 + 0.00
949
SNAS (Xie et al. (019) and GDAS (Dong & Yang 2019)
5

---

Published as a conference paper at ICLR 2021
leverage the gumbel-softmax trick to learn the exact chitecture parameter. However, their reparameterization motivated from reinforcement learning perspective, which is an approximation with softmax rather than an architecture distribution. Besides, their methods require tuning of temperature schedule (Yan et al. 2017 Caglar Gulcehre (017). GDAS linearly decreases the temperature from 10 to 1 while SNAS anneals it from to 0.03. In comparison, the proposed method can automatically learn the architecture distribution without the requirement of handcrafted scheduling. BayesNAS (Zhou al.) 2019) applies Bayesian Learning in NAS. Specifically, they cast NAS model compression problem and use Bayes Neural Network as the super-networds which is difficult to optimize and requires oversimplified approximation, While our method considers the stochasticity in architecture mixing weight, as it is directly related to the generalization of differentiable NAS algorithms (Zela al. 2020a Chen & Hsieh 2020).
Memory overhead When dealing with the large memory consumption of differentiable NAS, previous works mainly restrain the number of paths sampled during the search phase. For instance, ProxylessNAS (Cai et al. 2019) employs binary gates and samples two paths every search epoch. PARSEC (Casale et al. 2019) samples discrete architectures according to a categorical distribution to save memory. Similarly, GDAS (Dong & Yang. 2019) and DSNAS (Hu et al. a discrete constraint after the gumbel-softmax reparametrization. However, such discretization manifests premature convergence and cause search instability (Zela et al. 2020b) Zhang al., 2020). Our experiments in section 4.3 also empirically demonstrate this phenomenon. As an alternative, PC-DARTS (Xu et al. 2020) proposes partial channel connection, where only a portion of channels is sent to the mixed-operation. However, partial connection can cause loss of information as shown in section 2.4 PC-DARTS searches on shallower network with less channels, suffering the search and evaluation gap. Our solution, by progressively pruning the operation space and meanwhile widening the network, searches a task-specific manner and achieves superior accuracy on challenging datasets like ImageNet (+2.8% over BayesNAS, +2.3% over GDAS, +2.3% over PARSEC, +2.0% over DSNAS, +1.+ +1.2% over ProxylesSNAS, and +0.5% over PC-DARTS).
2020) both enforce
EXPERIMENTS
In this section, we evaluate our proposed DrNAS on two search spaces: the CNN search space in DARTS (Liu et al. 2019) and NAS-Bench-201 (Dong & Yang 2020). For DARTS space, we conduct experiments on both CTFAR-10 and ImageNet in section 4.1 and 4.2 respectively. For
201, we test all 3 supported datasets (CIFAR-10, CIFAR-TO0, ImageNet- 6--20 (Chrabaszez et al.) 2017) in section 4.3 Furthermore, we empirically study the dynamics of exploration and exploitation throughout the search process in section 4.4
4.1 RESULTS ON CIFAR-10
Architecture Space For both search and evaluation phases, we stack 20 cells to compose the network and set the initial channel number as 36. We place the reduction cells the 1/3 and 2/3 of the network and each cell consists N = nodes.
Search Settings We equally divide the 50K training images into two parts, one is used for optimiz- ing the network weights by momentum SGD and the other for learning the Dirichlet architecture distribution by an Adam optimizer. Since Dirichlet concentration B must be positive, we apply the shifted exponential linear mapping 3 = ELU(n) + 1 and optimize over 1) instead. We use l2 norm to constrain the distance between 7) and the anchor n = The 7) is initialized by standard Gaussian with scale 0,001, and À in (2) is set to 0,001. The ablation study in Appendix AA.3 reveals the effectiveness of our anchor regularizer, and DrNAS insensitive to wide range of X. These settings are consistent for all experiments. For progressive architecture learning, the whole search process consists of 2 stages, each with 25 iterations. In the first stage, we set the partial channel parameter K as fit the neter-network into a single GTX 1080Ti GPU with IIG memory, i.e., only 1/6 features are sampled on each edge. For the second stage, we prune half candidates and meanwhile widen the network twice, i.e., the operation space size reduces from 8 to and K becomes 3.
Retrain Settings The evaluation phase uses the entire 50K training set to train the network from scratch for 600 epochs. The network weight is optimized by an SGD optimizer with cosine

---

Published as a conference paper at ICLR 2021
Table 2: Comparison with state-of-the-ar image classifiers on CIFAR-10.
Test Error
Params
Search Cost
Search
Architecture
(%)
(M)
(GPU days)
Method
DenseNet-Be BC Huang et al. 2017]"
3.46
25.6
manual
NASNet-A (Zoph el all 2018
2.65
3.3
2000
RL
AmochaNet-A (Real ci al. 2019
3.34 + 0.06
3.2
3150
evolution
AmoebaNet-B [Real ct 2019
2.55 + 0.05
2.8
3150
evolution
PNAS (Liu et al. 2018
3.41 0.09
3.2
225
SMBO
ENAS (Pham ct al. 2018
2.89
4.6
RL
DARTS (Ist) [Lii ot al.
2019
3.00
3.3
gradient
DARTS (2nd) [Lau
2019
2.76 0.09
3.3
1.0
gradient
SNAS (moderate) [Xte ct al. 2019
2.85
2.8
1.5
gradicnt
GDAS (Dong Yang 2019
2.93
gradient
BayesNAS (Zhou
2019
+.04
3.4
0.2
gradient
PAoxylessNAS (Cai
2019
2.08
5.7
gradient
PARSEC (Casale
+.01
gradient
P-DARTS (Chen
2019
2.50
3.4
0.3
gradient
PC-DARTS Xu
2020
0.07
gradicnt
SDARTS-AUD ADV (Chen Hstch h 2020
2.61 0.02
3.3
1.3
gradient
GAEA PC- PC-DARIS (L. et al. 2020
2.50 + 0.06
3.7
0.1
gradient
DrNAS (without progressive learning)
2.54 +: 0.03
0.41
gradient
DrNAS
2.46 0.03
4.1
0.6
gradient
Obtained without cutout augmentation.
Obtained different space with PyramidNet (Han al. 2017 the backbone.
Recorded single GTX 1080Ti GPU.
annealing learning rate initialized as 0.025, a momentum of 0.9, and weight decay of 3 x 10-4, To allow fair comparison with previous work, also employ cutout regularization with length 16, drop-path (Zoph et al. 2018) with probability 0.3 and an auxiliary tower of weight 0.4.
Results summarrzemrriizs the performance of DrNAS compared with other popular NAS methods, and we also visualize the searched cells in Appendix A.2 DrNAS achieves an average test error of 2.46%, ranking top amongst recent NAS results. ProxylessNAS is the only method that achieves lower test error than us, but it searches on a different space with a much longer search time and has larger model size. We also perform experiments to assign proper credit to the two parts of our proposed algorithm, i.e., Dirichlet architecture distribution and progressive learning scheme. When searching on proxy task with 8 stacked cells and initial channels as the convention (Liu et al.. 2019 Xu et al. 2020), we achieve test error of 2.54% that surpasses most baselines. Our progressive learning algorithm eliminates the gap between the proxy and target tasks, which further reduces the test eπrоr. Consequently, both of the two parts contribute lot our performance gains.
4.2 RESULTS ON IMAGENET
Architecture Space The network architecture for ImageNet is slightly different from that for CIFAR-10 in that we stack 14 cells and set the initial channel number as 48. We also first downscale the spatial resolution from 224 x 224 to 28 x 28 with three convolution layers of stride 2 following previous works (Xu al. 2020 Chen et al. 2019). The other settings are the same with section 4.1
Search Settings Following PC-DARTS (Xu et al. 2020). we randomly sample 10% and 2.5% images from the 1.3M training set to alternatively learn network weight and Dirichlet architecture distribution by momentum SGD and an Adam optimizer respectively. We use 8 RTX 2080 Ti GPUs for both search and evaluation, and the setup of progressive pruning is the same with that on CIFAR-10, i.c., 2 stages with operation space size shrinking from 8 to 4, and the partial channel K reduces from 6 to 3.
Retrain Settings For architecture evaluation, train the network for 250 epochs by an SGD optimizer with momentum of 0.9, a weight decay of x 10 and a linearly decayed learning rate initialized 0.5. We also use label smoothing and auxiliary tower of weight 0.4 during training. The learning rate warm-up is employed for the first 5 epochs following previous works (Chen et al. 2019 Xu al. 2020).
7

---

Published as a conference paper at ICLR 2021
Table 3: Comparison with sate-of-theearr image classifiers on ImageNet in the mobile setting.
Test Error %)
Params
Search Cost
Search Method
Architecture
top-1
top-5
(M)
(GPU days)
Inception-v] I (Szegedy al. 2015
30.1
10.1
6.6
manual
MobileNet (Howard
2017
29.4
10.5
4.2
manual
ShuffleNet 2x (VI) Zhang
2018
26.4
10.2
5
manual
ShuffleNet 2x (v2) Ma et
2018
25.1
5
manual
NASNet-A (Zoph
2018
26.0
8.4
2000
RI.
AmoehaNet:C [Real
2019
24.3
7.6
3150
evolution
PNAS (Liu al.
25.8
225
SMBO
MnasNer-92 Tan et 2019
25.2
8.0
RL
DARTS (2nd) (Liu ct al.
2019
26.7
8.7
4.7
gradient
SNAS (mild) (Xie al. 2019
27.3
9.2
gradient
GDAS (Dong Yang 2019
26.0)
8.5
0.3
gradient
BayesNAS (Zhou et al. 2019
26.5
8.9
3.9
0.2
gradient
DSNAS (Hu et al, 2020
25.7
8.1
gradient
ProxylessNAS (GPU) (Cai al. 2019
24.9
7.5
7.1
8.3
gradient
PARSEC (Casale al.
2019
26.0
8.4
5.6
gradient
P-DARTS (CIFAK-IU)
Chen ct al.
2019
24.4
7.4
4.9
0.3
gradient
P-DARTS (CIFAR- 100)
Chen
2019
24.7
7.5
5.1
0.3
gradient
PC-DARTS (CIFAR-10) 10)
Xu et
2020
25.1
7.8
5.3
0.1
gradient
PC-DARTS (ImageNet)
Xa
2020
24.2
7.3
3.8
gradient
GAEA + PC-DARTS (Li et al. 2020
24.0
7.3
3.8
gradient
DrNAS (without progressive learning)
24.2
7.3
5.2
3,9
gradient
DrNASt
23.7
7.1
5.7
4.6
grudient
The architecture is searched ImageNet, otherwise is searched on CIFAR-10 CIFAR-100.
Results As shown in Table 3. achieve a top-1/5 test error of 23.7% outperforming all the-arttrrt art performance in the ImageNet mobile setting. The
compared baselines and achieving state-of satte-of-the-sse searched cells are visualized in Appendix A.2 Similar to section 4.T we also report the result achieved with 8 cells and 16 initial channels, which is common setup for the proxy task on ImageNet (Xu al. 2020). The obtained 24.2% top-1 accuracy is already highly competitive, which demonstrates the effectiveness of the architecture distribution learning large-scale tasks. Then our progressive learning scheme further increases the top- accuracy for 0.5%/0.2%. Therefore, learning in task tasc-specific manner is essential to discover better architectures.
4.3 RESULTS ON
Recently, some researchers doubt that the expert knowledge applied to the evaluation protocol plays an important role the impressive results achieved by leading NAS methods (Yang al.) 2020 Li
& Talwalkar 2019). So to further verify the effectiveness of DrNAS, we perform experiments NAS-Bench- 201 (Dong& & Yang 2020). where architecture performance can be directly obtained by querying in the database. ImageNet- 120 (Chrabaszez et al.) 2017)) and has a unified cell-based search space containing 15.625 architectures. We refer to their paper (Dong & Yang 2020) for details of the space. Our experiments are performed in a task-specific manner, i.c., the search and evaluation are based on the same dataset. The hyperparameters for all compared methods are as their default and for DrNAS, we use the same search settings with section 4.1 We run every method 4 independent times with different random seeds and report the mean and standard deviation Table
provides support for 3 dataset (CIFAR-14) 10, CIFAR-100,
As shown, we achieve the best accuracy on all 3 datasets. On CIFAR-100, we even achieve the global optimal. Specifically, DrNAS outperforms DARTS, GDAS, DSNAS, PC-DARTS, and SNAS by 103.8% 35.9%, 30.4%, 6.4%, and 4.3% on average. We notice that the two methods (GDAS and DSNAS) that enforce a discrete constraint, i.c., only sample a single path every search iteration, perform undesirable especially on CIFAR-100, In comparison, SNAS, employing similar Gumbel- softmax trick but without the discretization, performs much better. Consequently. a discrete constraint during search can reduce the GPU memory consumption but empirically suffers instability. In comparison, we develop the progressive learning scheme on top of the architecture distribution learning, enjoying both memory efficiency and strong search performance.
8

---

Published as a conference paper at ICLR 2021
Table 4: Comparison with otate-of-the-art NAS methods on JAS-Bench-201..
Method
CIFAR-10
CIFAR-100
ImageNet-16-120
validation
ResNer He nl. 2016]
90.83
93.97
70.42
70.86
44,53
43.63
Random (Baseline)
90.93 = 0.36
93.70
0.36
70.00
1.37
70.65
1.38
12.92 = 2.00
12.
2.15
RSPS (L Talwalkar 2019
81.16 + 1.69
87.66 = 1.69
45.78 = 6.33
46.60 = 6.57
31.00 = 5.65
30.78 =: 6.12
Reinforce Zoph el 2015
91.08
0.37
93.85
0.37
70.05
1.67
70.17
1.61
43.04 1. 2.18
43.16 2.28
ENAS PPham
30.71 + 0.00
54.30
+ 0.00
10.23 + 0.12
10.02 0.27
16.43 + 0.00
16.32 + 0.00
DARTSTIST [Li a 2
39.77 + 0.00
61.30 = 0.00
38.57 = 0.00
38. 38.97 = 0.00
13.87 = 0.00
18:41 + 0.00
DARTS (2nd)
39.77 +
54.30
0.00
38.57 + 0.00
38.97
18.87 +
18.41 + 0.00
GDAS Dong & Yang 2019
90.01 + 0.46
93.
0.23
21.03 8.12
21.20 = 8.08
40.66 = 0.00
11.02 0.00
SNAS Xie
90.10 + 1.04
02.77 0.83
69.69 2.39
69.34 1.98
42.84 +
43.16 2.64
DSNAS (Hi 2020
89.06 + 0.29
93 08 = 0.13
30.87
16:40
31.01 16.38
40.61 = 0.09
11.07 = 0.09
PC-DARTS eurctal
93.
DINAS
91.55 + 0.00
94.36 + 0.00
73 19 0.00
73.51 = 0.00
46. 16.37 + 0.00
46. 16.34 0.00
optimal
4.4 EMPIRICAL STUDY ON EXPLORATION V.S. EXPLOITATION
We further conduct empirical study the dynamics of exploration and exploitation in the search phase of DrNAS on NAS- B-nch-nnhhh.. After every search epoch, We sample 100 As from the learned Dirichlet distribution and take the arg max to obtain 100 discrete architectures. We then plot the range of their accuracy along with the architecture selected by Dirichlet mean (solid line Figure D. Note that in algorithm, simply derive the architecture according the Dirichlet mean as described in Section 2.2 As shown in Figure[] the accuracy range of the sampled architectures starts very wide but narrows gradually during the search phase. indicates that DrNAS learns encourage exploration in the search space at the early stages and then gradually reduces it towards the end as the algorithm becomes more and more confident of the current choice. Moreover, the performance of our architectures can consistently match the best performance of the sampled architectures, indicating the effectiveness of DrNAS.
10
Texk Mccusyy (4)
Tex acurrcy
66-
60
60
40
40
40
60
100
20
60
80
100
20
60
80
:00
(a) CIFAR-10
(b) CIFAR-100
(c)
ImageNet1661222
Figure I: Accuracy range (min-max) of the 100 sampled architectures. Note that the solid line is our derived architecture according to the Dirichlet mean as described in Section 2.2
CONCLUSION
In this paper, we propose Dirichlet Neural Architecture Search (DrNAS), We formulate the differen- tiable NAS constraint distribution learning problem, which explicitly models the stochasticity in the architecture mixing weight and balances exploration and exploitation in the search space. The proposed method can be optimized efficiently via gradient-bart based algorithm, and possesses theoretical benefit to improve the generalization ability. Furthermore, we propose a progressive learning scheme to eliminate the search and evaluation gap. DrNAS consistently achieves strong performance across several image classification tasks, which reveals its potential to play a crucial role in future end-to-end deep learning platform.
LCKNOWLEDGEMENT
This work is supported in part by NSF under IIS- 15015279 2I0-2008173, IIS-2048280 and by Army Research Laboratory under agreement number W91 INF-20-2-0158.

---

Published as a conference paper at ICLR 2021
REFERENCES
Charles Sutton Akash Srivastava. Autoencoding variational inference for topic models. In Inter- national Conference Learning Representations, 2017. URL htt.ttttp://aattta wwwp:/iv.orr///g. abg/
1703. 01488
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. In Jennifer Dy and Andreas Krause (eds.). Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 550-559, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR.
mir wttpp/pp//ss///e/eeeee...ee htttp://ppoeeddcnnnss...ppeee//p
html
Christopher Bishop. Pattern Recognition and Machine Learning. Springer New York, 2016.
Yoshua Bengio Caglar Gulcehre, Sarath Chandar. Memory augmented neural networks with wormhole connections, 2017.
Han Cai, Tianyao Chen, Weinan Zhang. Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference Learning Representations, 2019, URL
htttps://ppenzeve//
net forum? hawpe//ooommmmddyymmmm Hiyuumm11-d1yyyyyyAl
Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search, 2019.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. International Conference on Learning Representations, 2016. URL httt:://rx//xi org/abs 1511. 05641
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1554-1565. PMLR, 13-18 Jul 2020. URL http::::/ppcceeeeiiiiee woigg.mlr.ee wttpsee...........//............
chen20f html
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1294-1303, 2019.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets, 2017.
Jon D. McAuliffe David M. Blei, Alp Kucukelbir. Variational inference: A review for statisticians, 2016.
Michael I. Jordan David M. Blei, Andrew Y. Ng. Latent dirichlet allocation. The Journal of Machine Learning Research, Mar 2003. ISSN 1532-4435. doi: 10.1162//m1::200334459999 URL
http://das dx. doi 1t....gg/...111///V..... 2017.243
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1761-1770, 2019.
Xuanyi Dong and Yi Yang. Nas-bench-200 Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020. URL https
//openreview.ro http://fooommmm1dooxeeeee
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2nd ed. edition, 2004.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Jul 2017. doi: 10.1 1099/vr. 2017.668. URL hRL.ttp:: htttp://dxx.. doi 1ttp:///ww..oii.o.100....1011 wwt.////...../......ooo
668
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. pp. 770-778, 2016. doi: 10.1109/CVRR.216.9900
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. Dsnas: Direct neural architecture search without parameter retraining, 2020.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jul 2017.
doi: 10.11099cyp:22017.24. URL http:/ /0:::/dax /xx doi. oww.oo.oi......... 1109/
WMPR.2017.0
Weonyoung Joo, Wonsung Lee, Sungrac Park, and Il-Chul Moon. Dirichlet variational autoencoder, 2019. URL/jttts::/oppnneeeee wtt.:ttee//oommmmaarggroooooo
Samuel Kessler, Vu Nguyen, Stefan Zohren, and Stephen Roberts. Hierarchical indian buffet neural networks for bayesian continual learning, 2019.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning," in international conference on learning representations. In ICLR, 2020.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search, 2019.
Liam Li, Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search, 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Lecture Notes doi: 10. 10.10079988-3300--002444
in Computer Science, pp. 19-35. 2018. ISSN 1611-3349.
URL http: doi org/ 10, 1007/ 978 - 3 030 01246-
Hanxiao I. Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL URL/sttps://pppeeeeriel
net
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In The European Conference Computer Vision (ECCV), September 2018.
David J. C. MacKay. Choice of basis for laplace approximation. Machine Language, October 1998. doi: 10.1023/441000555881155 10. 15..0023/2 : 1at.:.1110155118880
URL /tR.:ttss//iikksprinnneeee petprinner.coo/arteeeeee ctw/articce/////oooottooccccccct
Fritz Obermeyer Martin Jankowiak. Pathwise derivatives beyond the reparameteriization trick. In
International Conference Machine Learning, 2018. URL https://axxiv.zg// 1806. 01851
Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang. Alan Yuille, and Jianchao Yang. Atomnas: Fine-grained end-to-end neural architecture search. In International Con-
ference on Learning Representations, 2020. BtLhhtts::/// h/tttp:://ppenevvee... www./eetutffiddd
BylOSxHEwz
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4095 4095 4104, Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018, PMLR. URL http::/proceedii ngs ml wtw..arwr.....e..m..m.mm prw.s pham 8a .ttml

---

Published as a conference paper at ICLR 2021
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence. doi: wtt.////aai.33311.3300000000
13:4780-4788, Jul 2019. ISSN 2159-5399.
htthttpp
7/dx.do1 wtwp:/doo.or//////././// 100 609 aaa .v3300v.31111. 3301 4780
Yao Shu, Wei Wang, and Shaofeng Cai. Understanding architectures learnt by cell-based neural architecture search. In International Conference Learning Representations, 2020. URL
httpps://ppeneeview wwwp:rrvvweww.eeeooeeeeeeee matt:fooummdddBBix222222
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions, In Computer Vision and Pattern Recognition (CVPR), 2015.
abs /1409. 4842
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection in differentiable NAS. In International Conference on Learning Representa- lions, 2021. URL htRphhtttts:::/eeereee
net/ forum? wwwpmmmmeddaaaaaeeeeeaaaaa
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019. URL /ttpps::
fest/forum?inn himmmmidddrrogggyyoo
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong.
PC-DARTS: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations, 2020. URL httptttttt///peeeeeettt wwwp:ewwwwwww..e.....e.ee..
(d=BJJ1634tPP
Shiyang Yan, Jeremy S. Smith, Wenjin Lu, and Bailing Zhang, Hicrarchical multi-scale attention networks for action recognition, 2017.
Antoine Yang, Pedro M. Esperança, and Fabio M. Carlucci. Nas evaluation is frustratingly hard. In International Conference on Learning Representations, 2020. URL htttps://openreee
Quanming Yao, Xiangning Chen, James T. Kwok, Yong Li, and Cho-Jui Hsieh. Efficient neural interaction function search for collaborative filtering. In Proceedings of The Web Conference 2020, WWW 20, pp. 1660-1670, New York, NY, USA, 2020a. Association for Computing Machinery. ISBN 9781450370233 doi: 10. TB.1444/336642333388277
BLLhttps://dol/ tttp///ti.og....oooo/o 1145/
3366423. 3380277
Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu. Efficient neural architecture search via proximal iterations. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):6664- 6,71, 2020b. doi: 10. 60..1609/a (00.1669/aaai..34444...... v34.04.6143. (tRp:ttps: //ojs aaai. php
6143
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hut- ter. Understanding and robustifying differentiable architecture search. In International Confer- ence on Learning Representations, 2020a. http:/ttt//ppeeeeeeee net forum?iddii
H1 #1cDNy=KDS
Arber Zela, Julien Siems, and Frank Hutter. WAS-BENCH-1SHOTT. Benchmarking and dissecting one-shot neural architecture search. In International Conference on Learning Representations, 2020b. URL https: http///ppppwweeppptetpptpepeeeep http:/et//otttee in id=33x9ggtUPPP
M. Zhang, H. Li, S. Pan, X. Chang, and S. Su. Overcoming multi-model forgetting in one-shot nas with diversity maximization. In 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7806-7815, 2020.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
12

---

Published as a conference paper at ICLR 2021
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In ICML, pp. 7603-7613, 2019. URL http://proceeeinn
mlr. wwwps///3//hhou1333
html
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. 2017. URL
https://arxiv. wrg..rrgggss... 0tt///1611.111111
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018. doi: 10.1109/cyp::218..0000 2010009
URL http://áse doi. dog..og/10
2.009/VVR.200880 00907
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF OF PROPOSITION I
Before the development of Pathwise Derivative Estimator, Laplace Approximate with Softmax basis has been extensively used to approximate the Dirichlet Distribution (MacKay
Preliminaries:
1998
Akash Srivastava 2017). The approximated Dirichlet distribution is:
p(0())13) = r(E. 3o) IL. [(3))
II
00000 g( 1(1hh) h)
Where (hh) is the an arbitrary density to ensure integrability (Akash Srivastava| 2017). The mean f and diagonal covariance matrix 2 of h depends on the Dirichlet concentration parameter B::
h, h follows multivariate normal distribution, and g(.) is
1
1
No log 3.
log B
Σg
(10)
10|
3
10
101
B
can be directly obtained from 10 that the Dirichlet mean
Σ.' p. - Softma((u). Sampling from
the approximated distribution can be down by first sampling from h and then applying Softmax func- tion to obtain 0. We will leverage the fact that this approximation supports explicit reparameterization derive our proof.
Apply the above Laplace Approximation to Dirichlet distribution, the unconstrained upper- level objective in (5) can then be written as:
Proof:
FarDDi(()) [Lrat( (w",0) ,0)
(11)
2 N(0,) [ (w" Softmaz(u++++ e))]
(12)
=EeNN(0.2) [oaa (w" 1 +)]
(13)
= =E [oaa( .1) + e (wo ,11) 1 y V / ViLuaalW" n)
(14)
1
=Loal(w" 11) + tr( (Fem.V(O,)) [ee [co"] Vi ,11)
(15)
2
(w" 11) + tr (EV) 14)
(16)
2
In our full objective, we constrain the Euclidean distance between learnt Dirichlet concentration and fixed prior concentration || - 1|12 S 6. The covariance matrix Σ of approximated softmax Gaussian can be bounded as:
Σ.
1
Σ
1
(17)
B.
0
1
B.
1
1
(1
+
(18)
1
0
0
1+
Then becomes:
FewDDr(0) (w" ,)]
(19)
x (w= 11) + z Ir LIV2
# Eeat (w", ,14)
(20)
2
"m" 1)) + 110
tr (V (V) Coot .4)
(21)
10
The last line holds when V " (wat 11) is positive semi-definite. In Appendix A.4 we provide an empirical justification for this implicit regularization effect of DrNAS.
A.2 SEARCHED ARCHITECTURES
We visualize the searched normal and reduction cells in Figure and 3 which is directly searched on CIFAR-10 and ImageNet respectively.
14

---

Published as a conference paper at ICLR 2021
weguem 203
(a) Normal Cell
(b) Reduction Cell
Figure 2: Normal and Reduction cells discovered by DrNAS CIFAR 10.
sqp_AnII_ 333
462_eamy_ 33
Jkg connect
5.00
1
cony 3k3
upp com 3x3
(a) Normal Cell
(b) Reduction Cell
Figure 3: Normal and Reduction cells discovered by DrNAS ImageNet.
A.3 ABLATION STUDY ON ANCIIOR REGULARIZER PARAMETER À
Table Jishwws the accuracy of the searched architecture using different value of while keeping all other settings the same. Using anchor regularizer? for wide range of value boost the accuracy and DrNAS performs quite stable under different λs.
Table 5: Test accuracy of the searched architecture with different Às on NAS-Bench-201 (CIFAR-10). le -3 is what we used for all of our experiments.
5e 4
le
5e 3
le
2
le
Accuracy 93.78
94.01
94.36
94.36
94.36
93.76
93.76
A.4 EMPIRICAL STUDY ON THE HESSIAN REGULARIZATION EFFECT
We track the anytime Hessian norm on NAS-Bench-201 in Figure The result is obtained by averaging from 4 independent runs. We observe that the largest eigenvalue expands about 10 times when searching by DARTS for 100 epochs. In comparison, DrNAS always maintains the Hessian norm at low level, which agreement with our theoretical analysis in section 2.3 Figure 5 shows the regularization effect under various λs. As we can see, DrNAS can keep hessian norm at a low level for a wide range of As, which is in accordance to the relatively stable performance in Table
Moreover, we compare DrNAS with DARTS and R-DARTS on 4 simplified space proposed in (Zela et al., 2020a) and record the endpoint dominant eigenvalue. The first space contains 2 popular operators per edge based on DARTS search result. For $2, S3, and S4, the operation sets are (3 x 3 separable convolution, skip connection), {3 x separable convolution, skip connection, zero}. and {3 3 separable convolution, noise) respectively. As shown in Table DrNAS consistently outperforms DARTS and R-DARTS. The endpoint eigenvalues for DrNAS are 0.0392, 0.0390, 0.0286, 0.0389 respectively. Figure 5 shows the Hessian norm trajectory under various X.
A.S CONNECTION TO VARIATIONAL INFERENCE
In this section, draw connection between DrNAS and Variational Inference (David M. Blei 2016). We use w, 6, and 3 to denote the model weight, operation mixing weight, and Dirichlet concentration parameters respectively, following the main text. The true posterior distribution can
15

---

Published as a conference paper at ICLR 2021
Dominant Eigenvalue of Hessian
DARTS
DrNAS
10
10a
10"
MM Ummm
MM
10-2
20
10
60
80
100
Search Epoch
Figure 4: Trajectory of the Hessian norm on when searching with CIFAR-10 (best viewed in color).
0.10
0.1
=
Dominant Eigenvalue Validation Loss
=
0.08
=
=.000
==10
0.04
0.00
20
60
80
100
Search Epoch
Figure 5: Trajectory of the Hessian norm under various As on JAS-Bench-201 when searching with CIFAR-10 (best viewed in color).
be written p(0 w. D), where D = (3rn Yn approximation of the true posterior; and assume that (01.3) follows Dirichlet distribution. We follow Joo et al. 2019) assume symmetric Dirichlet distribution for the prior p(0) well,
J >== the dataset. Let g(0 denote the variational
Table 6: CIFAR-10 test error on simplified spaces.
s1
s3
s4
DARTS
3.84
4.85
3.34
7.20
R-DARTS (DP)
3.11
3.48
2.93
3.58
R-DARTS (L2)
2.78
3.31
2.51
3.56
DINAS
2.74
2.47
2.4
2.59
16

---

Published as a conference paper at ICLR 2021
Le., p(0) Dir(1). The goal is minimize the KL divergence between the true posterior and the approximated form, i.c. ming [ ww D)). can be shown that this objective is equivalent to maximizing the evidence lower bound as below (David M. Blei 2016
[(3) = E ()) [log ( DJ0, w)] K [I((((]3)]l(())))))
(22)
The upper level objective of the bilevel optimization under variational inference framework is then given as:
min E 10 - log PDDeatiall0 w*) + K (L(((0)3)llx())
(23)
8
2 if we use the negative log likelihood as the loss function and replace d(,;) with KI. divergence. In practice, we find that using simple 12 distance regularization works well across datasets and search spaces.
Note that eq. 23 resembles eq.
17


Reviews:
[{"review_id": "uj51zpUtNTK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work proposes a modified DARTS optimiser for NAS which assumes a  factorised dirichlet distribution over architecture parameters. It uses pathwise derivatives to learn an MLE estimate of these concentration parameters and adds appropriate regulariser terms to stabilise the training. Furthermore, it employs a protocol for progressively increasing channel fraction to stabilise training within a computation budget. The paper is easy to follow, and relevant experiments have been included. \n\n- With the given probabilistic formulation of this work, it would be useful to include details on how the factorisation of appropriate distributions varies between PARSEC and DrNAS? It might be useful to ground the modelling assumptions within a prior and approximate-posterior framework for better clarity.\n- On the same line it would be useful to get insights on how the obtained models qualitatively differ from ProxylassNAS, PARSEC and SNAS. \n- The interplay of progressive learning with modelling assumption is a bit unclear. The number of parameters and test error in Table 2 are inversely correlated across SNAS, ProxylessNAS, PARSEC and DrNAS which is perhaps not as surprising. I was wondering if authors have any insights on what aspect of the algo (with and without the two stage progressive learning) contributed to network size.\n- Would it be possible to employ the progressive learning policy mentioned in section 4.1 across other DART flavours and understand its impact on model performance?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kSXxdKeqfQL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes Dirichlet Neural Architecture Search which formulates Neural architecture search as a distribution architecture search problem. They derive a bound that shows that their formulation implicitly regularizes the Hessian norm with respect to the architecture parameters which has been shown to allow more robust architecture search. They empirically show that their method keeps the dominant eigenvalue of the Hessian much lower compared to DARTS. To reduce memory usage and increase training speed, they incorporate partial channel connections and propose progressively widening the channel connections during training. They demonstrate strong empirical results on several benchmark datasets and NASbench201.\n\nReasons for score: \nOverall I would recommend accepting this paper. It proposes a new efficient distribution learning-based NAS algorithm which regularizes the Hessian with respect to the architecture parameters. The algorithm is demonstrated by efficiently finding high performing networks on Cifar10, Cifar100, and Imagenet. They provide ablation experiments that demonstrate the effects of different parts of their final algorithm. While the benefits of using the Dirichlet distribution could be somewhat better demonstrated empirically separate from the progressive search, compared to existing NAS distribution learning algorithms which don't discretely sample, it is beneficial that the trade-off between exploration and exploitation can be controlled by a penalty term as compared to the popular Gumbel-softmax based methods which arbitrarily anneal the temperature parameter.\n\nPros:\n1. Proposed approach is shown to regularize the Hessian with respect to the architecture parameters which works have shown to lead to better network generalization.\n\n2. The paper provides many results that show that it can find high performing networks quickly with limited compute quite quickly.\n\n3. Provides ablation experiments are which demonstrate the benefits of different parts of the algorithm.\n\nCons:\n1. The paper would be improved greatly if the robustness of the search and final network architectures was empirically explored. This could be accomplished by analyzing the architecture distribution while searching on NASBench-201. \n\n2. The direct effect of the regularizer parameter on the hessian could be better explored. For example something like figure A4 with different values for $\\lambda$ would be helpful.\n\n2. The paper would benefit from a proper baseline for the proposed progressive search method. Currently it is somewhat unclear if DrNAS with more time and memory would perform better if used to search the architecture space without a proxy or if the progressive search is regularizing the search to perform better. The reverse experiment of using the progressive search with another distribution learning algorithm would also be beneficial.\n\n\nQuestions:\nWas there a particular reason $\\lambda$ = 1e-3 was used in the experiments?\nIn table 4, do you know if there is a particular reason there is no variance in the results for DARTS, ENAS, and your algorithm. It seems a bit strange that DARTS and ENAS all appear to be stuck in the same quite poor performing local minima in this experiment on CIFAR10.\n\n\nUpdates:\nThanks to the authors for addressing my concerns and responding to my questions. The newly added experimental results make the paper stronger and addressed many of my concerns. I recommend this paper to be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NuE499r3zwb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a differentiable NAS algorithm based on the Dirichlet architecture distribution. Different from the previous differentiable and stochastic NAS algorithms that used the Gumbel-softmax trick or the Categorical distribution, the proposed DrNAS does not require any temperature scheduling for balancing exploration and exploitation, and moreover it does not suffer premature convergence and instability during search. In addition, in order to reduce the memory consumption when searching based on the super-net that mixes all possible operations, the proposed DrNAS applies the progressive learning scheme by combining the network widening with the operation pruning.\n\nPros.\nThe proposed algorithm is technically sound, and the motivation and both of the theoretical and empirical analysis are reasonable to support the use of the Dirichlet architecture distribution with the progressive learning. The experimental results also show that the proposed DrNAS consistently outperforms all previous NAS algorithms on CIFAR-10, ImageNet, and especially NAS-Bench-201. \n\nCons.\nMy main concern of the proposed method is how to produce the sparse solution during search to reduce the architectural bias between the search and retraining phases. This naturally brings up the question of how and when it automatically changes the exploration to the exploitation during search, like a temperature annealing for the Gumbel-softmax trick. How can the proposed method remove the retraining?\n\nWhen performing NAS on ImageNet, why did the proposed method use the proxy task with the reduced training set, even though it can retain a low memory overhead like a discrete architecture sampling such as DSNAS? How is the performance variance of the proposed NAS method on ImageNet?\n\nMinor: what if the proposed method searches different cell structures for each layer without the repetition?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DJMJ6ISVYkH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe authors present a new differentiable Neural Architecture Search (NAS) method which places a Dirichlet prior on the edges of a cell for NAS: DrNAS. The authors adapt the original bilevel optimisation to incorporate this new prior over the operation mixing weight. The authors add a regularisation to the Dirichlet parameters and effectively study its effect in an ablation. The authors also introduce a progressive architecture learning to make the optimisation more computationally efficient. DrNAS produces very strong results on three different NAS scenarios and is compared to a wide variety of baselines.\n\n\n##########################################################################\n\nReasons for score: \n\nOverall I vote for a weak reject. I think there are some issues surrounding the technical presentation of the work - see the Cons. Also I have some issues with the notation (see below), fixing these will help a lot with the readability of the paper. I have asked for some clarification and I am happy to raise my score in light of clarification from the authors.  The experimental analysis on the other hand is very strong.\n \n##########################################################################\n\nPros: \nInteresting use of the Dirichlet prior for selecting an edge in a cell in the search space for NAS. A hyperparameter is introduced to regularise the Dirichlet parameters and an suitable ablation is provided to demonstrate the regularisation’s sensitivity.\nInference of the prior’s parameters is performed using the new pathwise estimator [4].\nImplicitly controlling the largest eigenvalue of the Hessian is very important for producing good results in differentiable NAS [1]. The authors experimentally show that DrNAS is very effective in doing so. I have a question with regards to Proposition 1 (Question 3), if it is indeed correct then the authors also provide an interesting bound to demonstrate that the objective in Equation 2 is implicitly attenuating the largest eigenvalue of the Hessian of the validation loss wrt to the Dirichlet parameters.\nDrNAS produces very strong results compared to a variety of benchmarks on three different NAS scenarios.\n\n \n##########################################################################\n\nCons: \nClaims in the text are not justified in the paper, see Question 2 and Question 3.\nGeneral issues surrounding notation**.\nThe formulation of DrNAS is not strictly formulated within a Bayesian framework with prior and likelihood to infer a posterior over operation mixing weights. The L2 regularization can be formulated as a Gaussian prior about the Dirichlet parameters. Instead the Dirichlet prior is added to the bilevel optimisation in Equation 1 in an ad hoc manner.\n \n##########################################################################\n\nQuestions: \n\nSection 2.2, paragraph 1: if \\theta is drawn from a distribution why is it less prone to overfitting? The references are text books.\nIn the paper, for instance in the abstract, you claim that DrNAS encourages better exploration. Of course the model has some stochasticity by sampling from a Dirichlet, but what explicit evidence do you have that having \\theta ~ Dirichlet leads to better exploration of the search space for NAS?\nIn proposition 1: it is assumed that \\nabla^2_\\mu \\tilde{L}_{val} is psd, in general the Hessian of a loss function is pd and psd if L is strictly convex and convex wrt \\mu respectively. Does this assumption hold for \\tilde{L}_{val}?\nYou use the pathwise derivative estimator, which is not very popular (I might be wrong here), and you also work the Laplace approximation for Proposition 1 and in related work [3]. What is the advantage of the pathwise derivative estimator versus the Laplace for inference over Dirichlet parameters?\nYou include stds in Table 4, but not in Table 3? How are the errors in Table 3 calculated?\n\n \n#########################################################################\n\nSome typos and notations issues and other comments**: \n\nSection 2.2, paragraph 3, first sentence no “the” before “exploration” needed.\nSection 2.2, paragraph 3, \\beta_0 is not defined.  \nEquation 2, d(.) not defined.\nEquation 3 needs a space.\nTitle in section 2.3 needs a “the”.\nSection 2.3 \\alpha and \\mu are not defined.\nAlthough these works regard Continual Learning (CL): it is worth mentioning two related works including [5] uses a Dirichlet prior over networks for CL and [6] learns new neurons in a Bayesian NN using an Indian Buffet Process prior.\n\n\n#########################################################################\n\n[1] A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter, “Understanding And Robustifying Differentiable Architecture Search,” in ICLR, 2020.\n[2] S. Lee, J. Ha, D. Zhang, and G. Kim, “A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning,” in ICLR, 2020.\n[3] C. Sutton and A. Srivastava, “Autoencoding Variational Inference for Topic Models,” in ICLR, 2017.\n[4] M. Jankowiak and F. Obermeyer, “Pathwise Derivatives Beyond the Reparameterization Trick,” ICML. 2018.\n[5] S. Lee, J. Ha, D. Zhang, and G. Kim, “A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning,” in International Conference on Learning Representations, 2020.\n[6] S. Kessler, V. Nguyen, S. Zohren, and S. Roberts, “Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning,” arxiv:1912.02290 2019.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes Dirichlet Neural Architecture Search (DrNAS), a new NAS algorithm that formulates NAS as a distribution architecture search problem. The paper shows theoretically that DrNAS implicitly regularizes the Hessian norm with respect to the architecture parameters (which has been previously shown to allow more robust architecture search) and presents very strong empirical results on several benchmarks, including the tabular benchmark NAS-Bench-201. 

The reviews and discussion put this paper very close to the acceptance threshold, so I read it in detail myself to act as a tie-breaker.
I see a lot of positive aspects of this paper:
+ it tackles a very important and timely problem
+ the method implicitly regularizes the Hessian norm with respect to the architecture parameters (which has been previously shown to allow more robust architecture search)
+ empirical results are very strong
+ the paper includes insightful ablation studies for the most important parts of the algorithm
+ the progressive architecture learning approach is a general contribution that reduces the memory complexity and also improves basic DARTS
+ the paper uses a tabular benchmark to yield results that are directly comparable to those of other papers
+ the method's hyperparameters are kept fixed across the different benchmarks, which underlines its robustness.

There are also some negative aspects:
- The method was not originally derived from a Bayesian point of view. Per request of the reviewers, the relation to variational inference has been added to the appendix, but the difference between the L2 regularization and using the explicit KL regularization that falls out of the Bayesian treatment remains. Nevertheless, the new appendix helps to clear up the relationship.

- The paper does not mention availability of the code. This is a must in modern NAS research, as many papers have exposed the poor reproducibility of research in NAS. Fortunately for the authors, I have already seen (independently of this submission) that the code is available on github, but I urge the authors to provide an anonymous repo for review in future submissions, since it was purely by chance that I saw it this time.

- Regarding the point of exploration vs. exploitation, the authors emphasize that in contrast to Gumbel-softmax based methods, such as GDAS and SNAS, with DrNAS there is no need for a cooling schedule. While it is nice to keep the number of hyperparameters small, this also appears to give up control of when the method switches from exploration to exploitation. In practical applications of AutoML, there will be a time budget, and while a cooling schedule can be adapted to fit this budget, it would be suboptimal if DrNAS is still in the exploration phase by the end of the budget, or has already switched to exploitation after, e.g., 5% of the budget. It would be good if the authors could briefly discuss this issue in their final version (if only by acknowledging that this can be a problem).

- Minor negative points
  * The paper sometimes uses jargon, and I believe not even always correctly: e.g., even by googling I did not find such a thing as the "iregularized incomplete beta function", it's also not in the original reference by Jankowiak. I only found the "incomplete beta function" (and the regular one). 
  * The author names for several references are garbled. This is likely due to not replacing a comma between the authors with an "and" in the bibtex file.
  * The paper lacks citations for several of the methods it uses, e.g., Adam, cutout, cosine annealing, label smoothing, auxiliary towers, etc. There is no limit on references, and it is standard to cite these concepts to remain more self-contained.
 * The experimental results of GDAS on NB201-CIFAR-100 do not seem to align with the numbers in the NB201 paper. Did you use the numbers from the paper or rerun this method yourself? Please clarify and check this for the final version. The point of tabular benchmarks is to have comparability and consistency across papers! 
 * Please have the paper proofread for Grammar again, there are several avoidable errors. E.g., in the first sentence, "lots of attentions" --> "lots of attention". Also things like "alone" -> "along", "down"->"done" etc.

Overall, I think this is a very nice paper, introducing an empirically very strong NAS method that is also theoretically shown to implicitly regularize the Hessian norm with respect to the architecture parameters (which has been previously shown to allow more robust architecture search). I am therefore recommending acceptance. I would like to ask the authors to go through all the reviews again and fix any remaining points in the paper for the final version.
Decision: Accept (Poster)