VAENAS: Sampling Matters in Neural Architecture Search
Authors: Shizheng Qin
Yichen Zhu
Pengfei Hou
Xiangyu Zhang
Wenqiang Zhang
Jian Sun
Venue:  (2020)

Abstract: Neural Architecture Search (NAS) aims at automatically finding neural network architectures within an enormous designed search space. The search space usually contains billions of network architectures which causes extremely expensive computing costs in searching for the best-performing architecture. One-shot and gradient-based NAS approaches have recently shown to achieve superior results on various computer vision tasks such as image recognition. With the weight sharing mechanism, these methods lead to efficient model search. Despite their success, however, current sampling methods are either fixed or hand-crafted and thus ineffective. In this paper, we propose a learnable sampling module based on variational auto-encoder (VAE) for neural architecture search (NAS), named as VAENAS, which can be easily embedded into existing weight sharing NAS framework, e.g., one-shot approach and gradient-based approach, and significantly improve the performance of searching results. VAENAS generates a series of competitive results on CIFAR-10 and ImageNet in NasNet-like search space. Moreover, combined with one-shot approach, our method achieves a new state-of-the-art result for ImageNet classification model under 400M FLOPs with 77.4\% in ShuffleNet-like search space. Finally, we conduct a thorough analysis of VAENAS on NAS-bench-101 dataset, which demonstrates the effectiveness of our proposed methods.


Under review as a conference paper at ICLR 2020
VAENAS: SAMPLING MATTERS IN NEURAL ARCHI- TECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) aims automatically finding neural network architectures within an enormous designed search space. The search space usu- ally contains billions of network architectures which causes extremely expensive computing costs in searching for the best- rest-performing architecture. One-shot and gradient-basse based NAS approaches have recently shown to achieve superior results on various computer vision tasks such as image recognition. With the weight shar- ing mechanism, these methods lead to efficient model search. Despite their suc- cess, however, current sampling methods are either fixed or hand-crafted and thus ineffective. In this paper, we propose a learnable sampling module based on vari- ational auto-encoder (VAE) for neural architecture search (NAS), named as VAE- NAS, which can be easily embedded into existing weight sharing NAS framework, e.g., one-shot approach and gradient-basee approach, and significantly improve the performance of searching results. VAENAS generates a series of competitive results on CIFAR-10 and ImageNet in NasNet-like search space. Moreover, com- bined with one-shot approach, our method achieves new state-of-the-an result for ImageNet classification model under 400M FLOPs with 77.4% in ShuffleNet- like search space. Finally, we conduct a thorough analysis of VAENAS on NAS- bench- 101 dataset, which demonstrates the effectiveness of our proposed methods.
INTRODUCTION
Deep neural networks have greatly pushed the frontier of various influential applications by design- ing novel neural architectures (Krizhevsky et al. (2012); GGooffelow et al. (2014); ) He et al. (2016)) Automatic model design of neural network architectures without human intervention, known ral architecture search (NAS), has drawn much attention the community recently. It has resulted in state-of-the-ard performance in the domain of image recognition (Zoph al. (2018); Real et al.
(2019) ), object detection (Ghiasi et al. (2019) Chen et al. (2019)) and semantic segmentation (Liu al. (2119))
Generally, the magnitude of search space for NAS tasks is enormous. For example, NasNet ZZoph al. 2018) presents a search space with 6 x 10 possible cells. Searching such huge designed space cost 2400 GPU days. Weight sharing mechanism has shown to be promising avenue for efficient NAS. Latest algorithms on efficient NAS fall into two categories: one-shot approaches
(Bender et al. (2018)) and gradient-based approaches (Liu et al. (2018b)) In one-shot approaches, prior works focus on adopting a fixed sampling strategy (Guo al. (2019) Bender et al. (2018): Chu et al. (2019)). In gradient- based approaches, search is typically performed without sampling proce- dure (Liu et al. (201 (2018b); Xie et al. (2018)) or hand-crafted sampling (Liu et al. (2018a)). Despite the success of these NAS methods various benchmarks, however, these sampling approaches do not interactively learn the architecture distribution as the search process goes along, which makes the sampling procedure ineffective.
In this paper, we propose a learnable and interactive architecture sampling module based on VAE for NAS, named VAENAS. There are two advantages of VAENAS: 1) it learns the good performrfonnnn architecture distribution which could be used to reduce search space. 2) it can be embedded into existing NAS framework and improve the performance of current NAS methods.
After developing the generic VAENAS approach, we study in detail the application of VAENAS module via two mainstream NAS approaches: one-shot approach (Brock al. (2017); Guo et al.

---

Under review as a conference paper at ICLR 2020
Top 50%
Update Architecture Distribution
Evaluate & Sort
Acc: 90.3% 88.5%
87 97.2%
85.1%
83.2%
81.7%
VAE model
Architecture Search System
Architecture Proposal Sampling
Figure 1: The illustration VAENAS mechanism. VAENAS mechanism contains two modules, one is VAE model, and the other is an architecture search system. These two modules are updated alternately. Specifically, three steps are carried out on VAENAS mechanism. 1) Architectures are sampled from VAE model and fed into the architecture search system. 2) The architecture search system evaluates and sorts sampled architectures. 3) The VAE model is trained with top-50% archi- tectures.
2019 and gradient-based approach (Liu et al. (2018b); Cai et al. (2018)) both of which have obtained state-of-the-are performance on neural architecture search tasks. We show how VAENAS module can be embedded in both approaches and make performance improvement consistently.
We validate VAENAS with various search space on two benchmark datasets (CIFAR-10 and Ima- geNet) for image recognition. On NASNET-like search space, apply VAENAS on both gradient- based methods and one-shot methods. Specifically, combined with one-shot approach, VAENAS achieves 2.26% test error CIFAR-10 and 75.8% accuracy ImageNet, outperforming state-of- the-art NAS methods. On a Shufflenet-like search space, VAENAS combined with one-shot achieves 77.4% top-1 accuracy with 365M FLOPs ImageNet classification, outperforming state-of-the-ar Efficient-BO by 1.1 % with 6.5% less computational complexity. Finally, we perform a thorough analysis of VAENAS on NAS-101 benchmark, show the effectiveness of our proposed architec- ture sampling methods.
2 RELATED WORK
Neural Architecture Search. Recently the design of efficient neural networks has largely shifted from leveraging human knowledge automatic methods, which is known neural architecture search(NAS). Early NAS methods adopt reinforcement learning (RI)(Zoph et al.) (018) or evolu- tionary strategy (Real et al., (2019) to search on thousands of individually evaluated networks, which is computaatooallconsuming Recent works focus on efficient search methods, which falls into two categories: one-shot approach (Guo al. (2019); Chu et al. (2019) and gradient-based approach (Liu et al. (2018)); Cai et al. (2018)), both achieve state-of offte--of-hee-rr the--arett results on a series of benchmark dataset (Chen al. (2019); Ghiasi et al. (2019); Liu al. (2019)) with various search space. We argue that sampling method matters in NAS and learnable sampling algorithm can significantly improve the NAS performance.
Sampling. Some prior works have applied sampling methods to NAS framework. For one-shot approach, penner
[Bender et al. (2018) randomly zero out a subset of the operations during super-network training. Guo al. (2019); Stamoulis et al. (2019) adopt uniform sampling and Chu al. (2019) em- play fair uniform sampling to reduce training bias in one-shot models. For gradient-
approach, Cai et al. (2018) propose multinomial distribution sampling to alleviate large memory consump- tion issue in conventional gradient NAS methods (Liu et al. (2018b)). Our approach presents a new perspective of sampling methods of NAS application.
2

---

Under review as a conference paper at ICLR 2020
Algorithm VAE Training Pipeline
Input:
Architectures and corresponding performance set S = {(a AC # of training itera- tions Nsterss parameters of VAE encoder Φ, parameters of VAE deoeder 6, ranking ratio K.
Output: Updated VAE parameters Φ. 0
Rank by the descending order ACC;;
2: Select top %% architectures VAE training dataset Sk. Sk C S
3: for ,..,, Psters do
4: Optimize @ and 0 with Sk.
5: end for
3 VAENAS
3.1
MOTIVATION
Many prior works in NAS have discussed modeling the architecture distribution I'(a). The typical ((a) is to adopt either fixed architecture distribution (e.g.. uniform distribution Bender et al. (2018) 8);
Guo al. (2019)), ) or hand-crafted architecture distribution (e.g.. Gumbel softmax distribution in
Liu et al.
2018b: Xie et al. (2018)).
Conceptually, it is expected that the architecture distribution [(a) can interactively learn from the weight sharing super-network as the searching process goes along. To this end, we propose learn- able architecture distribution, where the distribution (a) at time is learned from the new incom- ing architecture and prior architecture distribution at t 1 time is (a),, we can formalize the distribution learning process as:
(a(a) + T(L--i(a), Metric,-1(a))
(I)
where Metric denotes the predefined metric, e.g., accuracy or loss, and denotes the stochastic process. In equation [1 the distribution of architectures at time f is learned by both the prior knowl- edge on architecture distribution la( and new architectures data a at the architecture search proceeds, we randomly sample multiple architectures, rank these architec- tures according to the predefined metric. The T incrementally refines the distribution of architec- tures by continually learning the good-performing architectures from the weight sharing network. Alternatively, the weight sharing network updates the architectures weights, where the architectures sampled from T. There many methods that be used to model this distribution, and choose VAE due to its training stability as well as the convenience to be embedded into the existing NAS frameworks.
time, as in Figure 1. As
3.2 THE USAGE OF VAE
VAE approximates the marginal likelihood of architecture distribution log by maximizing variational lower bound:
- - Dirlgo((la)))lo( (ssgg
Specifically, the left term is the Kullback-Leibler divergence between the approximated posterior (recognition model) (2(2)))) and the prior p(z). And the right term is the expectation of reconstruc- tion loss where pe (a|2) denotes variational inference. As result, sample a from the marginal distribution po (..). VAE samples latent variable according to the prior distribution po(z) and feed
into the inference model (=())))). Commonly, the prior distribution po (=( can be set as standard Gaussian distribution ().
Let a = {o: T be a sequence of operation o, representing an architecture string, where n is the total number of nodes in network or cell. We use an encoder E to take as input and maps it into parameters Ha and a corresponding to a normal distribution N( oa). Symmetrically, a decoder D is used map z. reconstructed architecture ā. where ze NM oe). The encoder E corresponds qe (2)a) and the decoder corresponds po (a)22. Specifically, the encoder E and the decoder D are implemented by two L.TM networks.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 2 VAENAS One-Shot
Input: Super-network training iterations ittrss VAE training iterations n tue interval to train
sup
VAE Mlyee # of sampled architectures (rank) N.. of architectures seed N seedss super-network
G, search space A. network weights W, exploitation increase parameter Cinc, ranking ratio K, parameters of VAE module o and 6.
Output: The final architecture o'"
1: Define ((x)
e) U(a) +e + Set
2: for i = 1, ..., Riters do
sup
3: Sample
I(a):
4: Update network weights Wa:
5: if mod Pvaae = then
6: Sample N, number of architectures {a} from P(aa,, ~ T(a);
7:
Evaluate {ar) obtain accuracy {ACC}:
8: Set = {(a,, ACC.)
9: Call algorithm VAB(S, niters K) to update VAE module;
10: end if
4-- + Eénci
12:
end for
Sample Noeeds number of architectures {a} from po(a)z), po(a|z);
14: Set Speeds {(a) Derive the final architecture a from Search(G, A. Seeede);
VAENAS FOR NEURAL ARCHITECTURE SEARCH
In this section, we demonstrate how VAENAS is integrated into existing NAS frameworks. Section 4.1 gives the formal definition of weight sharing mechanism. Section 4.2 gives detailed descrip- tion of VAENAS combined with one-shot framework. Section 4.3 shows VAENAS combined with gradient-based framework.
4.1 WEIGHT-SSHARIN MECHANISM
Weight sharing approaches construct a super-network G(a, W) that contains all the valid architec- tures with shared weights W. Any architecture is sub-graph in the super-network and inherits corresponding weights W.. Thus we can formalize the training of weight sharing mechanism as solving bi-level optimization problem:
W arg min E.))( (aa Lerain (G((a, Wa))]
(3)
W.
arg min Leat W;))
aca
where [(a) is architecture distribution. There are main frameworks which utilize weight- sharing mechanism: one-shot approach and gradient-based approach.
4.2 ONE-SHOT FRAMEWORK
Review of One-Shot Architecture Search. Generally, one- shot approaches (Liu al. (2018b); Cai
et al. (2018); Xie et al. (2018)) consist of three steps: 1) train a weight sharing super-network. 2) rank architectures based on performance on the super-network. 3) derive the final architecture
a search strategy. Previous methods typically adopt hand-crafted sampling methods (Bender et al. 2018
Guo et al. 2019 Chu al. 2019).
Combine One-Shot Approach with VAENAS. Algorithm 2 shows the detailed pipeline of VAE-
NAS combined with one-shot approach. VAENAS-OS utilizes one-shot framework and optimizes sampling distribution:
r(a) = (1- U(a)) c* pe(a[2)
(5)
where (ax) is random uniform distribution, Po (a( [x) is variational distribution, and e is exploitation parameter. On supernet training, architectures sampled from po (ai]z and (aa) with probability
4

---

Under review as a conference paper at ICLR 2020
Algorithm 3 VAENAS- Gradient-Basee
Input: Ranking ratio K, network training iterations Niteras VAE training iterations iiters interval to train VAE Reaes # of sampled architectures (rank) Na. network weights W, architecture weights w, parameters of VAE module and 0.
Output: The final architectures a²
for i= 1, Riters do
2: Sample (all))
3: Update architecture weights Wa-
4: Update super-network weight Wo
5:
i mod Thaae == 0 then
6: Sample N₂ number of architectures {a} from U(a), a U(a); Evaluate {a), obtain accuracy {ACC);
7:
8:
set S = ((as, A Call algorithm VAE(S, o, 0, niters K) to update VAE module:
9:
10: end if'
end for
12:
Derived the final architecture a* based on the learned w.
and 1- As the search phase continues, the e inereases, thus the super-network trains more sam- ples from VAENAS. The good-performing architectures, for example, with 1% highest accuracy, are collected every Nlaee epochs and fed into the VAENAS module to train. the training phase proceeds, the variational distribution po (a(2z) would converge to generating good-performing archi- tectures. On search phase, the initial seeds are sampled from pa (a) =) and then search with genetic algorithm Guo et al. (2019). Due to the weights of wood-performing architectures trained more adequately, intuitively, the super-network would be more predictive. Algorithm (2 shows the detailed pipeline of VAENAS combined with the one-shot approach.
Sampling for One-Shot? The conventional sampling methods for one-shot NAS, such zero out random operations (Bender et al. (2018)), uniform sampling (Guo et al. 2019)) and fair sampling (Chu et al. (2019) are fixed sampling methods.
Why Architecture-Deeendder
One issue of fixed architecture distribution in one-shot approach is its ineffectiveness sample good-performing architectures, both during super-network training and architecture search. When train super-network with architectures sampled from a pre-defined architecture distribution such as uniform distribution, it is likely that the best-performing architectures are not picked; thus the search algorithm fails seek them out due to the low performance of best architectures on the super-network. Moreover, the search strategy usually adopts evolution/RL. algorithms, where a set of architectures are selected as seeds via sampling method. Due the evaluation process is ex- pensive, only small set of architectures are evaluated and ranked. Thereby the initialized set for search strategy can be sub-optimal. VAENAS resolve above issues by adopting an architecture- dependent distribution super-network training and search phase. By learning the distribution of perfoorinf architectures, VAENAS ensures the sampling procedure effective since the sam- ples from the variational distribution Pa(ce) zz) are guaranteed to have good performance the super- network.
4.3 WAADIENTBBSEED FRAMEWORK
Review of Handient-Baseed Architecture Search. Gradient-based approaches (Liu et al. (2018b) (b);
Cai et al. (2018); Xie et al. 2018)) employ an independent discrete architecture parameter to model the networks, thus we can follow the exact bi-level optimization process we describe in equation 3 and 4 The common practice of gradient-baset approach does not contain sampling process. Proxylessnas (Cai et al. (0188) introduce multinomial distribution sampling process prior to the optimization step of architecture weights.
Combine Maadient-Based Approach With VAENAS. Alternatively, we consider replacing the sampling procedure by VAENAS:
T(a)= (x(22)
(6)
5

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison with state-of-the-art architectures on CIFAR-10 under the NASNET-like search space. denotes model is trained with cutout. OS denotes one-shot approach. G denotes gradient-based approach. indicates hard parameters constraint in search phase.
Architecture
Test Error
Params (M)
Search Method
(%)
NASNet-A' Zoop et al. (2018)
2.65
3.3
RL
AmoehaNet-B [Real [ 2019
2.55:1:0.05
2.8
Evolution
Hierarchical Evolution (Liu al. 2017)
1.
15.7
Evolution
PNAS (Liu et al. (2018a))
3.41
3.2
SMBO
ENAS Pham et al (2018)
2.89
4.6
RL
M0--wiightshee sharing (Luo et al. (2118)
3.53
2.5
Gradient
DARTS' LLu et al. (018b)
2.76.10.09
3.4
Gradient
SNAS Xie et al. 2018))
2.76.10.09
3.4
Gradient
Graphelypernet (Zhang
2018a
4.310.1
5.110.66
Gradient
DSO-share
Zhang el al. (2018))
2.8400.07
3.0
Gradient
BayesNAS
Zhou al.
2019)
0.00
2.8110.04
3.4
Gradient
VAENAS-G"
2.40
4.4
Gradient
VAENAS-OS'
2.50
3.4
Evolution
VAENAS-OS
2.26
5.2
Evolution
Specifically, we sample an architecture a; from (al=)) and update super-network weights W., and multinomial distribution parameters z; accordingly. Algorithm the detailed pipeline.
Why Independent Sampling Module Necessary? Conventional gradient-based methods (Liu et al. (2018b) do contain the sampling process: all architecture operations are updated during training at each iteration. These methods bring overwhelming memory and computation consumption issue, such that direct model search and direct dataset search are incapable. Proxylessnas (Cai el. (2018) present multinomial distribution to sample the architectures in the search phase.
However, the probability of multinomial distribution is obtained by applying softmax architecture weights. The principal limitation is that the model weights and architecture weights are mutually interactional. To be concrete, if architecture gets more training, more likely will be sampled again in the next iteration. This winner takes all effect causes premature convergence and reduces the diversity of architectures. Moreover, the gradient tends to flow through non-parameter operations, such identity (the architectures derived by gradient-based method is typically small, i.e., 3.4M on CIFARIO (Liu et al. (201 (2018b)). The VAEÑAS sampling module, which serves as an independent sampling module, helps training process, as a result, alleviate the premature convergence phenomenon, 2) search for large models. Large models naturally obtain better performance than small models.
increase the diversity of architectures that can be exploited in the
EXPERIMENTS
5.1 NASNET-LIKE SEARCH SPACE.
The detailed training strategy, search space and parpem-prrmeters settings can be found in Appendix A.I Our VAENAS is evaluated on both gradient-based and one-shot approaches, and compared with the state-of-the-art NAS methods under the same search space. We use 'OS" to denote one-shot approach and use 'G' to denote gradient- based approach.
Performance on CIEAR-10. We present the results Table [ Combined with gradient-based method, our method achieves 2.4% test error with 4,4M parameters. Combined with one-shot ap- proach, our method gives two results. The first one searched with parameter constraint of 3.5M parameters, and obtain 2.5% test error, and the second one is trained without any constraint and obtain 2.26% test error with 5.2M parameters.
Performance on ImageNet. We present the results on Table 2 Combined with gradient- based approach, our method obtains 75.6% top-1 accuracy with 568M FLOPs. Combined with one-shot

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison with state-of-the-art architectures ImageNet (mobile setting) under NASNET-like search space. i denotes direct search on ImageNet.
Architecture
FLOPs
Params
Top-1 Acc.
Top-5 Acc.
(M)
(M)
(%)
(%)
NASNet-A Zoph et (2018)
564
5.3
74.0
91.6
AmoebaNet-A Real et 2019
555
5.3
74.0
91.5
AmoebaNet-B [Ral et al.
2019
555
5.1
74.5
92.0
AmoebaNet-C (Real et al. 2019)
570
6.4
75.7
92.4
PNAS Cai et al. (2018))
588
5.1
74.2
91.9
Graphl Hypernett (Zhang al. (2018a)
569
6.1
73.0
91.3
BayesNAS (Zhou et al. (2019)
3.9
73.5
91.1
DARTS (Liu et al. (2018b)
574
4.7
73.3
91.3
SNAS (Xie et al. (018))
522
4.3
72.7
90.8
VAENAS-Gf
568
6.0
75.6
92.3
VAENAS-OSt
573
6.1
75.8
92.7
Table 3: Comparison with state-of-the-art architectures on ImageNet (200M-400M FLOPs) under ShuffleNet-like search space. denotes results reported using AutoAugment.
Architecture
FLOPs
Params
Top-1 Acc.
Top-5 Acc.
(M)
(M)
(%)
(%)
ShuffleNet V2 1.5 (Ma et al. (2018)
300
72.6
MobileNet V2 1.0 (Sandler al. 2018
300
3.4
72.0
91.0
MobileNet V3 Large 1.0 (Howard et al. 2019))
219
5,4
75.2
92.2
MnasNet-A2 Tan et al. (2119)
340
4.8
75.6
92.7
FBNet-B Wu al. (2019)
74.1
Proxyless GPU CCai al. (2018))
320
4.0
74.6
92.2
Single-Path NAS [Guo of al. (2019)
365
4.3
75.0
92.2
FairNAS-A (Chu al. (2019)
388
4.6
75.3
92.4
EfficientNet-OO (Tan & Le 2019
390
5.3
76.3
93.2
Baseline VAENAS-OS
360
6.7
77.1
93.3
365
6.7
77.4
93.6
approach, our method obtains 75.8% top-1 accuracy with 573M FLOPs. Both results surpass previ- ous tlate-of-the-at results.
5.2 IMAGENET CLASSIFICATION UNDER MOBILE SETTING
Comparison with State-of-the-arr methods. Table 3 lists a number of state-of-the-art models' FLOPs ImageNet dataset. We select models with the range of FLOPS from 200M 400M. Notably, VAENAS-OS outperforms state-of-the-art models by large margin. particular, accuarcy of VAENAS is 1.1% higher than EfficientNet-BO (Tan & Le 2019). recent state-of-the- art method, while using 6.5% less computational budget. To best our knowledge. VAENAS-OS the first model surpass 77% under 400M FLOPs without using strong regularization and data augmentation techniques such as Auto-uumeent
5.3 VAENAS ON NAS- NAS-BENCH-101 DATASET
MAS-Bench-110 101 dataset (Ying et al. 2019) is carefully constructed architecture dataset for NAS research, including over 5 million already trained architectures CIFAR-10. The ground-truth ac- curacy of architectures can be gotten from NAS-Bon Benc-- dataset directly. We compare VAENAS sampling with uniform sampling (also known random search Li & Talwalkar (2019)).

---

Under review as a conference paper at ICLR 2020
Table 4: Comparison of VAENAS and random search NAS-Bench-101 dataset.
Methods
Max accuracy for K samples (%)
Evolution
10
50
100
300
Random Search
92.93
93.30
93.53
93.86
93.39
Weight-sharing
93.72
93.95
93.98
94.23
94.12
Non-weight-sharing noise
93.56
93.71
93,95
94.13
94.14
Non-weight-sharing + noise 0, %
93.64
93.73
93.97
94.17
93.75
whnnwigg noise 0.5%
92.24
93.35
93.63
93.83
93.33
We design two methods training VAENAS. The weight-sharing method follows Section 4.1,
where VAENAS is trained with super-network on NAS-Bench- search space. The design of super-network is described in Appendix A.5 The non-weight-sharing method trains VAENAS with set of high- architectures. 6000 trained architectures are sampled and the top- 1500 models form the training set.
Influence of noise on accuracy. The training of VAENAS is based on high-accuracy architectures, while ground-truth accuracy architectures is hard to be obtained in most cases. We add various levels of noise the accuracy, then rank architectures and select aigh-accuracy architectures to train corresponding VAENAS module. As displayed in Table and Fig. 2 VAEÑAS is robust to a certain extent of noise. Futhermore, the accuracy of architectures in weight-sharing methods have a noise on super-network, while the accuracy on super-network is inconsistent with ground-truth accuracy. The results of VAENAS trained by weight-sharing method proof that VAENAS works well with NAS frameworks.
Comparisons of sampling methods. We sample a number architectures by VAENAS and random method and evaluate the maximum accuracy of architectures, which is shown in Table VAENAS, trained with super-network or clear ground-truth accuracy. outpeforms random sampling (+0.79% with 10 samples). In addition, we sample 5000 architectures with random sampling and VAENAS sampling, then display the distribution of samples in Fig. [ The distribution of VAENAS covers high-accurase architectures, while the distribution of random sampling is relatively dispersed.
Combination with evolution search. We execute evolution search on NAS-Bench-101 and replace random sampling by VAENAS during the initialization of population. The results are shown in Table VAENAS outperforms uniform sampling by 0.7%. proves that VAENAS is helpful to the search strategy, since the convergence of evolution search is accelerated.
3000
3000
5000
mandom VAE
random VAE
2500
random
meddo
VAE
VAE
1500
1300
1500
1500
V00
500
0.50
0.00
050 094
0.20
(b)
(d)
Figure 2: Compare the distribution of VAE Sampling with Random Sampling. (a) VAENAS trained with super-network. (b) VAENAS trained with ground truth accuracy. (c) VAENAS trained with 0.1 % noise on accuracy. (d) VAENAS trained with 0.5% noise on accuracy.
CONCLUSION
In this paper, we argue sampling matters in NAS and present a learnable sampling method based on VAE. The VAENAS learns the architecture distribution and is interactive with NAS search process. As result, by embedding into the existing efficient NAS framework, we achieve series of state-of- the-art results on various dataset and search space. Overall, we believe that the VAENAS approach provides a practical new perspective sampling methods in neural architecture search.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 549-558, 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv:1708.05344, 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint 1rXiv:/812.0000 00332, 2018.
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas:
Neural architecture search on object detection. arXiv preprint arXiv:1903.10%9 2019.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairnas: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845, 2019.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 7777.0000552 2017.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv: 00420, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp- 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chia Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint arXiv: 1902004400 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.0700 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Chenxi Liu, Liang-Chieh Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 82-92, 2019.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. arXiv preprint arXiv: 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint / 09055, 2018b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in neural information processing systems, pp. 7816-7827, 2018.

---

Under review as a conference paper at ICLR 2020
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: (rXi::8802.03268,, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780 4789, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Dimitrios Stamoulis, Ruizhou Ding, Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Device-aware efficient convnet design. arXiv preprint arXiv:1905.04159, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1905. 1946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv: 1812.09926, 2018.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. arXiv preprint 1rXiv:9902.09655, 2019.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv://10.05744 201 8a.
Xinbang Zhang, Zehao Huang, and Naiyan Wang. Single shot neural architecture search via direct sparse optimization. 2018b.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for 2019.
neural architecture search. arXiv preprint
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
10

---

Under review as a conference paper at ICLR 2020
Table 5: The comparison of architectures searched with (w/) without (wo) VAENAS.
Model
Dataset
w/o VAENAS
w/ VAENAS
CIFAR-10
Params
Top-1 Acc.
Params
Top-1 (gcingain.)
Gradient
/
3.7M
2.84%
4.4M
2.40%((+0.44%)
One-Shot
4.3M
3.33%
5.2M
2.26%(+0.93%%)
Model
Dataset
w/o VAENAS
w/ VAENAS
ImageNet
FLOPs
Top-1 Acc.
FLOPs
Top-1 (gc.ngain.)
Gradient
/
519M
74.1%
568M
75.6%(+1..59)
One-Shot
/
553M
73.3%
573M
A APPENDIX
A.1 IMPLEMENTATION DETAILS OF VAENAS
VAENAS-OS ShuffleNet training details. VAENAS-OP OS is trained for 480 epochs with batch size 1024 on 2080-Ti GPUs. based on the Pytorch (Paskke et al. (017). ). The network parameters are optimized using an SGD optimizer with an initial learning rate of 0.5 (decayed linearly after
each iteration), a momentum of 0.9 and weight decay of 10-5 Additional enhancements
including label smoothing (Szegedy et al. (2 and Dropout with 0.5 on last FC layer. We follow the common practice to use inception augmentation. No other data augmentation method is used.
VAENAS-OS//AENAS-G NASNet training details:
ImageNet. The model is trained for 300 epochs with batch size 512 on 8 2080-Ti GPUs. is based on the Pytorch (Paszke et al. (017))) The network parameters are optimized by using an SGD opti- mizer with an initial learning rate of 0.5 (decayed linearly after each iteration), a momentum of 0.9 and weight decay of x 10-5 Additional enhancements include label smoothing (Szegedy al. (2016)) and auxiliary tower with 0.4. We follow the common practice to use inception augmentation. The initial width is 48 and depth is 14, which follows the same setting Liu al. (2018b).
CIFAR-10. The model is trained by following the common pipeline (Liu et al. (2018b) 8b); Real et al. 2019); Zoph al. (2018); Pham al. (2018); Liu al. (2018ab). it is trained for 600 epochs with batch size 48 on single GPU. Additional enhancements include label smoothing (Szegedy et al. (016 ) auxiliary tower with 0.4, drop path with 0.2 and Cutout VDeVs Taylor (2017).
A.2 IMPORTANCE OF VAENAS
More ablation study will be available in the Appendix A.3 In this section, we use gradient-based NAS approach and one-shot NAS approach as two baseline framework, and compare the result with VAENAS and without VAENAS using the same search and training strategy. The baseline method of gradient- tradient-based approach is re-implemented based on Cai et al. (2018) and one-shot approach is re-implemented based on Guo et al. (2019). The results are shown in Table 5. We can observed that the the framework with VAENAS achieve significantly better performance on both CIFAR-I0 and ImageNet.
A.3 ABLATION STUDY
Influence Depth Gap between Search and Evaluation. One issue encountered in prior works with both one-shot NAS approach and gradient-based approach is the challenge of search on the network with the same depth as the evaluated models. We therefore study the effect VAENAS from the perspective of direct model search. Table 6 shows the results of baseline with different depth that one-shot method fails to find the good architecture when directly search on target model. One explanation is that the search space is too large for the conventional uniform sampling.
11

---

Under review as a conference paper at ICLR 2020
Table 6: NAS framework with different model depth on CIFAR-10.
Model
Params
Top-1 Acc.
Search Depth
One-Shot Baseline
4.3M
3.33%
20
One-Shot Baseline
3.7M
2.89%
VAENAS-OS
5.2M
2.26%
20
One-Shot Baseline
3.4M
2.91%
8
One-Shot Baseline
3.7M
2.84%
20
MAENAS-Gradien
4.3M
2.40%
20
A.4 NETWORK ARCIITECTURE
This section shows the details of network structures which we searched for NASNet-like search space and ShuffleNet-like search space.
NhuffleNet-like search space. We designed search space based on ShuffleNerV2. Specifically, construct over-parameterized network with 20 layers, each layer choose between four operations:
Shuffle Convolution 3x3
Shuffle Convolution 5x5
Shuffle Convolution 7x7
Shuffle Xception 3x3
ReLU non-linearity are replaced by Swish if conv 5x5, conv 7x7 or Xception are chosen. Squeeze- and-Exited module are also adopted if conv 7x7 and Xception are chosen. We utilized VAENAS combined with one-shot framework to do the search.
NASNet search space. We modify the search space of DARTS (Liu et al. 2018b). A cell is directed acyclic graph, and every edge in the graph represents kind computation operation. Different from DARTS, in our search space, one node could have more than two predecessors in one cell. The maximum number edges in one cell is 3 4 = 14, while 'None' is an available operation in our search space. The list of candidate operations is listed as follows:
None
Average Pooling 3x3
Max Pooling 3x3
Skip Connect
Separate Convolution 3x3
Separate Convolution 5x5
Separate Convolution 7x7
Dilated Convolution 3x3
The magnitude of search space is 82x14, while normal cell and reduction cell are searched jointly. We present the cell searched by our methods in Fig.4 and Fig.5.
A.5 SUPER-NETWOR AND VAENAS ON MAS-BENCHA 101 SEARCH SPACE
Super-network. NAS-Benche 101 is rell-based search space, and the details of NAS-Bench-IDD 101 search space introduced in Ying et al. (2019). While HAS-Bench. 101 not proposed for weight- sharing NAS methods, we design one super-network which covers architectures in
The output of one cell in NAS- MAS-Bench-100 101 search space is the concatenation of input nodes. The illustration Fig.6 shows sampled cell on the super-network. However, the output channels of
12

---

Under review as a conference paper at ICLR 2020
280200105
20022
200204100
20020
149160000
14:14030
14x
14:144200
740072
20022
7:00 120112000
Xasption
sF 107-
SF S5511
Xegs ae
S 0Ax
Con Xx
Figure 3: VAENAS-OS ShuffleNet architectures. We highlight the input and output tensor shape. Conv denotes convolution layer. SF denotes ShuffleNetV2 module. SE denotes Squeeze-and- Excitation module.
sep. comu. 9x3
sap conn
<.Kk
CONTV 333
C
sony 5
300 CNTY 3.
skip cot
sep comy 33
5ep conv seg_corv.55 5xc
(k- 1)
cony 3x3
Figure 4: VAENAS-OS normal cell trained on CIFAR-10.
seip connert
Lony 3x3
Wkip connect
skip connest
c.(k-2)
Seg Cane 36l
Aug pool 2u3
Dee poo) 3x3
a.g_pooo 1o.. 3r3
dlip connect
pooi 3x3
shpp connect
c.(k 1)
eig pool 3x5
Figure 5: VAENAS-OS reduction cell trained CIFAR-10.
13

---

Under review as a conference paper at ICLR 2020
Input
Channels=128
1x1
Channels- 64
1x1
3x3
Channels=64
3x3
Channels=64
33x
3x3
3x3
KxK
Channels=64
C
Channels=64
Figure 7: One example of weight-shared ten- sors for convolution kernel. O
Cutput
K is the size of kernel. The blue cube shows the whole tensor, and the green eube shows K K con-
Channels=128
the part used for the 2 volution operation.
Figure 6: One sampled cell on NAS-Bench- 101 super-network. Every path contains 9 edges, while some edges may be useless in the cell.
one middle node is fixed, so weight- shared tensor is utilized to save the parameters of convo- lutional operations. The illustration Fig.7 shows one example of dividing convolutional kernel for computation.
Representation and Sampling. In NAS-Bench-Ie 101 dataset, one architecture can be represented as a 7x7 upper-triangl, adjacent matrix and 5 operations. To simply the representation of architectures, the adjacent matrix is converted to list of edges
{ai, 02..... a2 E {0,1), 1}, and operations compose vector op = {op1, op2, ..., ops ) op: E {convl x1, come3x3, marpool). The integrated concal op).
architecture is a vector
Two constraints of edges determine the validity of one sampled cell. One is the number edges in one cell is more than 9 in cell. The other is that at least one path connect the input node and output node in a cell, which indicate the connectivity of the cell. Under these two constraints, we design the following sampling strategy. 21 number in range [0. is sampled by random sampling or VAE sampling, then the indexes of top-9 numbers is selected as IDX. We set a, = when ag IDX, and aj = 0 when j & IDX. If the sampled cell is not connected, we perform above sampling process again.
Evolution Search NAS-Bench-101. The setting of evolution search follows (Guo et al. 2019 In addition, to make sure the validity of descendants, the mutation and crossover of the edge list should be planned carefully.
For mutation, an index i, a; = and another index a, = O are selected randomly. Another new sample a' is generated by exchanging a, and ay. We repeat this process until a' satisfy connectivity.
For crossover, two edge lists a" and a are selected randomly. Then an index i, of = 1, a = 0 and another index j. a5 a are selected. One new sample a' generated by exchanging a² and a.. If a' does not maintain connectivity, it will be dropped, and the crossover process will be repeated.
14


Reviews:
[{"review_id": "SylMS-x2tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "============ comments after rebuttal\nI would like to thank the authors for addressing some of my concerns. I believe the new results under Q2 and Q3 are useful additions to strengthen the paper.\n\nAs for the authors' comments for Q1, I'd like to point out that a \"larger\" search space is not necessarily more difficult (a more meaningful metric would be the average accuracy of random architectures). It is still possible that the current search space is putting the proposed method at advantage, especially if having dense connections is a useful prior.\n\nGiven the above, I would like to increase my score from 1 to 3 (weak reject).\n\n============ previous comments\nNeural architecture search can be formulated as learning a distribution of promising architectures (the sampling policy). Such a distribution is usually represented in a fully factorized fashion (e.g., as a set of multinational distributions as in DARTS). This paper proposes to model the architecture distribution using a VAE instead, where the encoder and decoder are implemented using LSTMs. The authors argue that the increased flexibility of the sampling policy leads to improved performance on CIFAR-10, NASBench and ImageNet.\n\nThe idea of representing the architecture distribution using VAEs is very natural, which in principle could offer better coverage over interesting regions in the search space as compared to traditional factorized distribution representation (which has a single mode only).\n\nWhile the method itself is interesting, I do not think it has been properly backed up by controlled experiments. This is largely due to the fact that the authors are comparing their method against baselines in fundamentally different search spaces. For instance:\n* For CIFAR-10 experiments, the authors mentioned in the appendix: \"Different from DARTS, in our search space, one node could have more than two predecessors in one cell\". This makes the search space very different from the existing ones as used by NASNet/AmoebaNet/DARTS/SNAS, and it hence remains unclear to what degree the resulting architecture has benefited from the increased in-degrees per node. Note the searched densely connected cells in Figure 4 & 5 in Appendix A.4 are clearly not part of the search space for many of the baselines.\n* For ImageNet experiments, the authors are using a ShuffleNet-like search space which has fundamentally different building blocks than other architecture search baselines (commonly built on top of inverted bottleneck layers). It is unclear to what degree the 77.4 top-1 accuracy @ 365 MFlops results have benefited from this different search space.\n\nWithout fair comparisons in a controlled setup, it is impossible for readers to draw any solid conclusion about the true empirical advantages of the method. I'm therefore unable to recommend acceptance for the paper at the moment, but am willing to raise my score if the authors can properly address those issues in the rebuttal.\n\nAdditional question: How can we isolate it to tell whether the gains come from the LSTMs or the VAEs? Is there any intuition why incorporating a generative sampler based on VAEs is potentially superior to method like ENAS (which involves LSTMs decoders only)?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkeu4_xZqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to use VAE to learn a sampling strategy in neural architecture search. The main idea is to use the currently high-performing networks to train a VAE from which the sampled architectures for the next iteration will likely supply both high-performing networks and better diversity coverage. The experiments are extensive, including results under various settings.\n\nThe idea is straightforward and reasonable. I do not work on neural architecture search myself, so I'm not sure how significant the experimental results are. Would the 0.1% (1.1%) absolute improvement over the second best in Table 2 (Table 3) be considered significant enough to justify the effectiveness of the proposed approach? \n\nI'm a little concerned about the fairness of the comparison experiments. A fairly heavy computation overhead is required to train the VAE models in the proposed method. Instead of taking this overhead, wouldn't it be easier to randomly sample more architectures? Intuitively, if we spend the cost of training a VAE model instead on sampling more architectures, the end  effects could be the same. \n\nAre the numbers in Table 2 and Table 3 swapped? \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJliF3yE9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use the variational auto-encoder (VAE) to sample the network architectures. The VAE is applied to both one-short and gradient descent scenarios, and shows consistent improvement on different NAS tasks. The proposed method is reasonable, but I have two major concerns:\n\n- I wonder whether the VAE based approach will consistently converge to a good local minimum. It will be very helpful if the authors could provide robust analysis or at least the variations of testing errors.\n\n- I understand the motivation of VAE + one shot, but I am not very convinced by VAE + gradient-based. In the last paragraph in Section 4, the paper claims (1) VAENAS-G can increase the diversity of architectures, which can be also achieved by sampling the data set. Also, it claims (2) VAENAS-G helps to search for large models, which  I do not see experimental supports. \n\nDetailed comments:\n- Algorithm 1: it is confusing to use S_K and S_k without explanation\n- Table 1: most of the other methods provide test error with standard variations. To be fair, I'd see VAENAS's test error variation. \n- Tables 2 and 3: Why is VAENASNet (table 3) different from VAENAS-G and VAENAS-OS?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to represent the distribution w.r.t. which neural architecture search (NAS) samples architectures through a variational autoencoder, rather than through a fully factorized distribution (as previous work did). 

In the discussion, a few things improved (causing one reviewer to increase his/her score from 1 to 3), but it became clear that the empirical evaluation has issues, with a different search space being used for the method than for the baselines. There was unanimous agreement for rejection. I agree with this judgement and thus recommend rejection.
Decision: Reject