Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets
Authors: Hayeon Lee
Eunyoung Hyung
Sung Ju Hwang
Venue: ICLR 2021 Poster (2021)
Keywords: Machine Learning
Neural Architecture Search
Meta-learning

Abstract: Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform human-designed networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets). Moreover, since such task-specific methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited. In this paper, we propose an efficient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search for a neural architecture for a novel dataset. The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning. Moreover, we also propose a meta-performance predictor to estimate and select the best architecture without direct training on target datasets. The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance. We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years. Code is available at https://github.com/HayeonLee/MetaD2A.

Published as a conference paper at ICLR 2021
RAPID NEURAL ARCHITECTURE SEARCH BY LEARNING TO GENERATE GRAPHS FROM DATASETS
Hayeon Lee'; Eunyoung Hyung',, Sung Ju Hwang!
KAIST AITRICS", South Korea
(hayeon926, 30 s sjhwang82 (0kaist.a)
ABSTRACT
Despite the success of recent Neural Architecture Search (NAS) methods on var- ious tasks which have shown to output networks that largely outperform human- designed networks, conventional NAS methods have mostly tackled the opti- mization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets). Moreover, since such task-specific methods search for a neural architecture from scratch for ev- ery given task, they incur large computational cost, which is problematic when the time and monetary budget are limited. In this paper, we propose an effi- cient NAS framework that trained once database consisting of datasets and pretrained networks and can rapidly search for neural architecture for a novel dataset. The proposed MetaD2A (Meta Aataset-o-AAchitcccurr) model can stochastically generate graphs (architectures) from a given set (dataset) via cross-modal latent space learned with amortized meta-learning. Moreover, we also propose a meta- meta--erroormnnc predictor to estimate and select the best archi- tecture without direct training on target datasets. The experimental results demon- strate that our model meta-learned from JAS--Bnnch 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-I00, with average search time of 33 GPU seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, transferable NAS method, with comparable performance. We believe that the MetaD2A proposes a new research direction for rapid NAS well as ways utilize the knowledge from rich databases of datasets and architectures accumulated over the past years. Code is available at https://gtthu..com/ wwt.:uubbooo/aaaaaeeeee ho..a//....o/oaooooeeeeeaDADAAD
subsets of ImageNet-1 K and architectures
INTRODUCTION
The rapid progress in the design of neural architectures has largely contributed to the success of deep learning many applications (Krizhevsky al., 2012; Cho et al., 2014; He al., 2016; Szegedy et al.; Vaswani et al., 2017; Zhang et al., 2018). However, due to the vast search space, designing novel neural architecture requires a time-consuming trial-and-error search by human experts. To tackle such inefficiency in the manual architecture design process, researchers have proposed vari- ous Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform numan-designed counterparts (Baker et al., 2017; Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2018; Luo et al., 2018; Pham et al., 2018; 8; Liu et al., 2019; Xu et al., 2020; Chen et al., 2021).
Recently, large benchmarks for NAS (NAS-101, NAS-201) (Ying et al., 2019; Dong Yang, 2020) have been introduced, which provide databases of architectures and their performances on bench- mark datasets. Yet, most conventional NAS methods cannot benefit from the availability of such databases, due to their task-specific nature which requires repeatedly training the model from scratch for each new dataset (See Figure/ Left). Thus, searching for an architecture for a new task (dataset) may require large number of computations, which may be problematic when the time and mon-
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2021
Conventional NAS Approach
Our NAS Approach Repld Search
Our Meta-Learning Framework
Heavy Computation Cost
NAS Model
NAS
NAS Model
Our Meta- Trained NAS Model
Search Space
U Training
N
O Training
NV
C Training
No Betraining Target
Volid Zone
Meta-Training
Mata-Test
Targee Datasett
Target Butawee
Torget Datweet
Target Datasset
Toargt
Tarpet
Doraset
Oataeet
Search Cost O(N)
Search Cost O(1)
Most conventional NAS approaches need repeatedly train NAS model on each given target dataset, which results in enormous total search time on multiple datasets. Middle: We propose novel NAS framework that generalizes to any new target dataset to generate specialized neural architecture without additional NAS model training after only meta-training on the source database. Thus, our approach cut down the search cost for training NAS model on multiple datasets from O(N) to O(1). Right: For unseen target dataset, we utilize amortized meta-knowledge represented as set-dependent architecture generative representations.
Figure 1: Left:
etary budget are limited. How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets, to better generalize over an unseen task?
In this paper, we introduce amortized meta-learning for NAS, where the goal is to learn NAS model that generalizes well over the task distribution, rather than a single task, to utilize the accu- mulated meta- weeaa-nnowledee to target tasks. Specifically, propose efficient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures, by learning generate a neural architecture from a given dataset. The proposed MetaD2A (Meta framework consists of set encoder and a graph decoder, which are used to learn a cross-modal la- tent space for datasets and neural architectures via amortized inference. For a new dataset, MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations, which are encoded from new dataset. and selects the final neural architecture based on their pre- dicted accuracies by a performance predictor, which is also trained with amortized meta-learning. The proposed meta-learning framework reduces the search cost from O(N) to O(1) for multiple datasets due to no training on target datasets. After one-time building cost, our model only takes just a few GPU seconds to search for neural architecture on unseen dataset (See Figure 1).
We meta-learn the proposed MetaD2A on subsets ImageNet- IK and neural architectures from the NAS-Bench201 search space. Then we validate it to search for neural architectures on multiple unseen datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100 100, Aircraft, and Oxford- Pets. In this experiment, meta- hetaaleerred model obtains neural architecture within 33 GPU seconds on average without direct training on a target dataset and largely outperforms all baseline NAS models. Further, compare our model with representative transferable NAS method (Lu al., 2020) on MobileNetV3 search space. We meta-learn our model on subsets of ImageNet- IK and neural architectures from the MobileNetV3 search space. The meta-learned our model successfully generalizes, achieving extremely fast search with competitive performance on four unseen datasets such as CIFAR- 10, CIFAR-100, Aireraft, and Oxford-III Pets.
To summarize, our contribution in this work is threefold:
We propose novel NAS framework, MetaD2A, which rapidly searches for neural archi- tecture on a new dataset, by sampling architectures from latent embeddings of the given dataset then selecting the best one based on their predicted performances.
To this end, propose to learn a cross- modal latent space of datasets and architectures, by performing amortized meta-learning, using a set encoder and a graph decoder on subsets of ImageNet-IK.
The meta-leamnd our model successfully searches for neural architectures on multiple un- seen datasets and achieves state-of ffite--f-thhe-eee art performance on them in NAS- AAS-Bench221 search space, especially searching for architectures within 33 GPU seconds average.
2

---

Published as a conference paper at ICLR 2021
Dotaset
Encoding (e(z(D)
Decoding (e((/x)
La
dependent fu(A)2)55)
Predictor
Target Dataset
No Retralsing forget
Neural
Arch.
Tosk
Graph
MetaDZA (0t 4 au
Deployment (#". ar)
Graph
CIFARID
Latent Space
MetaD2A
Graph
Srtt-dopemdenttereeeeee for graph generation
Meta-training
Metartest (Searching)
Figure 2: Overview of MetaD2A The proposed generator with θ and & meta-learns the set-dependent graph representations on the meta- meta-training tasks, where each task contains subset of ImageNet- IK and high high-quality architecture for the subset. The proposed predictor with meta-learns to predict performance, considering the dataset well the graph. In the meta-test (searching) phase, the meta-leamed MetaD2A generalizes output set-specialized neural architecture new target datasets without additional NAS model training.
RELATED WORK
Neural Architecture Search (NAS) NAS is automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space. NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le. 2017; Zoph et al., 2018; Pham et al., 2018) 8), evolutionary algorithm-basee -based methods (Real et al., 2019: Lu al., 2020), and gradient- based methods (Liu et al., 2019; Cai et al., 2019; Luo al., 2018; Dong & Yang, 2019b; Chen et al., 2021; Xu et al., 2020; Fang et al., 2020). Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al., 2018), which maps DAGs onto continuous latent embedding space. However, while NAO performs graph reconstruction for a single task, ours generates data Sata-dependent Directed Acyclic Graphs (DAGs) across multiple tasks. Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space (Cai et al., 2019; Liu et al., 2018; Pham et al., 2018; Liu et al., 2019; Chen et al., 2021). GDAS (Dong & Yang. 2019b) tackles this by optimizing sampled sub-graphs of DAG. PC-DARTS (Xu et al., 2020) reduces GPU overhead and search time by partially selecting channel connections. However, due the task-specific nature of those methods, they should be retrained from the scratch for each new unseen task repeatedly and each will take a few GPU hours. The accuracy-predee predictor-basee transferable NAS called NSGANetV2 (Lu et al., 2020) alleviates this issue by adapting the ImageNet- I pre-trained network to multiple target datasets, however, this method is still expensive due adapting procedure on each dataset.
Meta-learning Meta-learning (learning to learn) aims to train model to generalize over a distri- bution of tasks, such that it can rapidly adapt to a new task (Vinyals al., 2016; Snell et al., 2017; Finn et al., 2017; Nichol et al., 201 8; Lee et al., 2019b; Hou et al., 2019). Recently, LEO (Rusu et al., 2019) proposed scalable meta-learning framework which learns the latent generative representa- tions of model parameters for given data in cow-dimensional space for few-shot classification. Similarly to LEO (Rusu et al., 2019), our method learns low-dimensional latent embedding space, but we learn cross-modal space for both datasets and models for task-dependent model generation.
Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-bass meta-learning (Elsken et al., 2020; Lian al., 2019; Shaw et al., 2019) have shown promising results on adapting to different tasks. However, they are only applicable on small scale tasks such few-shot classification tasks (Elsken al., 2020; Lian et al., 2019) and require high-computation time, due to the multiple unrolling gradient steps for one meta-update of each task. While some attempt bypass the bottleneck with a first-order approximation (Lian et al., 2019; Shaw et al., 2019) or parallel computations with GPUs (Shaw et al., 2019), but their scalability is intrinsically limited due to gradient updates over a large number of tasks. To tackle such a scalability issue, we perform amortized inference over the multiple tasks by encoding a dataset into the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update.
3 METHOD
Our goal is to output high- performifo neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and their corresponding
3

---

Published as a conference paper at ICLR 2021
neural architectures. To this end, we propose Meta Dataset-to Arrchttectte (MetaD2A) framework which learns the tross-modal latent space of datasets and their neural architectures. Further, we introduce settormerfomecce predictor, which predicts accuracies of given architectures without training the predictor on an unseen target dataset. Overview of the proposed approach is illustrated in Figure 1.
3.1 META-TRAINING NAS MODEL
To formally define the problem, let us assume that we have a source database of N,, number of tasks, where each task T = {D, G. 8) consists of a dataset D, a neural architecture represented as Directed Acyclic Graph (DAG) G and an accuracy 8 obtained from the neural architecture ç trained on D. In the peeattraining phase, the both tetuuse-tt--rchiteettue generator and meta-pre meta-predictor learn to generalize over task distribution p(T) using the source database. We describe how to empirically construct source database Section 4.1.1.
3.1.1 LEARNING TO GENERATE GRAPHS FROM DATASETS
We propose a dataset-to dataset-to-architee architecture generator which takes a dataset and then generates high-quality architecture candidates for the set. We want the generator to generate even novel architectures, which are not contained in the source database, at meta-test. Thus, the generator learns the continuous cross-modal latent space of datasets and neural architectures from the source database. For each task the generator encodes dataset D vector through the set encoder q0 (z(z (z(D) parameterized by & and then decodes a new graph Ć from z which are sampled from the prior p(z) by using the graph decoder pe (z) parameterized by 0. Then, our goal is that & generated from to be the true G which is pair of 'D. We meta-learn the generator using set-amortized inference, by maximizing the approximated evidence lower bound (ELBO) as follows:
max 0.0
Σ (D,9)
(1)
T-p((T)
where
C (% = =E (2 log po (9 [)]] - A L
(2)
Each dimension of the prior p(z) factorizes into (N, 1). T.k is the KI. divergence between two multivariate Gaussian distributions which has simple elosed form (Kingma & Welling, 2014) and A is the scalar weighting value. Using the reparameterization trick on z, we optimize the above objective by stochastic gradient variational Bayes (Kingma & Welling, 2014). We use set encoder described in Section 3.1.3 and we adopt Graph Neural Network (GNN)-based decoder for directed acyelic graph (DAG)s (Zhang et al., 2019), which allows message passing to happen only along the topological order of the DAGS. For detailed descriptions for the generator, see Section A of Suppl.
3.1.2 PERFORMAE PREDICTOR
While many performance predictor for NAS have been proposed (Luo et al., 2018; Cai al., 2020; Lu et al., 2020; Zhou et al., 2020; Tang et al., 2020), those performance predictors repeatedly col- lect architecture-accuracy database for each new dataset, which results in huge total cost on many datasets. Thus, the proposed predictor (s (D,)) G) takes dataset well as graph support multiple datasets, while the existing performance predictor takes a graph only. Then, the proposed predictor meta-learns set-dependent performance proxy generalized over the task distri- bution P(T) in the meta-trainin meta-training stage. This allows the meta-learned predictor to accurately predict performance unseen datasets without additional training. The proposed predictor fwr consists of a dataset encoder and a graph encoder, followed by two linear layers with relu. For dataset encod- ing, use the set encoder of Section 3.1.3 which takes D an input. We adopt direct acyclic graph encoder (Zhang et al., 2019) for DAG G (Please refer to Section B of Suppl.). We concate- nate the outputs of both graph encoder and the set encoder, and feed them to two linear layers with relu to predict accuracy. We train the predictor fa to minimize the MSE loss (s, D, G) between the predicted accuracy and the true accuracy s of the model on each task sampled from the source database:
input to
min Σ (,,,, Σ (o- Tmp(T)
(3)
Tmp(TT
4

---

Published as a conference paper at ICLR 2021
3.1.3 SET ENCODER
The efficacy of the proposed framework is dependent how accurately set encoder captures the distribution the target dataset and extracts information related with the goal of the generator and the predictor. To compress the entire instances from dataset 'D into single latent code z, the set encoder should process input sets of any size and summarize consistent information agnostically to Existing set encoders such DeepSet (Zaheer
the order of the instances (permutation-invariinnn et al., 2017), SetTransformer (Lee et al., 2019a), and StatisticsPooling (Lee et al., 2020) fulfill those requirements and might used. However, DeepSet and SetTransformer are non-hierarchical pool- ings, thus cannot accurately model individual classes in the given dataset. Moreover, DeepSer and StatisticsPooling resort to simple averaging of the instance- wise representations.
Therefore, we introduce novel set encoder which stacks two permutation invvarian modules with attention-based learnable parameters. The lower-level intra-class encoder captures the class proto- types that reflect label information, and the high-level inter-class encoder considers the relationship between class prototypes and aggregates them into a latent vector. The proposed structure of the set encoder models high-order interactions between the set elements allowing the generator and predictor to effectively extract useful information to achieve each goal.
Specifically, for given dataset D {x, Y), where X (X.)= and Y (Ye) the set of instances and target labels of C' classes respectively. We randomly sample instances (x|x Be} Rb of class c, where is d, dimensional feature vector, B X, and Bell = be- We input the sampled instances into the IntraSetPool, the intra-class encoder, to encode class prototype Ve Rlxd,, for each class 1 ..... C. Then we further feed the class-specific set representations {ve c= into the InterSetPool, the inter-class encoder, to generate the dataset representation h. E R 11dd follows:
1C
Ve IntraSetPool [x|x E Bet), he InteSeePPol((vv11
Both the set poolings are stacked attention -based blocks borrowed from Lee et al. (2019a). Note that while Lee et al. (2019a) is attention-based set encoder, it ignores class label information of given dataset, which may lead to poor performance. Please see Section C of the Suppl. for more details.
3.2 META-TEST (SEARCHING)
In the meta-test stage, for an unseen dataset D), we can obtain n set-dependent DAGs { with the meta-traines generator parameterized by o² and 0", by feeding D as an input. Through such set- level amortized inference, our method can easily generate neural architecture(s) for the novel dataset. The latent code E R' xd, can be sampled from a dataset-conditionn Gaussian distribution with diagonal covariance where NN NN, are single linear layers:
2 (z)D) NN,, where μ.. NN NN,)
In the meta-test, the predictor for (D G:) predicts accuracies { for given unseen dataset D and each generated architecture of (9.7% and then select the neural architecture having the highest predicted accuracy among [
4 EXPERIMENT
We conduct extensive experiments validate MetaD22 framework. First, we compare our model with conventional NAS methods on NAS-Bench-201 search space in Section 4.1. Second, we com- pare our model with transferable NAS method under a large search space in Section 4.2. Third, we compare model with other Meta-NAS approaches few-shot classification tasks in Section 4.3. Finally, we analyze the effectiveness of our framework in Section 4.4.
4.1 NAS-BENCH-200 SEARCH SPACE
4.1.1 EXPERIMENT SETUP
We learn our model souree database consisting of subsets of ImageNet-1K 1K and neural architec- tures of
(Dong & Yang, 2020) and (meta- test our model by searching for architec- tures on 6 benchmark datasets without additional NAS model training.
5

---

Published as a conference paper at ICLR 2021
NAS-Bench-201 search space contains cell-based neural architectures, where each cell is repre- sented as directed ayclic graph (DAG) consisting of the 4 nodes and the 6 edge connections. For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip con- nection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.
Source Database To meta-learn our model, we practically collect multiple tasks where each task consists of (dataset, architecture, accuracy). We compile ImageNet-1K IK (Deng et al., 2009) as mul- tiple sub-sets by randomly sampling 20 classes with an average of 26K images for each sub-sets and assign them to each task. All images are downsampled by 32 x 32 size, We search for the set- specific architecture of each sampled dataset using random search among high-quality architectures which are included top-5000 performance architecture group on ImageNet- -ivee-16-11 16- 622 or GDAS (Dong & Yang. 2019b). For the predictor, we additionally collect 2,920 tasks through random sampling. We obtain its accuracy by training the architecture on dataset of each task. We collect N,- =1,3 =1,310/4,230 meta-training tasks for the generator/predicto and 400/400 vela-vaaio tasks for them, respec- tively. Meta-training time is 12.7/8 GPU hours for the generator/the predictor and note that meta- training phase is needed only once for all experiments of NAS-Bench-22 search space.
Meta- Test Datasets We apply our model trained from source database to 6 benchmark datasets such as 1) CIFAR-10 (Krizhevsky et al., 2009), 2) CIFAR- 100 (Krizhevsky et al., 2009), 3) MNIST (Le- Cun & Cortes, 2010), 4) SVHN (Netzer et al., 201 1111),55 5) Aircraft (Maji et al., 2013). and 6) Oxford- IITT Pets (Parkhi et al., 2012). On CIFAR1O and CIFAR 100, the generator generates 500 neural architectures and we select 30 architectures based on accuracies predicted by the predictor. Follow- ing SETN (Dong & Yang, 2019a), we retrieve the accuracies of N architecture candidates from the NAS-bench-201 and report the highest final accuracy for each run. While N 1000 in SETN, we set a smaller number of samples (N = 30) for MetaD2A. We report the mean accuracies over 10 runs of the search process by retrieving accuracies of searched architectures from MAS--Bnnc-2211 On MNIST, SVHN, Aireraft, and Oxford-IIIT Pets, the generator generates 50 architectures and select the best one with the highest predicted accuracy. we report the accuracy averaging over runs with different seeds. For fair comparison, the searched architectures from our model are trained on each target datasets from the scratch. Note that once trained MetaD2A can be used for more datasets without additional training. Our model is performed with single Nvidia 2080ti GPU.
4.1.2 RESULTS ON UNSEEN DATASETS
Table 1 shows that our model meta meta-learmed on the source database can successfully generalize to 6 unseen datasets such as MNIST, SVHN CIFAR-10, CIFAR- 100, Aircraft, and Oxford- IIIT Pets by outperforming all baselines. Since the meta-learned MetaD2A can output set-specialized architec- tures on target datasets through inference process with no training cost, the search speed is extremely fast. As shown in Table 1, the search time of MctaD2A averaging 6 benchmark datasets is within 33 GPU second. This is impressive results in that it is at least 147x x (maximum: 12169 x) faster than conventional set-specific NAS approaches which need training NAS models on each target dataset. Such rapid search of REA, RS, REINFORCE and BOHB is only possible where all of the accu- racies are pre-computed like NAS- Bennh2n that it can retrieve instantly the target dataset, therefore, it is difficult to apply them to other non-benchmark datasets. Especially, we observe that MetaD2A which is learned over multiple tasks benefit to search set-dependent neural architectures for fine-grained datasets such as Aircraft and Oxford Pets.
4.2 MOBILENETV3 SEARCH SPACE
4.2. EXPERIMENT SETUP
We apply our meta-trained model on four unseen datasets, comparing with transferable NAS (NS- GANetV2 (Lu et al., 2020)) under the same search space of MobileNetV3, where contains more than 1019 architectures. Each CNN architecture consists of five sequential blocks and the targets of searching are the number of layers, the number of channels, kernel size, and input resolutions. For fair comparison, we also exploit the supernet for the parameters as NSGANetV2 does. We collect N, = 3, 018/153, 408 meta-training tasks for the generator/predictor and 646/32, /2, validation tasks, respectively as a source database from the ImageNet- IK dataset and architectures of MobileNetV3 search space. Meta-training time is 2.21/1.41 GPU days for the generator/the predictor. Note that the meta-training phase is needed only once the source database.
872 meta-

---

Published as a conference paper at ICLR 2021
Table 1: Performance on Unseen Datasets (Meta-Test) MetaD2A conducts amortized inference on unseen target datasets after training on source database consisting of subsets of ImageNet- IK and architectures of O'NAS-Bee LuS--ench-201 search space. Meta- veta-training time is 12.7/8.4 4 GPU hours for the generator/the predictor. For fair comparison, the parameters of scarched architectures transferring parameters from ImageNet. the time construct precomputed architecture database for each target. report accuracies with 95% confidence intervals.
trained each dataset from scratch instead of
Target Dutaset
NAS Method
NAS Training-free
Target
Params (M)
Search Time
(GPU Sce)
Speed Up
Search Cost
Accuracy (%)
2016]
93.97
REA (Real et al,, 2019)
0.02 +
93.92..410
(Bergstra Bengio, 2012)
93.70=028
REINFORCE (Williams, 1992)
0.12+1'
93. -1.3
BOHB (Fulkner 2018)
3.59- T
93.61
CIFAR-10
RSPS (LF Talwaka. 20/00
10200
14%
4.1
3-507=56)
SETN (Dong & Yang, 2019a)
30200
437x
12.25
87 00.64
GDAS (Dong Yang, 2019b)
363x
1.008
PC AC-DARTS (Xu al.,, 2020)
1.17
10395
150%
4,21
93.661:41
DrNAS (Chen 2021)
1.53
21760
315x
8.82
94. 94.36.1:10
MetaD2A (Ours)
1.11
69
1x
0.028
94.37 1.0
ResNet al.,
0.86
N/A
70.86:9100
REA (Real et al., 2019)
0.02+T
84
(Bergstra Dengio, 2012)
0.01+T
REINFORCE (Williams, 1992)
0,12+T
71.71
BOHB (Falkkerr al. 2018)
3.59+T
70.85
CIFAR-100
RSPS (LT Talwakkr 20/9)
13KIT
1967
32
SETN (Dong & Yang. 2019t)
58808
612x
23.85
59.09=
GDAS (Dong Yang, 2019b)
51580
S37x
20.91
70. 70.70
PC- PC-DARTS (Xu al., 2020)
0.26
19951
207
8.09
66.64
DINAS (Chen 2021)
1.20
34529
359X
14.00
73.51 s:
(Ours)
1.07
1x
1.44
ResNet (He al., 2016)
0,86
NIA
N/A
N/A
99.67
(Li Talwalkar, 2019)
0.25
22457
3208x
9.10
99 29.63
SETN (Dong & Yang, 2019a)
0.56
69656
9950x
28.24
99 1916
(Dong Yang. 2019b)
PC-DARTS (Xu al., 2020)
0.62
24857
3551
10.08
99 29
DrNAS (Chen 2021)
1.53
44131
6304x
17.89
99.59
MetaD2A (Ours)
0.61
1x
0.002
99.71 HaA
ResNet (He al., 2016)
0.86
N/A
N/A
N/A
96. 13:0
RSPS (Li & Talaakkrr 2019)
0.48
27962
3894
11.34
96. 171112
SETN (Dong & Yang, 3019a)
0.48
85189
12169
34.54
96.02+2.1
SVHN
GDAS (Dong Yang. 2019b)
0.24
71595
10227
10.17
55.57
PC-DARTS (Xu et al., 2020)
0.47
31124
4446x
12.62
95.40
DrNAS (Chen 2021)
52791
7541x
96-30+210
MetaD2A (Ours)
0.86
1x
0.004
96.34:au
ResNet
RSPS (L) & Talwalkar, 2019)
0.22
18697
1869%
7.58
42. 19
SETN (Dong & Yong, 2019s)
0.44
18564
1856
14
Aircratt
GDAS (Dong & Yang. 2019b)
0.62
18508
1850x
7.50
52
PC-DARTS (Xu al. 2020)
0.32
3524
352*
1.42
26:33
(Chen et 2021)
3452x
17.08
MetaDZA (Ours)
0.83
10
L
0.004
58.43
ResNet ol., 2016]
N/A
RSPS (1. & Talwalkar, 2019)
0.32
3360
420x
1.36
22.91 11:0
(Dong & Yang.
25.17
Oxford-IIIT Pers
GDAS (Dong Yang, 2019b)
0.83
6965
870x
2.82
24.02.215
PC-DARTS (Xu al., 2020)
0.44
2844
355X
1.15
25.31
DrNAS (Chen et al., 2021)
0.44
6019
752X
2.44
26.73-22)
MetaD2A (Ours)
0.83
1X
0.003
41.50
CIFAR10
CIFAR100 D D16GANUV2
AIRCRAFT
Oxford-IIIT-Pen
G K5GA/WtV2
MetaD2A
MetaDZA
NSLANAIT Metatzz
CINGANIV2 MecadiA
98.3
25.35
97.9
26.5
%0
97.7
85.
97.5
B5-7
85
34.55
360
250
300
320
265
300
355
230
JUU
350
400
256
272
(LOPS)M)
(LOPS(M)
FLOPSIM]
FUOPSIMI
Figure 3: Performance on Unseen Datasets (Meta-Test) We show accuracy over flop both MetaD2A a transferable NAS referred as to NSGANetV2 (Lu et al., 2020) after meta-training MetaD2A on source datahase consisting of subsets of ImageNet-1K K and architectures in MobileNerV3 search space. Note that each plot point is searched within 125 GPU seconds by MetaD2A.

---

Published as a conference paper at ICLR 2021
4.2.2 RESULTS ON UNSEEN DATASETS
We search and evaluate the architecture multiple times with both NSGANeIV2 and ours on four unseen datasets such as CIFAR-IO, CIFAR-100, Aircraft, and Oxford-IIIT Pets with different ran- dom seeds. Search times of MetaD2A for CIFAR-10. 10. CIFAR-100, Aireraft, and Oxford-IIIT Pets are within 57, 195, 77, and 170 GPU seconds on average with a single Nvidia RTX 2080ti GPU respectively, while NSGANerV2 needs GPU day with 1080ti GPUs on each dataset, which 5,523 times slower than MetaD2A. Besides the huge speed up, Figure 3 shows that our model can search for comparable architecture to the NSGANetV2 over flops without performance drop. Interestingly.
we use naive flop filtering and NSGANetV2 uses objective function for flop constraints, MetaD2A performs consistently comparably to NSGANetV2 over the different flops. Overall, the results demonstrate that model also can generalize unseen datasets not only un- der the MAS-Bench-201 space but also under a larger MobileNetV3 space with its meta-knowledge.
4.3 COMPARISON WITH META-NAS APPROACHESS
We further compare our method against Meta-NAS methods (Kim et al., 2018;
Method
NAS
Params (K)
5way Isbot
MinilmageNet
5way
Elsken al., 2020; Lian et al., 2019; Shaw et al., 2019) on few-shot classification tasks,
MAML (Pinn et 2017)
32.9
48.70
63.11
MAML++ (Antonion 2018)
32.9
52.15
68.32
AutoMett (Kim
28
65.09
which are the main setting existing Meta- NAS methods have been consider. Follow- ing (Elsken et al., 2020; Lian et al., 2019). we adopt bi-level optimization (e.g.. MAML framework) to meta-learn initial weights of neural architectures searched by model the Table 2, the few-shot classification results on MinilmageNet further clearly show the MetaD2A's effectiveness over existing Meta-NAS methods, well the conventional meta-learning methods without NAS (Finn et al., 2017; Antoniou et al., 2018).
BASE (Shaw et 2019)
1200
T-NAS (Lian et al., 2019)
26.5
54.11
69.59
MetaNAS (Elskem et al., 20200
30
49.7
62.1
MetaDZA (Ours)
28.9
54.71
70.59
Table 2: Performance on Few-shot Classification Task
meta-training of mini mini-imagenet. As shown in
4.4 EFFECTIVENESS OF METAD2A
Now, we verify the efficacy of each component of MetaD2A with further analysis.
Ablation Study on MetaD2A We train different variations of our model on the subsets of ImageNet-IK, IK, and test on CI- FARIO, CIFARIO0, and Aircraft in Ta- ble 3 with the same experimental setup as the main experiments in Table 1. The MetaD2A generator without the perfor- mance predictor (Generator only) outper- forms the simple random architecture sampler (Random Sampling), especially by 15.3% Air- craft, which demonstrates the effectiveness of MetaD2A over the random sampler. Also, we observe that combining the meta-performance predictor the random architecture sampler (Predictor only) enhances the accuracy of the final architecture on all datasets. Finally, MetaD2A combined with the performance predictor (MetaD2A) outperforms all baselines, especially by 20.28% on Aircraft, suggesting that our MetaD2A can output architectures that are more relevant to the given task.
Target Dataset
Model
G
P
CIFARIO CIFAR100
Aircraft
Random Sampling
93.061ess
38.15
Generator only
93.96eaz
71.54
53.45+sa
Predictor only
93.70
72.33
53.39
MetaD22
< <
94.37
73.51
58.4321
Table 3: Ablation Study of MetaD2A on Unseen Datasets
Effectiveness of Set-to-Arreitecture Generator
MNST SAIK
The Number of Networks
1750
Random Generator
The Number of Networks
1250
Randem
1000
750
750
DIAR100
300
AIRERAT
250
40
80
80
90
NAS-Bench-201 Search Space
CIFAR10 Accuracy
CIFAR10D Accuracy
Figure& 4: T-SNE vis. of Latent Space We first visualize cross-modal latent embeddings {2} of unseen datasets encoded by the meta-
Figure 5: The Quality of Generated Architectures
8

---

Published as a conference paper at ICLR 2021
learned generator with T-SNE in Figure 4. Each marker indicates {z) of the sampled subsets of each dataset with different seeds. We observe that the generator classifies well embeddings {z} by datasets in the latent space while clusters z of the subset of the same dataset. Furthermore, investigate the quality of generated architectures from those emheddings {z). In the Figure 5, the generator sample 2000 architecture candidates from the embeddings encoded each target dataset and computes the validate accuracy of those architectures. The proposed generator generates more high-performing architectures than the simple random architecture sampler for each target dataset. These results are consistent with Table 3, where the generator (Generator only) enhances the per- formance compared with the simple random architecture sampler (Random Sampling) consistently on CIFAR10 10 and CIFAR 100. The meta-learned generator allows us to effective and efficient search by excluding the poor-performing architectures of broad search space. We believe the generator replaces the random sampling stage of other NAS methods. We leave to valid it as the future work.
Could the Generator Create Novel Architectures? Since the generator maps set-architecture pairs in the continuous latent space, it can generate novel ar- chitectures in the meta-test, which not contained in the source database. To validate it, we evaluate generated 10,000 neural architecture samples of both search space with the measures Volidity, Uniqueness, and Novelty following (Zhang et al., 2019) in Table Each is defined as how often the model can generate valid neural architectures from the prior distribution, the proportion of unique graphs out of the valid generations, and the propor- tion of valid generations that are not included in the training set, respectively. For NAS-Bench- search space and MobileNetV3 search space, respectively, the results show the meta-learned gener- ator can generate 67.31 %/100% new graphs that do not belong to the training set and can generate 35.19% various graphs, not picking always-the-sane architecture seen of the source database.
Search Space
Validity
Uniqueness
Novelty
NAS Bench-201
1.0000
0.3519 1.0000
0.6731
MobileNeIV3
0.9831
1.0000
Table 4: Analysis of Generated Architectures
Effectiveness of Meta-Predictor We first demonstrate the necessity of set encod- ing to handle multiple datasets with a single predictor. In Table 5, we meta- train all models on the source database of
Predictor Model
Inpur Type
Pearson Corr. Coeff.
Data Griph
Graph Encoder (GE) Only
0.6439
DeepSet (Zaheer et al,, 2017)+ GE
V
0,7286
SefTranformer et al., 2019s) +
0.7744
Statistical Pooling (Lee et al., 2020) GE
V
0,7796
search space and mea- sure Pearson correlation coefficient on the validation tasks (400 unseen tasks) of the source database. Pearson correlation co- efficient is the linear correlation between the actual performance and the predicted performance (higher the better). Using both the dataset and the computational graph of the target architecture as inputs, instead of using graphs only (Graph Encoder Only), clearly leads to better performance to support multiple datasets. Moreover, the predictor with the proposed set encoder clearly shows a higher correlation than other set encoders (DeepSet (Zaheer et al., 2017), SefTransformer (Lee et al., 2019a), and Statistical Pooling (Lee et al., 2020)).
The Proposed Set Encoder GE (Ours)
/
0.8085
Table 5: Effectiveness of Set Encoding to Accurately Pre- diet the Accuracy of Multiple Datasets
5 CONCLUSION
We proposed a novel NAS framework, MetaD2A (Meta Dataset-oo-Archieecturee, that can output neural architecture for an unseen dataset. The MetaD2A generator learns a dataser-to-archieeene transformation over database of datasets and neural architectures by encoding each dataset using a set encoder and generating each neural architecture with a graph decoder. While the model can generate novel architecture given a new dataset in an amortized inference, we further learn a meta-performann predictor to select the best architecture for the dataset among multiple sampled architectures. The experimental results show that our method shows competitive performance with conventional NAS methods on various datasets with very small search time as it generalizes well across datasets. We believe that our work is a meaningful step for building a practical NAS system for real- world scenarios, where we need to handle diverse datasets while minimizing the search cost.
This work was conducted by Center for Applied Research in Artificial Intel- ligence (CARAT) grant funded by DAPA and ADD (DD190031RDD) IRD).
Acknowledgements

---

Published as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml, arXiv preprint 1.Xv:1810.00900. 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini arXiv: 1607.0640, 2016.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In In International Conference on Learning Representations (ICLR), 2017.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter ootpriaiiaiiiioiiiiiitiiiiiooo The Journal of Machine Learning Research, 13(1):281-305 2012.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search target task and hardware. In International Conference on Learning Representations (ICLR), 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representationss, 2020. URL
http:p/////..9091111 pdf.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Dr(nas}: Dirichlet neural architecture search. In International Conference on Learning Representations, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Deng, W. Dong, R. Socher, I..J. Li, K. Li, and L. Fei-Fei. ImageNet: Large-Scale Hierarchi- cal Image Database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template network. In Proceedings of the IEEE International Conference Computer Vision (CVPR), 2019a.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Pro- ceedings of the IEEE Conference on computer vision and pattern recognition (CVPR), 2019b.
Xuanyi Dong and Yi Yang. Nas-bench-2011 Extending the scope of reproducible neural architecture search. In International Conference Learning Representations (ICLR), 2020.
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2020.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- 2018.
mization at scale. arXiv preprint
Jiemin Fang, Yuzhu Sun, Kangjian Peng, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Fast neural network adaptation via parameter remapping and architecture search. arXiv preprint arXiv:2001.02525 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leane -etaalleaning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
10

---

Published as a conference paper at ICLR 2021
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. International Conference on Machine Learning (ICML), 2018.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in neural information processing systems (NeurlPS), 2018.
Jaehong Kim, Sangyeul Lee, Sungwan Kim, Moonsu Cha, Jung Kwon Lee, Youngduck Choi, Yongseok Choi, Dong- Yeon Cho, and Jiwon Kim. Auto-meta: Automated gradient based meta learner search. arXiv preprint arXiv: weop:00066.0777 2018.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2014.
Auto-encoding variational bayes. arXiv preprint
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:///aaa. lecun. com/ exdb /m.ist/.
Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. In International Conference Learning Representations (ICLR), 2020.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set ionnuutationinnvarian neural networks. In Interna-
transformer: A framework for attention-based tional Conference on Machine Learning (ICML), 2019a.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019b.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2019.
Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In Interna- tional Conference on Learning Representations (ICLR), 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), 2018
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In In International Conference Learning Representations (ICLR), 2019.
Zhichao Lu, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Ns- ganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search. In European Conference on Computer Vision, pp. 35-51. Springer, 2020.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yas Yan Liu. Neural architecture optimization. In Advances in neural information processing systems (NeurIPS), 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5157 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order leta-learning algorithms. arXiv preprint arxiv 1803 029999 2018
11

---

Published as a conference paper at ICLR 2021
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Hicu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICML), 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the caai conference on artificial intelligence (AAAT), 2019.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. 2019.
Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NIPS). 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.
Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. semi-supervised assessor of neural architectures. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1810-1819, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NIPS), 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems (NIPS), 2016.
Ronald J Williams. Simple statistical folliintt algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--555 1992.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong- Pc-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations (ICLR), 2020.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning (ICML), pp. 7 7105-7114, 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems (NIPS), 2017.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and
Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11396 11404, 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In In Interna- tional Conference on Learning Representations (ICLR), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 2018.
12

---

Published as a conference paper at ICLR 2021
A DETAILS OF THE GENERATOR
A.1 GRAPH DECODING
To generate the 7t node vs we compute the operation type ou, R RIx", over no operations based on the current graph state hg := h and then predict whether the edge exists between the node 2 and other existing nodes. Following (Zhang et al., 2019), when we compute the edge probability ((yy, we consider nodes {ıyl = - I..... } in the reverse order to reflect information from nodes close to U; to the root node when deciding whether edge connection. Note that the proposed process guarantees the generation of directed acyclic graph since directed edge is always created from existing nodes to a new node.
The graph decoder starts from initial hidden state h.p NNinii()), where NN is an MLP followed by tanh. For zth node i according to topological order, we compute the probability of each operation type Ou, R. Ixn over n, operations, given the current graph state the last hidden node hg == h That is, ou, = NN (hg)), where NNee is an MLP followed by softmax When the predicted v; type the end-of-graph,
stop the decoding process and connect all leaf nodes V.. Otherwise we update hidden state (!! at time step as follows:
h = UPDATE(Í, m.! (1)
where m
(1) UL
Σ
AGGREGATE(1
(6)
MEVI
The function UPDATE is a gated recurrent unit (GRU) (Cho et al., 2014), is the order of ex and (*) is the incoming message to U4- The function AGGREGATE consists of mapping and gating functions with MLPs, where Vin is a set of predecessors with incoming edges to v4. For all previously processed nodes {ıyli = -- 1..., 1) }], we decide whether to link an edge from By to v2 by sampling the edge based on edge connection probability ((y,,.)) = NNedge (h,, h.). where NNedge MLP followed by sigmoid. We update h by Eq. (6) whenever new edge is connected to
e. For meta-test, we select the operation with the max probability for each node and edges with (fy.,.)) 0.5.
A.2
OBJECTIVE
We meta-learn the model using Eq. (I). The expectation of the log-likelihood Ex)gg((a))) log pe (G]z)) of (2) can be rewritten with negative cross-entropy loss - for nodes and binary cross-entropy loss BC for edges, and we slightly modify it using the generated set-dependent graph & and the ground truth graph G the input follows:
[TTCe( oi) + Σ LBCE (eu,))
iEV
JEV:
We substitute the likelliolhood term of Eq. (2) such as Eq. (7) and learn the proposed generator by maximizing the objective (1) to learn Φ,, 0, which are shared across all tasks.
B GRAPH ENCODING OF THE SET-DEPENDENT PREDICTOR
For given graph candidate G, we sequentially perform message passing for nodes from the pre- decessors following the topological order of the DAG G. We iteratively update hidden states h!!) using the Eq. (8) by feeding in its predecessors' hidden states {u E Vin
h (1+1) UPDATE( (Ye,, m!!) bi
where m.
AGGREGATE(1(5)
(8)
NEVIS
For starting node vn which the set of predecessors is the empty. we output the zero vector as the hidden state of vo- We use the last hidden states of the ending node as the output of the graph
13

---

Published as a conference paper at ICLR 2021
encoder hy. Additionally, exploit Bi-directional encoding (Zhang al., 2019) which reverses the node orders to perform the encoding process. In this case, the final node becomes the starting point. Thus, the backward graph encoder outputs hn which is the last hidden states of the starting node. We concatenate the outputs hy of the forward graph encoder and h2 of the backward graph encoder the final output of the Bi-directional graph encoding.
C BUILDING BLOCKS OF THE SET ENCODER
We use Set Attention Block (SAB) and Pooling by Multi-head Attention (PMA) (Lee et al., 2019a) 9a),,, where the former learns the features for each element in the set using self-attention while the latter pools the input features into & representative vectors. Set Attention Block (SAB) is attention- based block, which makes the features of all of the instances in the set reflect the relations between itself and others such as:
SAB (X) = LN(H + MLP(H))
where H = L + MH(X, (XXXXXXX))
where LN and isP the layer normalization (Ba al., 2016) and the multilayer perceptron respec- tively, and H E IR xdи is computed with multi-head attention MH(Q, K, V) (Vaswani et al., 2017) which queries, keys, and values are elements of input set X.
Features encoded from the SAB layers can be pooled by PMA on learnable seed vectors S E Rxxdd to produce & vectors by slightly modifying H calculation of Eq. (9):
PMA(X) = LN(H + MLP(H)) where H LN(X + (H MLP(X)) MLP(X)))
(10)
While &t can be any size (i.e. k=1,2,10,16) we set k for generating the single latent vector. For extracting consistent information not depending the order and the size of input elements, encoding functions should be constructed by stacking germutaation-qquivrrinn condition for any permutation æ on a set X (Zaheer et al., 2017):
layers E, which satisfies below
E(fx|x E xX)) =x([x|x X))
(11)
Since all of the components in SAB and PMA are row-wise computation functions, SAB and PMA permutation equivarint by definition Eq. (11).
D SEARCH SPACE
Following the NAS- Bench-201 (Dong & Yang, 2020), We explore the search space consisting of 15,625 possible cell-based neural architectures for all experiments. Macro skeleton is stacked with one stem cell, three stages consisting of 5 cells for each, and residual block (He et al., 2016) between stages. The stem cell consists of 3-by-3 convolution with 16 channels and cells of the first, second and third stages have 16, 32 and 64, respectively. Residual blocks have convolution layer with the stride 2 for down-sampinn sampling. A fully connected layer is attached to the macro skeleton for classification. Each cell is DAG which consists of the fixed 4 nodes and the fixed 6 edge connections. For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling. To effectively encode the operation information as the node features, we represent edges of graphs in NAS- -AS-Bench- 201 nodes, and nodes of them edges. Additionally, to the cell during training. All nodes which have no predecessors (suceessors) are connected to the starting (ending) node, which we delete after generating the full neural architectures.
add starting node and an ending node
E EXPERIMENTAL SETUP
E. DATASET
1) CIFAR-10 (Krizhevsky et al., 2009): This dataset is a popular benchmark dataset for NAS, which consists of 32x32 colour images from 10 general object classes. The training set consists of 50K images, 5K for each class, and the test set consists of 10K images, IK for each class. 2) CIFAR- 100 (Krizhevsky et al., 2009): This dataset consists of colored images from 100 fine-grained general
14

---

Published as a conference paper at ICLR 2021
object classes. Each class has 500/100 images for training and test, respectively. 3) MNIST (LeCun & Cortes, 2010): This is a standard image classification dataset which contains 70K 28 x8 grey colored images that describe 10 digits. We upsample the images to 32x32 pixels to satisfy the minimum required pixel size of the NAS-Bench 201 due to the residual blocks in the macro skeleton. We use the training/test split from the original dataset, where 60K images are used for training and 10K are used for test. 4) SVHN (Netzer et al., 2011): This dataset consists of 32 x2x32 color images where each has digit with a natural scene background. The number of classes is 10 denoting from digit 1 to 10 and the number of training/test images is 73257/26032, respectively. 5) Aireraft (Maji al., 2013) This is fine-gratd mne-grained classification benchmark dataset containing 10K images from 30 different aircraft classes. We resize all images into 32x 32. 6) Oxford-IIIT Pets (Parkhi et al., 2012) This dataset is for fine-grained classification which has 37 breeds of pets with roughly 200 instances for each class. There is no split file provided, so we use the 85% of the dataset for training and the other 15% are as a test set. We also resize all images into 32x32. For CIFAR10 and CIFARIOU we used the training, validation, and test splits from the NAS- -AA--Bench-201 and use random validation/test splits for MNIST, SVHN, Aircraft, and Oxford-IIIT Pets by splitting the test set into two subsets of the same size. The validation set is used update the searching algorithms as supervision signal and the test set is used to evaluate the performance of the searched architectures.
E.2 BASELINES
We now briefly describe the baseline models and our MetaD2A model. 1) ResNet (He et al., 2016) This is a convolutional network which connects the output of previous layer as input to the current layer. It has achieved impressive performance on many challenging image tasks. We use ResNet56 in all experiments. 2) REA (Real et al., 2019) This is an evolutional basedd search method by us- ing aging based tournament selection, showing evolution can work in NAS. 3) RS (Bergstra & Bengio, 2012) This is based random search and we randomly samples architectures until the total time of training and evaluation reaches the budget. 4) REINFORCE (Williams, 1992) This is RL-bbased NAS. We reward the model with the validation accuracy after 12 epochs of train- ing. 5) BOHB (Falkner et al., 2018) This combines the strengths of tree-structured parzen estimator based baysian optimization and hyperband, performing better than standard baysian optimization methods. 5) RSPS (Li & Talwalkar, 2019) This method is combination of random search and weight sharing, which trains randomly sampled sub- tib-rrapss from weight shared DAG of the search space. The method then selects the best performing sub-graph among the sampled ones as the final neural architecture. 6) SETN (Dong Yang, 2019a) SETN is an one-shot NAS method, which selectively samples competitive child candidates by learning to evaluate the quality of the candi- dates based on the validation loss. 7) GDAS (Dong & Yang. 2019b) This is a Gumbel-Softman based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs. 8) PC-DARTS (Xu et al., 2020) This is a gradient- based NAS which partially samples channels to apply operations, to improve the efficiency of NAS in terms of memory usage and search time compared to DARTS. We exploit the code at ht https: : / / h/gitttuuu thub. com/ yuhui i xul 993/2C-DARTE
9) DrNAS (Chen al., 2021) This is a NAS approach that introduces Dirichlet distribution to approximate the architecture distribution, to enhance the generalization performance of differentiable architecture search. We use the code at https:/ /gittubb com/ xi siangningg-hh chwneDheennsseeeeeeee We report the results on CIFAR10 and CIFAR100 in this paper using the provided code from the authors on the split set of NAS -ench 201 while their reported results in the paper of the authors are 94.37 and 73.51, respectively on ran- dom training/test splits on CIFARIO and CIFARI00. 10) MetaDZA (Ours) This is our meta-NAS framework described in section 3. which can stochastically generate task-dependent computational graphs from a given dataset, and use the performance predictor to select the best performing candi- dates. We follow the same settings of MAS--ench-200 (Dong & Yang, 2020) for all baselines and use the code at https: : github. com/D- D-m/DDXX x- Y ADt Pro jett except for 8), 9) and 10).
15

---

Published as a conference paper at ICLR 2021
E.3 IMPLEMENTATION DETAILS
Hyperparameter
Value
The number of inputs of class b
20
Dimension of V.. dv. Dimension of h. dn, Dimension of S ds Dimension of dz Dimension he, for generator Dimension of ha, for predictor
56
56
56
56
56
512
The number of operators no
5
Learning rate Batch size
le-4
32
KL-divergence weighting value
5e-3
Training epoch
400
Table Hyperparameter setting of MetaD2A on NAS-Bench-201 Search Space
We use embedding features as inputs of the proposed set encoder instead of raw images, where the embedding features are generated by ResNet18 (He et al., 2016) pretrained with ImageNet-IK (Deng et al., 2009). We adopt the teacher forcing training strategy (Jin et al., 2018), which performs the current decoding process after correcting the decoded graph the true graph until the previous step. This strategy is only used during meta-training and we progress subsequent generation based on the currently decoded graph part without the true graph information in the meta-test. We use mini-batch gradient descent to train the model with Eq. (1). The values of hyperparameters which we used for both MetalD2A generator and predictor in this paper are described in Table 6. To train searched neural architectures for all datasets, we follow the hyperparameter setting of NAS-Bench- 201 (Dong & Yang, 2020), which is used for training searched neural architectures on CIFAR and CIFAR 100. While report accuracy after training 50 epoch for MNIST, the accuracy of 200 epoch are reported for all datasets except MNIST.
16


Reviews:
[{"review_id": "Xi63zgnTCkk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1.\tThis paper proposes new scene, where a meta model is pre-trained on some datasets, and transfer the learned meta feature onto other datasets to do fast adoption. This scene can be applied in variety of domains.\n2.\tThe experiment shows this method can fast adapt NAS from one image dataset to others and achieve SOTA performance. \n3.\tThe paper is well-organized, the paper structure is clear.\n\nCons:\n1.\tThe three parts of your model are of little novelty. The dataset encoding part is just borrowed and there is no improvement to adapt NAS tasks. Similar graph decoder is proposed in previous NAS works [1] and performance predictor are proposed more times. It seems that the proposed framework is just putting existing models together. \n2.\tNo comparing to other methods on fast adaptation by NAS such as [2]. Besides, meta-learning methods may also be compared.\n3.\tAccording to Figure 8 and Figure 9, it is likely that your graph decoder can only generate one type of edge connections. Your graph decoder may fail. Since your framework needs other methods (GDAS / NAS-Bench-201) to provide training material. These materials are all good architectures. It is possible that your framework just gives architectures the same as those good architectures rather than using meta features. Your experiment should prove that your model can generate variety of architectures.\n\nOverall Review:\nThis paper proposes a new scene of fast adaption of NAS, which may be a good direction of NAS & meta-learning. The paper proposes a framework to generate good architectures according to the datasets. However, the model may need to improved and more experiment need to be done to solve the problems mentioned above.\n\n[1]Does unsupervised architecture representation learning help neural architecture search? NeurIPS 20’\n[2]Fast neural network adaptation via parameter remapping and architecture search ICLR 20’ \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NRH0FUrwmw3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address neural architecture search (NAS) scenarios. In particular, a framework, MetaD2A, is proposed, which yields a neural architecture for a new dataset. In a nutshell, the framework learns a \"dataset-to-neural-network-architecture\" transformation using a database of datasets and architectures. Each dataset is encoded via a \"set encode\" and the architecutres are obtained via a \"graph decoder\". The experiments demonstrate the usefullness of the approach and its improvements over conventual NAS approaches. The approach could be described \n\nPositive:\n- The results look very solid and indiciate improvements (time/prediction performance) over existing approaches\n- The paper is well written and structured\n- Additional details (e.g., implementation details) are provided in the appendix\n\nNegative:\n- The authors claim that NAS with meta learning has only been done with small datasets in the past. However, the authors do not really use big datasets as well (see Appendix C; the ImageNet subset considered is small as well, if I understand this correctly)\n\nTo sum up, I think the work might be a suitable candidate for being accepted at ICLR. I have to admit that this is not precisely an area of my expertise, so I might be missing something.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GVuW-hYIztc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel framework named MetaD2A. Their motivation is interesting to me and they clarify their difference compared with traditional metaNAS in Figure 1. There are mainly three components in MetaD2A: a set encoder, a graph decoder and a meta-performance predictor. I think the main contribution of this paper is their intuition that performing neural architecture search rapidly from the datasets, while the components are all proposed before in different NAS scenarios. In summary, the paper has following drawbacks that need to be further concerned:\n\n1. Since the predictor consists of two linear layers, we cannot take architecture with different nodes as input. Thus, it will limit the generalization ability of the whole algorithm. Moreover, there are also different kinds of performance predictors in NAS field like LSTM and GCN predictor [1]. And I prefer to see the effect of predictor part with different predictors.  \n\n2. In Table 1, MetaD2A is pretrained on Meta-ImageNet while other baselines are trained from the scratch. I think this is unfair to compare them. It's better to compare MetaD2A with some MetaNAS methods listed in related work part such as [2]. In my view, I think MetaD2A pays more emphasis on meta-learning in NAS field.\n\n2. In ablation study section, they only compare MetaD2A with random, which is a somewhat weak baseline. However, the improvement of their method is not significant. I think there are some other perspectives to perform ablation study. For instance, they can replace hierarchical set pooling with flatten set pooling and then see the importance of different component.\n\n[1] Shi, Han, et al. \"Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS.\" Neural Information Processing Systems. 2020.\n\n[2] Lian, Dongze, et al. \"Towards fast adaptation of neural architectures with meta learning.\" International Conference on Learning Representations. 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed a meta learning framework for NAS, namely MetaD2A (Meta Dataset-to-Architecture), that can stochastically generate graphs (architectures) from a given set (dataset) via a dataset-architecture latent space learned with amortized meta-learning. Each dataset is encoded via a set encoder and the architecutres are obtained via a graph decoder. MetaD2A is trained once on a database consisting of datasets and pretrained networks and can rapidly search a neural architecture for a novel dataset. While the set encoder and graph decoder for NAS have been introduced by existing work, the main contribution of the paper is to show that the meta-learning of a "dataset-conditioned architecture generation" framework can enable fast generation of a good architecture without training on the target dataset. The proposed method is interesting and effective, however it requires an existing pool of good architectures for a given task, which may limit its applicability to a diverse set of real-world problems. I strongly encourage the authors to include experiments on a larger pool of architectures than the NAS-Bench-201 search space to show the strength of their proposed method in generating good architectures. While training MetaD2A with pairs of MetaImageNet and randomly sampled graph shows that the proposed framework can generate graphs with different types of edges, it doesn't show that it can successfully meta-learn to produce better architectures for a new task from an existing pool of good architectures. 

We believe that many of the reviewers comments were addressed in the rebuttal, so while the scores are low, they do not reflect neither the contribution nor the reviewers opinion well (e.g., R3, in his last post, seems to suggest that his review should be updated but it has not happened).
Decision: Accept (Poster)