Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

1 INTRODUCTION . The rapid progress in the design of neural architectures has largely contributed to the success of deep learning on many applications ( Krizhevsky et al. , 2012 ; Cho et al. , 2014 ; He et al. , 2016 ; Szegedy et al . ; Vaswani et al. , 2017 ; Zhang et al. , 2018 ) . However , due to the vast search space , designing a novel neural architecture requires a time-consuming trial-and-error search by human experts . To tackle such inefficiency in the manual architecture design process , researchers have proposed various Neural Architecture Search ( NAS ) methods that automatically search for optimal architectures , achieving models with impressive performances on various tasks that outperform human-designed counterparts ( Baker et al. , 2017 ; Zoph & Le , 2017 ; Kandasamy et al. , 2018 ; Liu et al. , 2018 ; Luo et al. , 2018 ; Pham et al. , 2018 ; Liu et al. , 2019 ; Xu et al. , 2020 ; Chen et al. , 2021 ) . Recently , large benchmarks for NAS ( NAS-101 , NAS-201 ) ( Ying et al. , 2019 ; Dong & Yang , 2020 ) have been introduced , which provide databases of architectures and their performances on benchmark datasets . Yet , most conventional NAS methods can not benefit from the availability of such databases , due to their task-specific nature which requires repeatedly training the model from scratch for each new dataset ( See Figure 1 Left ) . Thus , searching for an architecture for a new task ( dataset ) may require a large number of computations , which may be problematic when the time and mon- ∗These authors contributed equally to this work . Conventional NAS Approach Training NAS Model NAS Model NAS Model ... Search Cost O ( N ) Target Dataset 1 Target Dataset 2 Target Dataset N Heavy Computation Cost Training Training Our Meta-Trained NAS Model Our NAS Approach ... Search Cost O ( 1 ) Target Dataset 1 Target Dataset 2 Target Dataset N Rapid Search No Retraining on Target Target Dataset Search Space Valid Zone Source Database Meta-Training Our Meta-Learning Framework Meta-Test Figure 1 : Left : Most conventional NAS approaches need to repeatedly train NAS model on each given target dataset , which results in enormous total search time on multiple datasets . Middle : We propose a novel NAS framework that generalizes to any new target dataset to generate specialized neural architecture without additional NAS model training after only meta-training on the source database . Thus , our approach cut down the search cost for training NAS model on multiple datasets from O ( N ) to O ( 1 ) . Right : For unseen target dataset , we utilize amortized meta-knowledge represented as set-dependent architecture generative representations . etary budget are limited . How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets , to better generalize over an unseen task ? In this paper , we introduce amortized meta-learning for NAS , where the goal is to learn a NAS model that generalizes well over the task distribution , rather than a single task , to utilize the accumulated meta-knowledge to new target tasks . Specifically , we propose an efficient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures , by learning to generate a neural architecture from a given dataset . The proposed MetaD2A ( Meta Dataset-to-Architecture ) framework consists of a set encoder and a graph decoder , which are used to learn a cross-modal latent space for datasets and neural architectures via amortized inference . For a new dataset , MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations , which are encoded from a new dataset , and selects the final neural architecture based on their predicted accuracies by a performance predictor , which is also trained with amortized meta-learning . The proposed meta-learning framework reduces the search cost from O ( N ) to O ( 1 ) for multiple datasets due to no training on target datasets . After one-time building cost , our model only takes just a few GPU seconds to search for neural architecture on an unseen dataset ( See Figure 1 ) . We meta-learn the proposed MetaD2A on subsets of ImageNet-1K and neural architectures from the NAS-Bench201 search space . Then we validate it to search for neural architectures on multiple unseen datasets such as MNIST , SVHN , CIFAR-10 , CIFAR-100 , Aircraft , and Oxford-IIIT Pets . In this experiment , our meta-learned model obtains a neural architecture within 33 GPU seconds on average without direct training on a target dataset and largely outperforms all baseline NAS models . Further , we compare our model with representative transferable NAS method ( Lu et al. , 2020 ) on MobileNetV3 search space . We meta-learn our model on subsets of ImageNet-1K and neural architectures from the MobileNetV3 search space . The meta-learned our model successfully generalizes , achieving extremely fast search with competitive performance on four unseen datasets such as CIFAR-10 , CIFAR-100 , Aircraft , and Oxford-IIIT Pets . To summarize , our contribution in this work is threefold : • We propose a novel NAS framework , MetaD2A , which rapidly searches for a neural architecture on a new dataset , by sampling architectures from latent embeddings of the given dataset then selecting the best one based on their predicted performances . • To this end , we propose to learn a cross-modal latent space of datasets and architectures , by performing amortized meta-learning , using a set encoder and a graph decoder on subsets of ImageNet-1K . • The meta-learned our model successfully searches for neural architectures on multiple unseen datasets and achieves state-of-the-art performance on them in NAS-Bench201 search space , especially searching for architectures within 33 GPU seconds on average . 2 RELATED WORK . Neural Architecture Search ( NAS ) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space . NAS methods can be roughly categorized into reinforcement learning-based methods ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ) , evolutionary algorithm-based methods ( Real et al. , 2019 ; Lu et al. , 2020 ) , and gradient-based methods ( Liu et al. , 2019 ; Cai et al. , 2019 ; Luo et al. , 2018 ; Dong & Yang , 2019b ; Chen et al. , 2021 ; Xu et al. , 2020 ; Fang et al. , 2020 ) . Among existing approaches , perhaps the most relevant approach to ours is NAO ( Luo et al. , 2018 ) , which maps DAGs onto a continuous latent embedding space . However , while NAO performs graph reconstruction for a single task , ours generates data-dependent Directed Acyclic Graphs ( DAGs ) across multiple tasks . Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space ( Cai et al. , 2019 ; Liu et al. , 2018 ; Pham et al. , 2018 ; Liu et al. , 2019 ; Chen et al. , 2021 ) . GDAS ( Dong & Yang , 2019b ) tackles this by optimizing sampled sub-graphs of DAG . PC-DARTS ( Xu et al. , 2020 ) reduces GPU overhead and search time by partially selecting channel connections . However , due to the task-specific nature of those methods , they should be retrained from the scratch for each new unseen task repeatedly and each will take a few GPU hours . The accuracy-predictor-based transferable NAS called NSGANetV2 ( Lu et al. , 2020 ) alleviates this issue by adapting the ImageNet-1K pre-trained network to multiple target datasets , however , this method is still expensive due to adapting procedure on each dataset . Meta-learning Meta-learning ( learning to learn ) aims to train a model to generalize over a distribution of tasks , such that it can rapidly adapt to a new task ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Nichol et al. , 2018 ; Lee et al. , 2019b ; Hou et al. , 2019 ) . Recently , LEO ( Rusu et al. , 2019 ) proposed a scalable meta-learning framework which learns the latent generative representations of model parameters for a given data in a low-dimensional space for few-shot classification . Similarly to LEO ( Rusu et al. , 2019 ) , our method learns a low-dimensional latent embedding space , but we learn a cross-modal space for both datasets and models for task-dependent model generation . Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-based meta-learning ( Elsken et al. , 2020 ; Lian et al. , 2019 ; Shaw et al. , 2019 ) have shown promising results on adapting to different tasks . However , they are only applicable on small scale tasks such as few-shot classification tasks ( Elsken et al. , 2020 ; Lian et al. , 2019 ) and require high-computation time , due to the multiple unrolling gradient steps for one meta-update of each task . While some attempt to bypass the bottleneck with a first-order approximation ( Lian et al. , 2019 ; Shaw et al. , 2019 ) or parallel computations with GPUs ( Shaw et al. , 2019 ) , but their scalability is intrinsically limited due to gradient updates over a large number of tasks . To tackle such a scalability issue , we perform amortized inference over the multiple tasks by encoding a dataset into the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update . 3 METHOD . Our goal is to output a high-performing neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and their corresponding neural architectures . To this end , we propose Meta Dataset-to-Architecture ( MetaD2A ) framework which learns the cross-modal latent space of datasets and their neural architectures . Further , we introduce a meta-performance predictor , which predicts accuracies of given architectures without training the predictor on an unseen target dataset . Overview of the proposed approach is illustrated in Figure 1 . 3.1 META-TRAINING NAS MODEL . To formally define the problem , let us assume that we have a source database of Nτ number of tasks , where each task τ = { D , G , s } consists of a dataset D , a neural architecture represented as a Directed Acyclic Graph ( DAG ) G and an accuracy s obtained from the neural architecture G trained onD . In the meta-training phase , the both dataset-to-architecture generator and meta-predictor learn to generalize over task distribution p ( τ ) using the source database . We describe how to empirically construct source database in Section 4.1.1 . 3.1.1 LEARNING TO GENERATE GRAPHS FROM DATASETS . We propose a dataset-to-architecture generator which takes a dataset and then generates high-quality architecture candidates for the set . We want the generator to generate even novel architectures , which are not contained in the source database , at meta-test . Thus , the generator learns the continuous cross-modal latent space Z of datasets and neural architectures from the source database . For each task τ , the generator encodes datasetD as a vector z through the set encoder qφ ( z|D ) parameterized by φ and then decodes a new graph G̃ from z which are sampled from the prior p ( z ) by using the graph decoder pθ ( G|z ) parameterized by θ . Then , our goal is that G̃ generated from D to be the true G which is pair of D. We meta-learn the generator using set-amortized inference , by maximizing the approximated evidence lower bound ( ELBO ) as follows : max φ , θ ∑ τ∼p ( τ ) Lτφ , θ ( D , G ) ( 1 ) where Lτφ , θ ( D , G ) = Ez∼qφ ( z|D ) [ log pθ ( G|z ) ] − λ · LτKL [ qφ ( z|D ) ||p ( z ) ] ( 2 ) Each dimension of the prior p ( z ) factorizes into N ( 0 , 1 ) . LτKL is the KL divergence between two multivariate Gaussian distributions which has a simple closed form ( Kingma & Welling , 2014 ) and λ is the scalar weighting value . Using the reparameterization trick on z , we optimize the above objective by stochastic gradient variational Bayes ( Kingma & Welling , 2014 ) . We use a set encoder described in Section 3.1.3 and we adopt a Graph Neural Network ( GNN ) -based decoder for directed acyclic graph ( DAG ) s ( Zhang et al. , 2019 ) , which allows message passing to happen only along the topological order of the DAGs . For detailed descriptions for the generator , see Section A of Suppl .

Summary: The authors address neural architecture search (NAS) scenarios. In particular, a framework, MetaD2A, is proposed, which yields a neural architecture for a new dataset. In a nutshell, the framework learns a "dataset-to-neural-network-architecture" transformation using a database of datasets and architectures. Each dataset is encoded via a "set encode" and the architecutres are obtained via a "graph decoder". The experiments demonstrate the usefullness of the approach and its improvements over conventual NAS approaches. The approach could be described