BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search

1 INTRODUCTION . Since the deep learning revolution in 2012 , neural networks have been growing increasingly more specialized and more complex ( Krizhevsky et al. , 2012 ; Huang et al. , 2017 ; Szegedy et al. , 2017 ) . Developing new state-of-the-art architectures often takes a vast amount of engineering and domain knowledge . A new area of research , neural architecture search ( NAS ) , seeks to automate this process . Since the popular NASNet paper ( Zoph & Le , 2017 ) , there has been a flurry of research on neural architecture search ( Liu et al. , 2018a ; Pham et al. , 2018 ; Liu et al. , 2018b ; Kandasamy et al. , 2018 ; Elsken et al. , 2018 ; Jin et al. , 2018 ; Sciuto et al. , 2019 ; Cortes et al. , 2017 ; Baker et al. , 2016 ; Liu et al. , 2017 ) . Many methods have been proposed for NAS , including random search , evolutionary search , reinforcement learning , Bayesian optimization , and gradient descent . Each technique has its advantages and disadvantages . For example , reinforcement learning with a controller neural network can learn very complex policies . However , as some past work has argued , NAS is an optimization problem – find the architecture with the lowest validation error – so there is no explicit need to maintain a state and transition function , and this approach may require evaluating a larger number of architectues to achieve the same result as other methods ( Kandasamy et al. , 2018 ; Elsken et al. , 2018 ; Jiang et al. , 2017 ) . Alternatively , Bayesian optimization with a Gaussian process model is a leading method for zeroth order optimization of expensive-to-evaluate functions , which is precisely the setting of deep learning model selection . Although Bayesian optimization has seen great success in hyperparameter optimization ( Golovin et al. , 2017 ; Li et al. , 2016 ) for deep models , a host of difficulties arise when using Bayesian optimization for neural architecture search . For example , while there is a substantial body of work on Bayesian optimization over Euclidean spaces , using Bayesian optimization for NAS has so far required specifying a distance function between neural architectures , in order to define a surrogate model . This is often a cumbersome task involving tuning hyperparameters of the distance function ( Kandasamy et al. , 2018 ; Jin et al. , 2018 ) . Furthermore , Bayesian optimization most commonly uses a Gaussian process ( GP ) model , and it can be quite challenging to achieve highly accurate prediction performance with GPs , given the potentially high dimensional input architectures ( Elsken et al. , 2018 ) . In this work , we run Bayesian optimization with a neural network model . That is , in every iteration of Bayesian optimization , we train a meta neural network to predict the accuracy of unseen neural architectures in the search space . This procedure avoids the aforementioned problems with Bayesian optimization NAS : the model is powerful enough to predict neural network accuracies , and there is no need to construct a distance function between neural networks by hand . Furthermore , using a neural network model scales far better than a GP model , as it avoids the computationallyintensive matrix inversion operation . Training a meta neural network to predict the accuracy of neural architectures is a challenging task . The majority of popular NAS algorithms are deployed over a directed acyclic graph ( DAG ) search space – the set of possible architectures is comprised of the set of all DAGs of a certain size , together with all possible combinations of operations on each node ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018b ; Ying et al. , 2019 ) . This poses a roadblock when using a meta neural network to predict the accuracies of neural architectures , since graph structures are difficult for neural networks to understand directly from the adjacency matrix ( Zhou et al. , 2018 ) . We use a path-based encoding scheme to encode a neural architecture , which drastically improves the predictive accuracy of our meta neural network . Each binary feature represents a unique path through the DAG from the input layer to the output layer . This method improves the accuracy of our meta neural network by a factor of 3 . With just 200 training points , our meta neural network is able to predict the accuracy of unseen neural networks to within one percent on average on the popular NASBench dataset ( Ying et al. , 2019 ) . This may be of interest beyond Bayesian neural architecture search . In our Bayesian optimization NAS algorithm , we train an ensemble of neural networks to predict the mean and variance of candidate neural architectures , from which we compute the upper confidence bound ( UCB ) acquisition function ( Srinivas et al. , 2009 ) . Finally , we use a mutation algorithm to optimize the acquisition function . We compare our NAS algorithm against a host of black-box optimization NAS algorithms including regularized evolution , reinforcement learning , Bayesian optimization with a GP model , AlphaX ( Wang et al. , 2018 ) , ASHA ( Li & Talwalkar , 2019 ) , DARTS ( Liu et al. , 2018b ) , and TPE ( Bergstra et al. , 2011 ) . On the NASBench dataset , our method sees significant improvements compared to all other algorithms we tried across 200 trials . It takes the next-best algorithm 3.8 times the number of queries to achieve the same mean test accuracy as our method , and it takes random search more than 100 times the number of queries . On the search space from DARTS , when given a budget of 100 neural architecture queries for 50 epochs each , our algorithm achieves a best of 2.57 % and average of 2.64 % test error , which beats all NAS algorithms with which we could fairly compare ( e.g. , same search space used , and same hyperparameters for the final training ) . Our NAS algorithm has several moving parts , including the meta neural network , the path-based feature encoding of neural architectures , the acquisition function , and the acquisition optimization strategy . We run a thorough ablation study by removing each piece of the algorithm separately . We show all components are necessary to achieve the best performance . Finally , to adhere to recent calls for reproducible NAS research ( Ying et al. , 2019 ; Li & Talwalkar , 2019 ; Lindauer & Hutter , 2019 ) , we address all points in the NAS research checklist ( Lindauer & Hutter , 2019 ) and will make our anonymized code is available at https : //www.github.com/iclr474/bananas . Our contributions . We summarize our main contributions below . • We propose a novel path-based encoding scheme for neural architectures . This is a method for featurizing neural architectures by enumerating paths along the DAG from the input node to the output node . When predicting the validation accuracy of neural networks , using this featurization drastically improves the performance compared to standard encodings such as the adjacency matrix encoding , making it feasible to use a meta neural network as a model in Bayesian optimization . • We design a Bayesian optimization algorithm using a meta neural network model to perform NAS over cell-based search spaces . We use an ensemble of neural networks to output a mean and variance for each candidate architecture and then use the UCB acquisition function to guide the search . We perform an extensive study of our algorithm and show it performs on par or superior to several state-of-the-art NAS algorithms on two different search spaces . 2 RELATED WORK . Neural architecture search have been studied since at least the 1990s ( Floreano et al. , 2008 ; Kitano , 1990 ; Stanley & Miikkulainen , 2002 ) , but the field was revitalized when NASNet gained significant attention ( Zoph & Le , 2017 ) . There has been a flurry of recent work , where the most popular techniques include evolutionary algorithms ( Shah et al. , 2018 ) , reinforcement learning ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018a ; Tan & Le , 2019 ) , Bayesian optimization ( Kandasamy et al. , 2018 ; Jin et al. , 2018 ; Zhou et al. , 2019 ) , and gradient descent ( Liu et al. , 2018b ) . Recent papers have called for fair and reproducible experiments in the future ( Li & Talwalkar , 2019 ; Sciuto et al. , 2019 ; Ying et al. , 2019 ) . In this vein , the NASBench dataset was created , which contains over 400k neural architectures with precomputed training , validation , and test accuracy ( Ying et al. , 2019 ) . A recent algorithm , AlphaX , uses a meta neural network to perform NAS ( Wang et al. , 2018 ) , where the meta neural network is trained to make small changes to the current architecture , given adjacency matrix featurizations of neural architectures . See the recent survey on NAS ( Elsken et al. , 2018 ) for more information on NAS research . There are many additional recent NAS papers ( Maziarz et al. , 2018 ; Borsos et al. , 2019 ; Wang et al. , 2019 ) . Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive ( Rasmussen , 2003 ; Frazier , 2018 ) , and it has seen great success in hyperparameter optimization for deep learning ( Rasmussen , 2003 ; Golovin et al. , 2017 ; Li et al. , 2016 ) . Most work has focused on Euclidean or categorical input domains using a GP model ( Rasmussen , 2003 ; Golovin et al. , 2017 ; Frazier , 2018 ; Snoek et al. , 2012 ) . There has also been some previous work on using neural network models in Bayesian optimization ( Snoek et al. , 2015 ; Springenberg et al. , 2016 ; Perrone et al. , 2017 ) , applied to the task of hyperparameter optimization . There are several approaches for predicting the validation accuracy of neural networks using a layer-wise encoding of the neural architectures ( Deng et al. , 2017 ; Istrate et al. , 2019 ) , or predicting the learning curve of different neural architectures ( Klein et al. , 2017 ; Domhan et al. , 2015 ; Baker et al. , 2017 ) . Ensembles of neural networks is a popular approach for uncertainty estimates ( Beluch et al. , 2018 ; Lakshminarayanan et al. , 2017 ; Choi et al. , 2016 ) . For more discussion on related work , see Appendix A . 3 PRELIMINARIES . We give background on three key ingredients of NAS algorithms , as well as on Bayesian optimization . Search space . Before deploying a NAS algorithm , we must define the space of neural networks that the algorithm can search through . Perhaps the most common type of search space for NAS is a cell-based search space ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018b ; Li & Talwalkar , 2019 ; Sciuto et al. , 2019 ; Ying et al. , 2019 ) . A cell consists of a relatively small section of a neural network , usually 6-12 nodes forming a directed acyclic graph ( DAG ) . A neural architecture is then built by repeatedly stacking one or two different cells on top of each other sequentially , possibly separated by specialized layers . The layout of cells and specialized layers is called a hyper-architecture , and this is fixed , while the NAS algorithm searches for the best cells . The search space over cells consists of all possible DAGs of a certain size , where each node can be one of several operations such as 1× 1 convolution , 3× 3 convolution , or 3× 3 max pooling . It is also common to set a restriction on the number of total edges or the in-degree of each node ( Ying et al. , 2019 ; Liu et al. , 2018b ) . In this work , we focus on NAS over convolutional cell-based search spaces . Search strategy . The search strategy is the optimization method that the algorithm uses to find the optimal or near-optimal neural architecture from the search space . There are many varied search strategies , such as Bayesian optimization , evolutionary search , reinforcement learning , and gradient descent . In Section 4 , we introduce a novel search strategy based on Bayesian optimization with a neural network model using a path-based encoding . Evaluation method . Many types of NAS algorithms consist of an iterative framework in which the algorithm chooses a neural network to train , computes its validation error , and uses this result to guide the choice of neural network in the next iteration . The simplest instantiation of this approach is to train each neural network in a fixed way , i.e. , the algorithm has black-box access to a function that trains a neural network for x epochs and then returns the validation error ( Pham et al. , 2018 ; Liu et al. , 2018a ; Shah et al. , 2018 ) . Algorithms with black-box evaluation methods can be compared by returning the architecture with the lowest validation error after a certain number of queries to the black-box function . There are also multi-fidelity methods , for example , when a NAS algorithm chooses the number of training epochs in addition to the architecture . In this work , we show our algorithm far outperforms several other NAS algorithms that use black box optimization , and we discuss multi-fidelity approaches in Section 6 . Bayesian optimization . In applications of Bayesian optimization for deep learning , the typical goal is to find a neural architecture and/or set of hyperparameters that lead to an optimal validation error . Formally , Bayesian optimization seeks to compute a∗ = argmina∈A f ( a ) , where A is the search space , and f ( a ) denotes the validation error of architecture a after training on a fixed dataset for a fixed number of epochs . In the standard Bayesian optimization setting , over a sequence of iterations , the results from all previous iterations are used to model the topology of f ( A ) using the posterior distribution of the model ( often a Gaussian process ) . The next architecture is then chosen by optimizing an acquisition function such as expected improvement ( EI ) ( Močkus , 1975 ) or upper confidence bound ( UCB ) ( Srinivas et al. , 2009 ) . These functions balance exploration with exploitation during the iterative search . We discuss UCB in the next section , and we discuss other acquisition functions in Appendix B .

Summary: The paper considers the neural architecture search using Bayesian optimisation. The paper first propose a path-based encoding scheme to featurise the neural architectures. A path encoding of a cell is created by enumerating all possible paths from the input node to the output node. Then, they propose to train 5 neural networks and ensemble these networks to get the prediction (including the predictive value and uncertainty). The paper optimises the acquisition function via a mutation procedure, where we randomly mutate the best architectures that we have trained so far, many times, and then select the architecture from this set which minimizes the acquisition function.