Multi-objective Neural Architecture Search via Predictive Network Performance Optimization

1 INTRODUCTION . Recently Neural Architecture Search ( NAS ) has aroused a surge of interest by its potentials of freeing the researchers from tedious and time-consuming architecture tuning for each new task and dataset . Specifically , NAS has already shown some competitive results comparing with hand-crafted architectures in computer vision : classification ( Real et al. , 2019b ) , detection , segmentation ( Ghiasi et al. , 2019 ; Chen et al. , 2019 ; Liu et al. , 2019a ) and super-resolution ( Chu et al. , 2019 ) . Meanwhile , NAS has also achieved remarkable results in natural language processing tasks ( Luong et al. , 2018 ; So et al. , 2019 ) . A variety of search strategies have been proposed , which may be categorized into two groups : oneshot NAS algorithms ( Liu et al. , 2019b ; Pham et al. , 2018 ; Luo et al. , 2018 ) , and sample-based algorithms ( Zoph & Le , 2017 ; Liu et al. , 2018a ; Real et al. , 2019b ) . One-shot NAS algorithms embed the architecture searching process into the training stage by using weight sharing , continuous relaxation or network morphisms . However , those methods can not guarantee the optimal performance of the final model due to those approximation tricks and is usually sensitive to the initial seeds ( Sciuto et al. , 2019 ) . On the other hand , sample-based algorithms are relatively slower but reliable . They explore and exploit the search space using some general search algorithms by providing potential candidates with higher accuracy . However , it requires fully training of huge amounts of candidate models . Typically , the focus of most existing NAS methods has been on the accuracy of the final searched model alone , ignoring the cost spent in the search phase . Thus , the comparison between existing search algorithms for NAS is very difficult . ( Wang et al. , 2019b ) gives us an example of evaluating the NAS algorithms from this view . They compare the number of training architectures sampled until finding the global optimal architecture with the top accuracy in the NAS datasets . Besides accuracy , in real applications , there are many other objectives we should concern , such as speed/accuracy trade-off . Hence , in this paper , we aim at designing an efficient multi-objective NAS algorithm to adaptively explore the search space and capture the structural information of architectures related to the performance . The common issue faced by this problem is that optimizing objective functions is computationally expensive and the search space always contains billions of architectures . To tackle this problem , we present BOGCN-NAS , a NAS algorithm that utilizes Bayesian Optimization ( BO ) together with Graph Convolutional Network ( GCN ) . BO is an efficient algorithm for finding the global optimum of costly black-box function ( Mockus et al. , 1978 ) . In our method , we replace the popular Gaussian Processes model with a proposed GCN model as the surrogate function for BO ( Jones , 2001 ) . We have found that GCN can generalize fairly well with just a few architecture-accuracy pairs as its training set . As BO balances exploration and exploitation during searching and GCN extracts embeddings that can well represent model architectures , BOGCN-NAS is able to obtain the optimal model architecture with only a few samples from the search space . Thus , our method is more resource-efficient than the previous ones . Graph neural network has been proposed in previous work for predicting the parameters of the architecture using a graph hypernetwork ( Zhang et al. , 2019 ) . However , it ’ s still a one-shot NAS method and thus can not ensure the performance of the final found model . In contrast , we use graph embedding to predict the performance directly and can guarantee performance as well . The proposed BOGCN-NAS outperforms current state-of-the-art searching methods , including Evolution ( Real et al. , 2019b ) , MCTS ( Wang et al. , 2019b ) , LaNAS ( Wang et al. , 2019a ) . We observe consistent gains on multiple search space for CV and NLP tasks , i.e. , NASBench-101 ( denoted NASBench ) ( Ying et al. , 2019 ) and LSTM-12K ( toy dataset ) . In particular , our method BOGCN-NAS is 128.4×more efficient than Random Search and 7.8×more efficient than previous SOTA LaNAS on NASBench ( Wang et al. , 2019a ) . We apply our method to multi-objective NAS further , considering adding more search objectives including accuracy and number of parameters . Our method can find more superior Pareto front on NASBench . Our algorithm is applied on open domain search with NASNet search space and ResNet Style search space , which finds competitive models in both scenarios . The results of experiment demonstrate our proposed algorithm can find a more competitive Pareto front compared with other sample-based methods . 2 RELATED WORK . 2.1 BAYESIAN OPTIMIZATION . Bayesian Optimization aims to find the global optimal over a compact subset X ( here we consider maximization problem ) : x∗ = arg max x∈X f ( x ) . ( 1 ) Bayesian Optimization considers prior belief about objective function and updates posterior probability with online sampling . Gaussian Processes ( GPs ) is widely used as a surrogate model to approximate the objective function ( Jones , 2001 ) . And Expected Improvement acquisition function is often adopted ( Mockus et al. , 1978 ) . For the hyperparameters of the surrogate model Θ , we define γ ( x ) = µ ( x ; D , Θ ) − f ( xbest ) σ ( x ; D , Θ ) , ( 2 ) where µ ( x ; D , Θ ) is the predictive mean , σ2 ( x ; D , Θ ) is the predictive variance and f ( xbest ) is the maximal value observed . The Expected Improvement ( EI ) criterion is defined as follows . aEI ( x ; D , Θ ) = σ ( x ; D , Θ ) [ γ ( x ) Φ ( γ ( x ) ; 0 , 1 ) +N ( γ ( x ) ; 0 , 1 ) ] , ( 3 ) where N ( · ; 0 , 1 ) is the probability density function of a standard normal and Φ ( · ; 0 , 1 ) is its cumulative distribution . 2.2 MULTI-OBJECTIVE OPTIMIZATION . Without loss of generality about max or min , given a search space X and m ≥ 1 objectives f1 : X → R , . . . , fm : X → R , variable X1 ∈ X dominates variable X2 ∈ X ( denoted X1 X2 ) if ( i ) fi ( X1 ) ≥ fi ( X2 ) , ∀i ∈ { 1 , . . . , m } ; and ( ii ) fj ( X1 ) > fj ( X2 ) for at least one j ∈ { 1 , . . . , m } . X∗ is Pareto optimal if there is no X ∈ X that domaines X∗ . The set of all Pareto optimal architectures consitutes the Pareto front Pf . A multi-objective optimization problem ( MOP ) aims at finding such input X ∈ X that X can not be dominated by any variable in X ( Marler & Arora , 2004 ) . 2.3 GRAPH CONVOLUTIONAL NETWORK . Let the graph be G = ( V , E ) , where V is a set of N nodes , and E is the set of edges . Let its adjacency matrix be A and feature matrix be X . The graph convolutional network ( GCN ) is a learning model for graph-structure data ( Kipf & Welling , 2016 ) . For a L-layer GCN , the layer-wise propagation rule is given by : H ( l+1 ) = f ( H ( l ) , A ) = ReLU ( D̃ 1 2 ÃD̃− 1 2H ( l ) W ( l ) ) , ( 4 ) where Ã = A + I , I is the identity matrix , D̃ is a diagonal matrix with D̃ii = ∑N j=1Aij , H ( l ) and W ( l ) are the feature map and weight at the l-th layer respectively , and ReLU ( · ) is the ReLU activation function . H ( 0 ) is the original feature matrix X , and H ( L ) is the graph embedding matrix . 3 BOGCN-NAS . To search for the optimal architecture more efficiently , we proposed BOGCN-NAS by using predictive network performance Optimization with the GCN ( Section 3.2 ) while utilizing the Bayesian Optimization . Figure 1 shows the overview of the proposed algorithm . 3.1 MULTI-OBJECTIVE NAS . We formulate NAS problem as a multi-objective optimization problem over the architecture search space A where objective functions can be accuracy , latency , number of parameters , etc . We aim to find architectures on the Pareto front of A . Specifically , when m = 1 , it reduces to single-objective ( usually accuracy ) NAS and the corresponding Pareto front reduces to one optimal architecture . 3.2 GCN PREDICTOR . GCN predictor predicts the performance ( like accuracy ) of an architecture . Compared with MLP and LSTM predictors proposed before ( Wang et al. , 2019b ) , GCN can preserve the context of graph data better . Another important characteristic of GCN is its ability to handle variable number of nodes , while an MLP can not take a larger architecture as the input . Even though the LSTM can handle variable-length sequences , its performance is not competitive because of the flat string encoding . A neural network can be viewed as a directed attributed graph , in which each node represents an operation ( such as convolution operation ) and each edge represents a data flow . As a concrete illustration , we use the architectures in the NASBench dataset ( Ying et al. , 2019 ) as an example . The idea can be easily extended to other architectures . In NASBench , each architecture is constituted by stacking multiple repeated cells . Thus , we will focus on searching the cell architecture . An example cell in NASBench is illustrated on the left side of Figure 1 , where “ input ” represents the input of the cell , “ output ” represents the output of the cell , “ 1 × 1 Conv , 3 × 3 Conv , Max Pooling ” are three different operations ( 5 operations totally ) . We propose to encode the cell into an adjacency matrix A ( asymmetric ) and a feature matrix X , as the input of our GCN predictor . Note that the vanilla GCN only extracts the node embeddings , while we want to obtain graph embedding . Following ( Scarselli et al. , 2008 ) , we add a global node to the original graph of the cell and let every node point at the global node . The adjacency matrix can be obtained directly from the graph structure . For the feature matrix , we use the one-hot coding scheme for each operation . Besides the original 5 different operations defined in NASBench , we add another operation ( global node ) into coding scheme . We feed A and X to a multi-layer GCN model to obtain the embedding of every node H ( L ) by ( Eq . 4 ) . For high-level prediction , we leave original nodes out and take the embedding of global node solely because it already has the overall context of the architecture . Followed by one fully-connected layer with sigmoid activation function , we can get the predicted accuracy . In training phase , we use MSE loss for regression optimization .

Summary: This paper proposed BOGCN-NAS that encodes current architecture with Graph convolutional network (GCN) and uses the feature extracted from GCN as the input to perform a Bayesian regression (predicting bias and variance, See Eqn. 5-6). They use Bayesian Optimization to pick the most promising next model with Expected Improvement, train it and take its resulting accuracy/latency as an additional training sample, and repeat.