DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search

1 INTRODUCTION . The advent of deep neural networks ushered in qualitative leaps in several challenging machine learning tasks including image recognition ( Krizhevsky et al. , 2017 ) and language modeling ( Sutskever et al. , 2014 ) . However , on top of being tedious and time consuming , deep neural network design requires deep technical knowledge . Automating this design process has reached important milestones in the last few years as advances in neural architecture search ( Zoph & Le , 2016 ; Zoph et al. , 2017 ) bridged the performance gap between manually designed architectures and those found via automatic search . Traditional approaches usually rely on Reinforcement Learning ( RL ) ( Baker et al. , 2016 ; Zoph et al. , 2017 ; Zoph & Le , 2016 ; Zhong et al. , 2017 ) , Genetic Algorithms ( GA ) ( Precup & Teh , 2017 ; Xie & Yuille , 2017 ; Liu et al. , 2017b ; Real et al. , 2018 ) or Sequential Model Based Optimization ( SMBO ) ( Liu et al. , 2017a ; Brock et al. , 2017 ) , to search for the best performing architectures . This search process is quite resource heavy which is why current work on architecture search shifted focus from merely outperforming manually designed architectures to accelerating the search process ( Liu et al. , 2017a ; Pham et al. , 2018 ; Liu et al. , 2018 ) . Nonetheless , very few works ( Dong et al. , 2018 ; Smithson et al. , 2016 ; Ye-Hoon et al. , 2017 ) attempted to take additional constraints such as memory consumption or inference time into consideration when evaluating candidate architectures , although it is of crucial importance when targeting mobile devices . In this article , we introduce Dvolver , a principled multi-objective neural architecture search approach using Pareto optimality to navigate the search space . We instantiate this approach with a standard evolutionary algorithm ( Deb et al. , 2002 ) on the NASNet search space ( Zoph et al. , 2017 ) with validation accuracy and the total number of floating point operations ( FLOPS ) as objectives . Our experiments show that ( i ) we do not sacrifice accuracy by looking for faster architectures ( ii ) we are able to explore the search space without loss in efficiency ( iii ) we are able to find architec- 1Using 20 NVIDIA V100 GPUs 2Code and checkpoints are publicly available at https : //goo.gl/1QZX6a . tures that outperform NASNet-A mobile and certain configurations of MobileNetV2 architectures ( Sandler et al. , 2018 ) when accounting for both accuracy and floating point operations count . 2 RELATED WORK . Our work is founded on previous successful contributions focusing on single objective neural network architecture search that we extend for multiple criteria . Recently , independent papers came out with a similar approach but their search space is orders of magnitude smaller than the best studies so far . Mono-objective Search Zoph et al . ( 2017 ) introduce a modular search space , based on reusable cells , which not only reduces the computational cost of the search process but also allows the search to be carried out on a small database and transfered to a larger one . They use RL to train a Recurrent Neural Network ( RNN ) that generates the structure of the cells . Real et al . ( 2018 ) showed that artificially-evolved architectures , on the same search space , are at least as good as their RL counterpart . On a similar note , Liu et al . ( 2017a ) adopted SMBO coupled with a progressive exploration of a variation of said space , greatly reducing the search cost while maintaining competitive results . Their approaches achieved state of the art performances on ImageNet ( Deng et al. , 2009 ) , with modules learned on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) but they only optimized for best validation accuracy . The best module is then combined in a scaled down network and compared with the best mobile architecture at the time . Even though , the results are impressive , this is sub-optimal and explicitly optimizing for validation accuracy and speed can lead to even better architecture . Multi-objective Search Smithson et al . ( 2016 ) used Pareto optimality as a selection criteria in an SMBO-based hyperparameter search , and showed that the obtained Pareto front is close to the one obtained via exhaustive search , even with two orders of magnitude fewer architecture evaluations . However , their work focused on hyperparameter search and did not explore a more general search space . Elsken et al . ( 2018 ) used network morphism to optimize network architecture search on multiple criteria namely test error and parameters count . While an interesting approach , this paper lacks clear comparison with single objective methods and only evaluates architecture transferability on ImageNet64x64 ( Chrabaszcz et al. , 2017 ) . In Ye-Hoon et al . ( 2017 ) , a multi-objective genetic algorithm is implemented to search for fast and accurate architectures , initializing the search process from a given baseline network . They were able to find architectures that outperformed common baselines 3 both in speed and accuracy on MNIST and CIFAR10 . Their method , while in principle similar to ours , explored the vicinity of a defined network and was not extended to a complex search space where the search process is initialized arbitrarily . Furthermore , it was not validated for learning architectures that transfer to large scale applications . Dong et al . ( 2018 ) work is perhaps the closest to ours in intent and scale , as it performs multiple objective architecture search using Pareto optimality and uses a proxy dataset to avoid an expensive search on ImageNet . It even goes one step further to incorporate device-dependent metrics . However , their search space is orders of magnitude smaller than ours in addition to being heavily biased towards mobile architectures . 3 PROPOSED METHOD . For a given search space , our goal is to find all the neural network architectures that offer the best trade-offs between multiple independent criteria . We formulate the search process as a multiobjective optimization ( MOO ) problem ( Section 3.1 ) where each objective to optimize is an independent scalar value . We refer to Figure 1 for a general overview of the search process . The MOO controller selects child networks with different architectures . The child networks are evaluated on multiple criteria by the Evaluation Engine . The resulting objective values are then used 3namely Lenet ( Lecun et al. , 1998 ) and an all convolutional net ( Springenberg et al. , 2014 ) by the controller as feedback so it can generates better architectures over time . At each iteration , Dvolver keeps the hall of fame of the best architectures found so far . 3.1 MULTI-OBJECTIVE OPTIMIZATION AND PARETO DOMINANCE . We define the search space X as the set of all feasible solutions and the objective space as the set of all possible values of the objective function f : X → Rk . Dvolver searches for solutions by maximizing the objective function f over X with the total order relation defined next . For a nontrivial multi-objective optimization problem , there is no solution that simultaneously optimizes all the objectives . In that case , the objective functions are said to be conflicting , and there exists a ( possibly infinite ) number of compromise solutions called Paretooptimal solutions . The set of Pareto optimal outcomes is often called the Pareto front . A simple approach to cope with multi-objective functions is to use scalarization which squashes all the objective functions into a single function falling back to single-objective optimization . While very simple in practice , this method suffers from many shortcomings such as ( i ) the difficulty to combine multiple unrelated criteria into a single value and ( ii ) inefficient sampling of the full Pareto front . A much better solution is to use Pareto dominance partially ordered relation defined as follow : A solution v Pareto-dominates a solution u , if all objective values for v are greater than the objective values for u but a least one is strictly superior . A feasible solution is Pareto optimal , if it is not dominated by another solution . Without additional subjective preference information , all Pareto optimal solutions are considered equally good . For the search process to be effective , it needs a totally ordered relation . Deb et al . ( 2002 ) defines the crowding distance value of a particular solution di of point i as a measure of the objective space around i which is not occupied by any other solution in the population . Here , we simply calculate this quantity di by estimating the perimeter of the cuboid ( Figure 8 ) formed by using the nearest neighbors in the objective space as the vertices ( See appendix A.2 for more details ) . In addition to the Pareto dominance criteria , the crowding distance defines the totally ordered relation used by Dvolver to perform its search process . 3.2 SEARCH ALGORITHM . Dvolver evolves a population ofN individual architectures . At each iteration , the population evolves closer and closer toward the Pareto front , effectively sampling the full Pareto front in a single search process . Our method starts by generating a random population and for each generation , it generates a child population , evaluates each child by training the networks for a few epochs on CIFAR-10 and selects the N dominant architectures from the 2×N individuals ( parents and children ) . The details of the algorithm is presented in appendix A.1 . Algorithm convergence is monitored with the hypervolume indicator defined as the volume under the Pareto front ( see A.3 ) . In Algorithm 1 , we present the Dvolver selection process inspired by NSGA-II ( Deb et al. , 2002 ) . At the end of an iteration , there are N parents and N children . We should select the best N individuals to form the parents ’ population for the next iteration . We compute the Pareto fronts for the 2 ×N individuals . The best fronts are considered first . If there are more than N individuals in the Algorithm 1 : Dvolver selection algorithm Inputs • N individuals from parents ’ population • N individuals from offsprings ’ population Output • N selected individuals Indssel 1 : Indssel ← ∅ ; 2 : fronts← ComputeParetoFronts ( parents ∪ offsprings ) ; 3 : i← 0 ; . Starts with Pareto front with rank 0 4 : rem← N ; . rem is the number of remaining individuals to be selected 5 : while rem > 0 do Indsi ← SortWithDominance ( frontsi ) ; if Len ( Indsi ) < rem then Indssel ← Indssel ∪ Indsi ; rem← rem− Len ( Indsi ) ; i← i+ 1 ; . Move to the next Pareto front else Indssel ← Indssel ∪ First ( rem , Indsi ) ; . Highest crowding distance individuals first rem← 0 ; best front , the first N ( with the help of the crowding distance ) are selected and the selection ends . Otherwise , all the individuals from the front are selected and we carry on with the next best Pareto front until N individuals are selected . 3.3 SEARCH SPACE . Our work is based on the work of Zoph et al . ( 2017 ) , in which network architecture is composed of two levels : a micro level architecture defined by parametrized cells composed of B blocks and a macro architecture where those cells are stacked a predefined number of times , in order to create the final network . Figure 2 describes the macro architecture used for CIFAR-10 and ImageNet architectures . There are two types of cells and Dvolver searches for both at the same time : ( i ) Normal Cell that returns a feature map of the same size as its input and ( ii ) Reduction Cell that returns a feature map of half the size of its input in the spatial dimensions . A cell is composed of B blocks arranged in a Directed Acyclic Graph ( DAG ) . Each block is a mapping of two input tensors to one output tensor and it can be represented by a 4-tuple , ( I1 , O1 , I2 , O2 ) where Ik specifies the input for the operation Ok . The result of the two operations O1 and O2 are summed together to form the output of the block . A block can take its input from previous blocks , the previous cell ck−1 or the previous previous cell ck−2 . Unused blocks are concatenated in the depth dimension to form the final output of the cell . Moreover , we noticed that the original NASNet search space can significantly benefit from extra connections from any given block , the previous cell and the previous previous cell to the final concatenation layer . In fact , these extra connections are already present but not described in the best models from Zoph et al . ( 2017 ) , Real et al . ( 2018 ) and Liu et al . ( 2017a ) . Instead of adding manually these connections after the search process like in the previous works , Dvolver optimizes for these extra connections in addition to the 4 parameters for each block . In practice , our search space is the same as the one used by Zoph et al . ( 2017 ) but no manual intervention is needed . For B = 5 , the total number of parameters for each type of cell is 20 parameters for the blocks and one list of indexes of extra connections . Similar to the approach of Liu et al . ( 2017a ) , we simplify the search space and consider only the following operations : • identity • 3x3 average pooling • 3x3 max pooling • 3x3 depthwise-separable conv • 5x5 depthwise-separable conv • 7x7 depthwise-separable conv The total number of combinations for the defined search space is about 1020 taking into account symmetries and possible redundant connections . Note that our search space is larger than the one defined in Liu et al . ( 2017a ) but we show in Section 4 that Dvolver is capable of finding competitive architectures faster .

Summary: The authors propose a multi-objective neural architecture search based on an evolutionary algorithm. The contradicting objective functions are optimized by ranking the candidates by Pareto-dominance, replace the bottom 50% with new candidates generated by the top 50% candidates through random mutations. The multi-objective function considers classification accuracy and an approximation of the inference speed. The method is compared to MobileNet and Mobile NASNet on ImageNet indicating an improvement with respect to search time.