DVOLVER: Efficient Pareto-Optimal Neural Network Architecture Search

1 INTRODUCTION . The advent of deep neural networks ushered in qualitative leaps in several challenging machine learning tasks including image recognition ( Krizhevsky et al. , 2017 ) and language modeling ( Sutskever et al. , 2014 ) . However , on top of being tedious and time consuming , deep neural network design requires deep technical knowledge . Automating this design process has reached important milestones in the last few years as advances in neural architecture search ( Zoph & Le , 2016 ; Zoph et al. , 2017 ) bridged the performance gap between manually designed architectures and those found via automatic search . Traditional approaches usually rely on Reinforcement Learning ( RL ) ( Baker et al. , 2016 ; Zoph et al. , 2017 ; Zoph & Le , 2016 ; Zhong et al. , 2017 ) , Genetic Algorithms ( GA ) ( Precup & Teh , 2017 ; Xie & Yuille , 2017 ; Liu et al. , 2017b ; Real et al. , 2018 ) or Sequential Model Based Optimization ( SMBO ) ( Liu et al. , 2017a ; Brock et al. , 2017 ) , to search for the best performing architectures . This search process is quite resource heavy which is why current work on architecture search shifted focus from merely outperforming manually designed architectures to accelerating the search process ( Liu et al. , 2017a ; Pham et al. , 2018 ; Liu et al. , 2018 ) . Nonetheless , very few works ( Dong et al. , 2018 ; Smithson et al. , 2016 ; Ye-Hoon et al. , 2017 ) attempted to take additional constraints such as memory consumption or inference time into consideration when evaluating candidate architectures , although it is of crucial importance when targeting mobile devices . In this article , we introduce Dvolver , a principled multi-objective neural architecture search approach using Pareto optimality to navigate the search space . We instantiate this approach with a standard evolutionary algorithm ( Deb et al. , 2002 ) on the NASNet search space ( Zoph et al. , 2017 ) with validation accuracy and the total number of floating point operations ( FLOPS ) as objectives . Our experiments show that ( i ) we do not sacrifice accuracy by looking for faster architectures ( ii ) we are able to explore the search space without loss in efficiency ( iii ) we are able to find architec- 1Using 20 NVIDIA V100 GPUs 2Code and checkpoints are publicly available at https : //goo.gl/1QZX6a . tures that outperform NASNet-A mobile and certain configurations of MobileNetV2 architectures ( Sandler et al. , 2018 ) when accounting for both accuracy and floating point operations count . 2 RELATED WORK . Our work is founded on previous successful contributions focusing on single objective neural network architecture search that we extend for multiple criteria . Recently , independent papers came out with a similar approach but their search space is orders of magnitude smaller than the best studies so far . Mono-objective Search Zoph et al . ( 2017 ) introduce a modular search space , based on reusable cells , which not only reduces the computational cost of the search process but also allows the search to be carried out on a small database and transfered to a larger one . They use RL to train a Recurrent Neural Network ( RNN ) that generates the structure of the cells . Real et al . ( 2018 ) showed that artificially-evolved architectures , on the same search space , are at least as good as their RL counterpart . On a similar note , Liu et al . ( 2017a ) adopted SMBO coupled with a progressive exploration of a variation of said space , greatly reducing the search cost while maintaining competitive results . Their approaches achieved state of the art performances on ImageNet ( Deng et al. , 2009 ) , with modules learned on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) but they only optimized for best validation accuracy . The best module is then combined in a scaled down network and compared with the best mobile architecture at the time . Even though , the results are impressive , this is sub-optimal and explicitly optimizing for validation accuracy and speed can lead to even better architecture . Multi-objective Search Smithson et al . ( 2016 ) used Pareto optimality as a selection criteria in an SMBO-based hyperparameter search , and showed that the obtained Pareto front is close to the one obtained via exhaustive search , even with two orders of magnitude fewer architecture evaluations . However , their work focused on hyperparameter search and did not explore a more general search space . Elsken et al . ( 2018 ) used network morphism to optimize network architecture search on multiple criteria namely test error and parameters count . While an interesting approach , this paper lacks clear comparison with single objective methods and only evaluates architecture transferability on ImageNet64x64 ( Chrabaszcz et al. , 2017 ) . In Ye-Hoon et al . ( 2017 ) , a multi-objective genetic algorithm is implemented to search for fast and accurate architectures , initializing the search process from a given baseline network . They were able to find architectures that outperformed common baselines 3 both in speed and accuracy on MNIST and CIFAR10 . Their method , while in principle similar to ours , explored the vicinity of a defined network and was not extended to a complex search space where the search process is initialized arbitrarily . Furthermore , it was not validated for learning architectures that transfer to large scale applications . Dong et al . ( 2018 ) work is perhaps the closest to ours in intent and scale , as it performs multiple objective architecture search using Pareto optimality and uses a proxy dataset to avoid an expensive search on ImageNet . It even goes one step further to incorporate device-dependent metrics . However , their search space is orders of magnitude smaller than ours in addition to being heavily biased towards mobile architectures . 3 PROPOSED METHOD . For a given search space , our goal is to find all the neural network architectures that offer the best trade-offs between multiple independent criteria . We formulate the search process as a multiobjective optimization ( MOO ) problem ( Section 3.1 ) where each objective to optimize is an independent scalar value . We refer to Figure 1 for a general overview of the search process . The MOO controller selects child networks with different architectures . The child networks are evaluated on multiple criteria by the Evaluation Engine . The resulting objective values are then used 3namely Lenet ( Lecun et al. , 1998 ) and an all convolutional net ( Springenberg et al. , 2014 ) by the controller as feedback so it can generates better architectures over time . At each iteration , Dvolver keeps the hall of fame of the best architectures found so far . 3.1 MULTI-OBJECTIVE OPTIMIZATION AND PARETO DOMINANCE . We define the search space X as the set of all feasible solutions and the objective space as the set of all possible values of the objective function f : X → Rk . Dvolver searches for solutions by maximizing the objective function f over X with the total order relation defined next . For a nontrivial multi-objective optimization problem , there is no solution that simultaneously optimizes all the objectives . In that case , the objective functions are said to be conflicting , and there exists a ( possibly infinite ) number of compromise solutions called Paretooptimal solutions . The set of Pareto optimal outcomes is often called the Pareto front . A simple approach to cope with multi-objective functions is to use scalarization which squashes all the objective functions into a single function falling back to single-objective optimization . While very simple in practice , this method suffers from many shortcomings such as ( i ) the difficulty to combine multiple unrelated criteria into a single value and ( ii ) inefficient sampling of the full Pareto front . A much better solution is to use Pareto dominance partially ordered relation defined as follow : A solution v Pareto-dominates a solution u , if all objective values for v are greater than the objective values for u but a least one is strictly superior . A feasible solution is Pareto optimal , if it is not dominated by another solution . Without additional subjective preference information , all Pareto optimal solutions are considered equally good . For the search process to be effective , it needs a totally ordered relation . Deb et al . ( 2002 ) defines the crowding distance value of a particular solution di of point i as a measure of the objective space around i which is not occupied by any other solution in the population . Here , we simply calculate this quantity di by estimating the perimeter of the cuboid ( Figure 8 ) formed by using the nearest neighbors in the objective space as the vertices ( See appendix A.2 for more details ) . In addition to the Pareto dominance criteria , the crowding distance defines the totally ordered relation used by Dvolver to perform its search process . 3.2 SEARCH ALGORITHM . Dvolver evolves a population ofN individual architectures . At each iteration , the population evolves closer and closer toward the Pareto front , effectively sampling the full Pareto front in a single search process . Our method starts by generating a random population and for each generation , it generates a child population , evaluates each child by training the networks for a few epochs on CIFAR-10 and selects the N dominant architectures from the 2×N individuals ( parents and children ) . The details of the algorithm is presented in appendix A.1 . Algorithm convergence is monitored with the hypervolume indicator defined as the volume under the Pareto front ( see A.3 ) . In Algorithm 1 , we present the Dvolver selection process inspired by NSGA-II ( Deb et al. , 2002 ) . At the end of an iteration , there are N parents and N children . We should select the best N individuals to form the parents ’ population for the next iteration . We compute the Pareto fronts for the 2 ×N individuals . The best fronts are considered first . If there are more than N individuals in the Algorithm 1 : Dvolver selection algorithm Inputs • N individuals from parents ’ population • N individuals from offsprings ’ population Output • N selected individuals Indssel 1 : Indssel ← ∅ ; 2 : fronts← ComputeParetoFronts ( parents ∪ offsprings ) ; 3 : i← 0 ; . Starts with Pareto front with rank 0 4 : rem← N ; . rem is the number of remaining individuals to be selected 5 : while rem > 0 do Indsi ← SortWithDominance ( frontsi ) ; if Len ( Indsi ) < rem then Indssel ← Indssel ∪ Indsi ; rem← rem− Len ( Indsi ) ; i← i+ 1 ; . Move to the next Pareto front else Indssel ← Indssel ∪ First ( rem , Indsi ) ; . Highest crowding distance individuals first rem← 0 ; best front , the first N ( with the help of the crowding distance ) are selected and the selection ends . Otherwise , all the individuals from the front are selected and we carry on with the next best Pareto front until N individuals are selected . 3.3 SEARCH SPACE . Our work is based on the work of Zoph et al . ( 2017 ) , in which network architecture is composed of two levels : a micro level architecture defined by parametrized cells composed of B blocks and a macro architecture where those cells are stacked a predefined number of times , in order to create the final network . Figure 2 describes the macro architecture used for CIFAR-10 and ImageNet architectures . There are two types of cells and Dvolver searches for both at the same time : ( i ) Normal Cell that returns a feature map of the same size as its input and ( ii ) Reduction Cell that returns a feature map of half the size of its input in the spatial dimensions . A cell is composed of B blocks arranged in a Directed Acyclic Graph ( DAG ) . Each block is a mapping of two input tensors to one output tensor and it can be represented by a 4-tuple , ( I1 , O1 , I2 , O2 ) where Ik specifies the input for the operation Ok . The result of the two operations O1 and O2 are summed together to form the output of the block . A block can take its input from previous blocks , the previous cell ck−1 or the previous previous cell ck−2 . Unused blocks are concatenated in the depth dimension to form the final output of the cell . Moreover , we noticed that the original NASNet search space can significantly benefit from extra connections from any given block , the previous cell and the previous previous cell to the final concatenation layer . In fact , these extra connections are already present but not described in the best models from Zoph et al . ( 2017 ) , Real et al . ( 2018 ) and Liu et al . ( 2017a ) . Instead of adding manually these connections after the search process like in the previous works , Dvolver optimizes for these extra connections in addition to the 4 parameters for each block . In practice , our search space is the same as the one used by Zoph et al . ( 2017 ) but no manual intervention is needed . For B = 5 , the total number of parameters for each type of cell is 20 parameters for the blocks and one list of indexes of extra connections . Similar to the approach of Liu et al . ( 2017a ) , we simplify the search space and consider only the following operations : • identity • 3x3 average pooling • 3x3 max pooling • 3x3 depthwise-separable conv • 5x5 depthwise-separable conv • 7x7 depthwise-separable conv The total number of combinations for the defined search space is about 1020 taking into account symmetries and possible redundant connections . Note that our search space is larger than the one defined in Liu et al . ( 2017a ) but we show in Section 4 that Dvolver is capable of finding competitive architectures faster .

Summary: The paper proposes a multi-objective search algorithm that designs resource-efficient convolutional architectures. The key idea is to maintain a population of networks and to iteratively approach the Pareto front through evolution. The normal & reduction cells are searched on CIFAR-10 and then transferred to ImageNet. The resulting architectures empirically lead to better trade-offs than other baselines.