NSGA-Net: A Multi-Objective Genetic Algorithm for Neural Architecture Search

1 INTRODUCTION . Deep convolutional neural networks have been overwhelmingly successful at a variety of image analysis tasks . One of the key driving forces behind this success is the introduction of many CNN architectures such as , AlexNet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2015 ) , GoogLeNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016a ) , DenseNet ( Huang et al. , 2017 ) etc . in the context of object classification and Hourglass ( Newell et al. , 2016 ) , and Convolutional Pose Machines ( Wei et al. , 2016a ) in the context of object alignment . Concurrently , network designs such as MobileNet ( Howard et al. , 2017 ) , XNOR-Net ( Rastegari et al. , 2016 ) , BinaryNets ( Courbariaux et al. , 2016 ) , LBCNN ( Juefei-Xu et al. , 2017 ) etc . have been developed with the goal of enabling realworld deployment of high performance models on resource constrained devices . These developments are the fruits of years of painstaking efforts and human ingenuity . Neural architecture search ( NAS ) methods , on the other hand , seek to automate the process of designing network architectures . State-of-the-art reinforcement learning ( RL ) ( Baker et al. , 2017 ; Zhong et al. , 2017 ; Zoph & Le , 2016 ; Zoph et al. , 2017 ; Hsu et al. , 2018a ; Cai et al. , 2018 ; Pham et al. , 2018 ) and evolutionary algorithm ( EA ) ( Miikkulainen et al. , 2017 ; Real et al. , 2017 ; 2018 ; Xie & Yuille , 2017 ; Kim et al. , 2017 ; Liu et al. , 2018 ; Elsken et al. , 2018 ; Liu et al. , 2017 ; Dong et al. , 2018 ) approaches for NAS focus on the single objective of minimizing an error metric on a task and can not be easily adapted to handle minimizing multiple , possibly conflicting , objectives . Methods like ( Real et al. , 2017 ) and ( Zoph et al. , 2017 ) are inefficient in their use of their search space and require 3,150 and 2,000 GPU days , respectively . Furthermore , most state-of-the-art approaches search over a single computation block , similar to an Inception block ( Szegedy et al. , 2015 ) , and repeat it as many times as necessary to form a complete network . In this paper , we present NSGA-Net , a genetic algorithm for NAS to address the aforementioned limitations of current approaches . The salient features of NSGA-Net are , ( 1 ) Multiobjective Optimization : Real-world deployment of NAS models is seldom guided by a single objective and often has to balance between multiple , possibly competing , objectives . For instance , we seek to maximize performance on compute devices that are often constrained by hardware resources in terms of power consumption , available memory , available FLOPS , and latency constraints , to name a few . NSGA-Net is explicitly designed to optimize such competing objectives . ( 2 ) Complete Architecture Search Space : The search space for most existing methods is restricted to a block that is repeated as many times as desired . In contrast , NSGA-Net searches over the entire structure of the network . This scheme overcomes the limitations inherent to repeating the same computation block throughout an entire network , namely , that a single block may not be optimal for every application and it is desirable to allow NAS to discover architectures with different blocks in different parts of the network . ( 3 ) Non-Dominated Sorting : The core component of NSGA-Net is the Non-Dominated Sorting Genetic Algorithm II ( NSGA-II ) ( Deb et al. , 2000 ) , a multi-objective optimization algorithm that has been successfully employed for solving a variety of multiobjective problems ( Tapia & Coello , 2007 ; Pedersen & Yang , 2006 ) . Here , we leverage it ’ s ability to maintain a diverse trade-off frontier between multiple , possibly conflicting , objectives , thereby resulting in a more effective and efficient exploration of the search space , ( 4 ) Crossover : To fully utilize the diverse frontier of solutions provided by non-dominated sorting , we employ crossover ( in addition to mutation ) to combine networks with desirable qualities across multiple objectives , and finally ( 5 ) BOA : We construct and employ a Bayesian Network inspired by the Bayesian Optimization Algorithm ( BOA ) ( Pelikan et al. , 1999 ) to fully utilize the promising solutions present in our search history archive and the inherent correlations between the layers of the network architecture . We demonstrate the efficacy of NSGA-Net on two tasks : image classification ( CIFAR-10 ( Krizhevsky & Hinton , 2009 ) ) and object alignment or key-point prediction ( CMU-Car ( Boddeti et al. , 2013 ) ) . For both tasks we minimize two objectives : an error metric and computational complexity . Here , computational complexity is defined by the number of floating-point operations or multiply-adds ( FLOPS ) . Experimentally , we observe NSGA-Net can find a set of network architectures containing solutions that are significantly better than hand-crafted methods in both objectives while being competitive in single objectives to state-of-the-art NAS approaches . Furthermore , by fully utilizing a population of networks through crossover and utilization of the search history , NSGA-Net explores the search space efficiently and requires less computation time for searching than competing methods . 2 RELATED WORK . Recent research efforts in NAS have produced a plethora of methods to automate the design of networks . Broadly speaking , these methods can be divided into EA approaches and RL approaches—with a few methods not falling into either category . Due to space constraints , we provide a comprehensive overview of relevant methods and a table summarizing their contributions in Appendix D. Here , we focus on multiobjective methods , and relate NSGA-Net to recent contributions . Kim et al . ( 2017 ) presented NEMO , one of the earliest multiobjective approaches involving neural networks . NEMO used NSGA-II ( Deb et al. , 2000 ) to minimize error and inference time of a network and searched over the space of the number of output channels from each layer within a restricted space of 7 different architectures . In contrast , NSGA-Net seeks to use NSGA-II to minimize error and computational complexity ( as measured by FLOPS ) , while searching over the vast space of possible network architectures but with fixed hyperparameters . Dong et al . ( 2018 ) proposed PPP-Net as a multiobjective extension to the progressive NAS method in ( Liu et al. , 2017 ) that uses a predictive model to choose promising networks to train in an effort to reduce computational strain . PPP-Net differs from NSGA-Net by not using crossover and the use of this predictive model . Elsken et al . ( 2018 ) present the LEMONADE method which is formulated to develop networks with low error and number of parameters through custom designed approximate network morphisms ( Wei et al. , 2016b ) . This allows newly generated networks to share parameters with their forerunners , obviating network training from scratch . NSGA-Net differs from LEMONADE in terms of the encoding scheme , the network morphisms as well in the selection scheme . NSGA-Net relies on , ( 1 ) genetic operations like mutation and crossover , encouraging population diversity , instead of custom network morphism operations , and ( 2 ) NSGA-II , rather than novelty-based sampling as the selection scheme in LEMONADE , with the former affording a more efficient exploration of the search space . Finally , we highlight MONAS , presented by Hsu et al . ( 2018a ) . This approach is one of the first—if not the only—RL NAS method to use a multiobjective scheme . MONAS searches the same space that is presented in ( Zoph et al. , 2017 ) while using a reward signal to optimize a recurrent neural network to generate convolutional networks . The reward signal however is a linear combination of accuracy and energy consumption of a network . However , it is well known in multiobjective optimization literature that a simple linear combination of objectives suffers from a number of drawbacks , including sub-optimality ( Koski , 1985 ) of the search ( see Section 3 for a more detailed discussion ) . 3 PROPOSED APPROACH . Compute devices are often constrained by hardware resources in terms of power consumption , available memory , available FLOPS , and latency constraints . Hence , real-world design of DNNs are required to balance multiple , possibly competing , objectives ( e.g. , predictive performance and computational complexity ) . Often , when multiple design criteria are considered simultaneously , there may not exist a single solution that performs optimally in all desired criteria , especially with competing objectives . Under such circumstances , a set of solutions that provides the entire trade-off information between the objectives is more desirable . This enables a practitioner to analyze the importance of each criterion , depending on the application , and to choose an appropriate solution on the trade-off frontier . We propose NSGA-Net , a genetic algorithm based architecture search method to automatically generate a set of DNN architectures that approximate the Pareto-front between performance and complexity on classification and regression tasks . The rest of this section describes the design principles , encoding scheme , and main components of NSGA-Net in detail . 3.1 DESIGN PRINCIPLES Population Based Methods : The two main approaches to obtaining an efficient set of trade-off solutions are : i ) classical point-based methods like a weighted sum of the objective functions ( e.g. , αf1 ( x ) + ( 1−α ) f2 ( x ) ) ; and ii ) population-based methods like genetic algorithms . Despite the well characterized convergence properties of pointbased methods , they often present the following challenges as illustrated in Fig.2 : 1 ) They require prior knowledge of the range of values each objective function can take for appropriate normalization before weighting , else , the solutions are often biased towards the objective with large values ; 2 ) weighted combinations of objectives can only find solutions on convex regions of the Pareto-front and are incapable of discovering solutions on concave parts of the Pareto-front ( Koski , 1985 ) ; finally 3 ) obtaining each solution on the Pareto-front requires the repetition of the entire search procedure for each combination weight α . On the other hand , population-based methods , like genetic algorithms , offer a flexible , viable alternative to find multiple efficient trade-off solutions in one execution . A population-based method is capable of introducing an implicit parallel search by processing multiple solution candidates jointly at each iteration . However , optimizing with N population members processes O ( N3 ) sub-regions of the search space in parallel ( Goldberg , 1989 ; Holland , 1975 ) . The efficiency afforded by such parallelism can not be matched by point-based methods like weighted combinations . Recent studies show that population-based methods can successfully solve problems with millions ( Chicano et al. , 2017 ) or billions ( Deb & Myburgh , 2017 ) of variables while classical point-based methods , like branch-and-bound , fail even with hundreds of variables . Multi-Criterion Based Selection : The general form of a multiobjective optimization problem is min x { f1 ( x ) , . . . , fM ( x ) } ( 1 ) where fi ( · ) are the criterion that we wish to optimize and x is the representation of a neural network architecture ( described in Section 3.2 ) . For the aforementioned problem , given solutions x1 and x2 , x1 is said to dominate x2 ( i.e. , x1 x2 ) if both of the following conditions are satisfied , 1. x1 is no worse than x2 for all objectives ( fi ( x1 ) ≤ fi ( x2 ) ∀i ∈ { 0 , . . . , M } ) 2. x1 is strictly better than x2 in at least one objective ( ∃ i ∈ { 0 , . . . , M } | fi ( x1 ) < fi ( x2 ) ) Therefore , a solution xi is non-dominated if these conditions hold for all xj and j 6= i . The core of NSGA-Net is a selection criterion that leverages non-dominated solutions . Specifically , given a population of network architectures { x1 , . . . , xn } and their fitness functions { f1xi , . . . , fM ( xi ) } , the ranking and selection procedure consists of two stages : ( 1 ) non-dominated solutions are selected over dominated solutions ; ( 2 ) explicitly ranking of solutions that are diverse w.r.t . to the trade-off between the objectives higher than solutions that are “ crowded '' on the trade-off front ( see Fig . 3c ) i.e. , how close a given solution is to its neighbors in the objective space . We adopt the non-domination ranking and crowded-ness measurements proposed by Deb et al . ( 2000 ) . The non-domination ranking indicates the front number that a solution belongs to ; these fronts are composed by the set of non-dominated solutions at the current search iteration . An example of the non-domination fronts and non-domination ranking is shown in Fig . 3a and Fig . 3b , respectively . It ’ s worth noting that both non-domination ranking and crowded-ness are relative measurements which need to be re-assessed when new solutions are created . Elitist-preserving : This term refers to the fact that the best solution ( in terms of objective values and crowded-ness ) in the parent population will always be carried on to the next population . This allows the previous best solution to always have a chance to share its genetic information with the next generation , without the risk of losing the information to the newly generated child population . As a consequence of this scheme , the best solution encountered during the entirety of the search will always be present in the final population .

Summary: This paper proposes a search method for neural network architectures such that two (potentially) conflicting objectives: maximization of final performance and minimization of computational complexity can be pursued simultaneously. The motivation for the approach is that a principled multiobjective search procedure (NSGA-II) makes it unnecessary to manually find the right trade-off between two objectives, and simultaneously finds several solutions spanning the tradeoff. It is also capable of finding solutions from the concave regions of the Pareto-front. Multiobjective search for architectures has been explored in recent work, so the primary contribution of this paper is to show its utility in a more general and perhaps more powerful setting.