NSGA-Net: A Multi-Objective Genetic Algorithm for Neural Architecture Search

1 INTRODUCTION . Deep convolutional neural networks have been overwhelmingly successful at a variety of image analysis tasks . One of the key driving forces behind this success is the introduction of many CNN architectures such as , AlexNet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2015 ) , GoogLeNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016a ) , DenseNet ( Huang et al. , 2017 ) etc . in the context of object classification and Hourglass ( Newell et al. , 2016 ) , and Convolutional Pose Machines ( Wei et al. , 2016a ) in the context of object alignment . Concurrently , network designs such as MobileNet ( Howard et al. , 2017 ) , XNOR-Net ( Rastegari et al. , 2016 ) , BinaryNets ( Courbariaux et al. , 2016 ) , LBCNN ( Juefei-Xu et al. , 2017 ) etc . have been developed with the goal of enabling realworld deployment of high performance models on resource constrained devices . These developments are the fruits of years of painstaking efforts and human ingenuity . Neural architecture search ( NAS ) methods , on the other hand , seek to automate the process of designing network architectures . State-of-the-art reinforcement learning ( RL ) ( Baker et al. , 2017 ; Zhong et al. , 2017 ; Zoph & Le , 2016 ; Zoph et al. , 2017 ; Hsu et al. , 2018a ; Cai et al. , 2018 ; Pham et al. , 2018 ) and evolutionary algorithm ( EA ) ( Miikkulainen et al. , 2017 ; Real et al. , 2017 ; 2018 ; Xie & Yuille , 2017 ; Kim et al. , 2017 ; Liu et al. , 2018 ; Elsken et al. , 2018 ; Liu et al. , 2017 ; Dong et al. , 2018 ) approaches for NAS focus on the single objective of minimizing an error metric on a task and can not be easily adapted to handle minimizing multiple , possibly conflicting , objectives . Methods like ( Real et al. , 2017 ) and ( Zoph et al. , 2017 ) are inefficient in their use of their search space and require 3,150 and 2,000 GPU days , respectively . Furthermore , most state-of-the-art approaches search over a single computation block , similar to an Inception block ( Szegedy et al. , 2015 ) , and repeat it as many times as necessary to form a complete network . In this paper , we present NSGA-Net , a genetic algorithm for NAS to address the aforementioned limitations of current approaches . The salient features of NSGA-Net are , ( 1 ) Multiobjective Optimization : Real-world deployment of NAS models is seldom guided by a single objective and often has to balance between multiple , possibly competing , objectives . For instance , we seek to maximize performance on compute devices that are often constrained by hardware resources in terms of power consumption , available memory , available FLOPS , and latency constraints , to name a few . NSGA-Net is explicitly designed to optimize such competing objectives . ( 2 ) Complete Architecture Search Space : The search space for most existing methods is restricted to a block that is repeated as many times as desired . In contrast , NSGA-Net searches over the entire structure of the network . This scheme overcomes the limitations inherent to repeating the same computation block throughout an entire network , namely , that a single block may not be optimal for every application and it is desirable to allow NAS to discover architectures with different blocks in different parts of the network . ( 3 ) Non-Dominated Sorting : The core component of NSGA-Net is the Non-Dominated Sorting Genetic Algorithm II ( NSGA-II ) ( Deb et al. , 2000 ) , a multi-objective optimization algorithm that has been successfully employed for solving a variety of multiobjective problems ( Tapia & Coello , 2007 ; Pedersen & Yang , 2006 ) . Here , we leverage it ’ s ability to maintain a diverse trade-off frontier between multiple , possibly conflicting , objectives , thereby resulting in a more effective and efficient exploration of the search space , ( 4 ) Crossover : To fully utilize the diverse frontier of solutions provided by non-dominated sorting , we employ crossover ( in addition to mutation ) to combine networks with desirable qualities across multiple objectives , and finally ( 5 ) BOA : We construct and employ a Bayesian Network inspired by the Bayesian Optimization Algorithm ( BOA ) ( Pelikan et al. , 1999 ) to fully utilize the promising solutions present in our search history archive and the inherent correlations between the layers of the network architecture . We demonstrate the efficacy of NSGA-Net on two tasks : image classification ( CIFAR-10 ( Krizhevsky & Hinton , 2009 ) ) and object alignment or key-point prediction ( CMU-Car ( Boddeti et al. , 2013 ) ) . For both tasks we minimize two objectives : an error metric and computational complexity . Here , computational complexity is defined by the number of floating-point operations or multiply-adds ( FLOPS ) . Experimentally , we observe NSGA-Net can find a set of network architectures containing solutions that are significantly better than hand-crafted methods in both objectives while being competitive in single objectives to state-of-the-art NAS approaches . Furthermore , by fully utilizing a population of networks through crossover and utilization of the search history , NSGA-Net explores the search space efficiently and requires less computation time for searching than competing methods . 2 RELATED WORK . Recent research efforts in NAS have produced a plethora of methods to automate the design of networks . Broadly speaking , these methods can be divided into EA approaches and RL approaches—with a few methods not falling into either category . Due to space constraints , we provide a comprehensive overview of relevant methods and a table summarizing their contributions in Appendix D. Here , we focus on multiobjective methods , and relate NSGA-Net to recent contributions . Kim et al . ( 2017 ) presented NEMO , one of the earliest multiobjective approaches involving neural networks . NEMO used NSGA-II ( Deb et al. , 2000 ) to minimize error and inference time of a network and searched over the space of the number of output channels from each layer within a restricted space of 7 different architectures . In contrast , NSGA-Net seeks to use NSGA-II to minimize error and computational complexity ( as measured by FLOPS ) , while searching over the vast space of possible network architectures but with fixed hyperparameters . Dong et al . ( 2018 ) proposed PPP-Net as a multiobjective extension to the progressive NAS method in ( Liu et al. , 2017 ) that uses a predictive model to choose promising networks to train in an effort to reduce computational strain . PPP-Net differs from NSGA-Net by not using crossover and the use of this predictive model . Elsken et al . ( 2018 ) present the LEMONADE method which is formulated to develop networks with low error and number of parameters through custom designed approximate network morphisms ( Wei et al. , 2016b ) . This allows newly generated networks to share parameters with their forerunners , obviating network training from scratch . NSGA-Net differs from LEMONADE in terms of the encoding scheme , the network morphisms as well in the selection scheme . NSGA-Net relies on , ( 1 ) genetic operations like mutation and crossover , encouraging population diversity , instead of custom network morphism operations , and ( 2 ) NSGA-II , rather than novelty-based sampling as the selection scheme in LEMONADE , with the former affording a more efficient exploration of the search space . Finally , we highlight MONAS , presented by Hsu et al . ( 2018a ) . This approach is one of the first—if not the only—RL NAS method to use a multiobjective scheme . MONAS searches the same space that is presented in ( Zoph et al. , 2017 ) while using a reward signal to optimize a recurrent neural network to generate convolutional networks . The reward signal however is a linear combination of accuracy and energy consumption of a network . However , it is well known in multiobjective optimization literature that a simple linear combination of objectives suffers from a number of drawbacks , including sub-optimality ( Koski , 1985 ) of the search ( see Section 3 for a more detailed discussion ) . 3 PROPOSED APPROACH . Compute devices are often constrained by hardware resources in terms of power consumption , available memory , available FLOPS , and latency constraints . Hence , real-world design of DNNs are required to balance multiple , possibly competing , objectives ( e.g. , predictive performance and computational complexity ) . Often , when multiple design criteria are considered simultaneously , there may not exist a single solution that performs optimally in all desired criteria , especially with competing objectives . Under such circumstances , a set of solutions that provides the entire trade-off information between the objectives is more desirable . This enables a practitioner to analyze the importance of each criterion , depending on the application , and to choose an appropriate solution on the trade-off frontier . We propose NSGA-Net , a genetic algorithm based architecture search method to automatically generate a set of DNN architectures that approximate the Pareto-front between performance and complexity on classification and regression tasks . The rest of this section describes the design principles , encoding scheme , and main components of NSGA-Net in detail . 3.1 DESIGN PRINCIPLES Population Based Methods : The two main approaches to obtaining an efficient set of trade-off solutions are : i ) classical point-based methods like a weighted sum of the objective functions ( e.g. , αf1 ( x ) + ( 1−α ) f2 ( x ) ) ; and ii ) population-based methods like genetic algorithms . Despite the well characterized convergence properties of pointbased methods , they often present the following challenges as illustrated in Fig.2 : 1 ) They require prior knowledge of the range of values each objective function can take for appropriate normalization before weighting , else , the solutions are often biased towards the objective with large values ; 2 ) weighted combinations of objectives can only find solutions on convex regions of the Pareto-front and are incapable of discovering solutions on concave parts of the Pareto-front ( Koski , 1985 ) ; finally 3 ) obtaining each solution on the Pareto-front requires the repetition of the entire search procedure for each combination weight α . On the other hand , population-based methods , like genetic algorithms , offer a flexible , viable alternative to find multiple efficient trade-off solutions in one execution . A population-based method is capable of introducing an implicit parallel search by processing multiple solution candidates jointly at each iteration . However , optimizing with N population members processes O ( N3 ) sub-regions of the search space in parallel ( Goldberg , 1989 ; Holland , 1975 ) . The efficiency afforded by such parallelism can not be matched by point-based methods like weighted combinations . Recent studies show that population-based methods can successfully solve problems with millions ( Chicano et al. , 2017 ) or billions ( Deb & Myburgh , 2017 ) of variables while classical point-based methods , like branch-and-bound , fail even with hundreds of variables . Multi-Criterion Based Selection : The general form of a multiobjective optimization problem is min x { f1 ( x ) , . . . , fM ( x ) } ( 1 ) where fi ( · ) are the criterion that we wish to optimize and x is the representation of a neural network architecture ( described in Section 3.2 ) . For the aforementioned problem , given solutions x1 and x2 , x1 is said to dominate x2 ( i.e. , x1 x2 ) if both of the following conditions are satisfied , 1. x1 is no worse than x2 for all objectives ( fi ( x1 ) ≤ fi ( x2 ) ∀i ∈ { 0 , . . . , M } ) 2. x1 is strictly better than x2 in at least one objective ( ∃ i ∈ { 0 , . . . , M } | fi ( x1 ) < fi ( x2 ) ) Therefore , a solution xi is non-dominated if these conditions hold for all xj and j 6= i . The core of NSGA-Net is a selection criterion that leverages non-dominated solutions . Specifically , given a population of network architectures { x1 , . . . , xn } and their fitness functions { f1xi , . . . , fM ( xi ) } , the ranking and selection procedure consists of two stages : ( 1 ) non-dominated solutions are selected over dominated solutions ; ( 2 ) explicitly ranking of solutions that are diverse w.r.t . to the trade-off between the objectives higher than solutions that are “ crowded '' on the trade-off front ( see Fig . 3c ) i.e. , how close a given solution is to its neighbors in the objective space . We adopt the non-domination ranking and crowded-ness measurements proposed by Deb et al . ( 2000 ) . The non-domination ranking indicates the front number that a solution belongs to ; these fronts are composed by the set of non-dominated solutions at the current search iteration . An example of the non-domination fronts and non-domination ranking is shown in Fig . 3a and Fig . 3b , respectively . It ’ s worth noting that both non-domination ranking and crowded-ness are relative measurements which need to be re-assessed when new solutions are created . Elitist-preserving : This term refers to the fact that the best solution ( in terms of objective values and crowded-ness ) in the parent population will always be carried on to the next population . This allows the previous best solution to always have a chance to share its genetic information with the next generation , without the risk of losing the information to the newly generated child population . As a consequence of this scheme , the best solution encountered during the entirety of the search will always be present in the final population .

Summary: This paper proposes an evolutionary-based method for the multi-objective neural architecture search, where the proposed method aims at minimizing two objectives: an error metric and the number of FLOPS. The proposed method consists of an exploration step and an exploitation step. In the exploration step, architectures are sampled by using genetic operators such as the crossover and the mutation. In the exploitation step, architectures are generated by a Bayesian Network. The proposed method is evaluated on object classification and object alignment tasks.