Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

1 INTRODUCTION . Deep learning has enabled remarkable progress on a variety of perceptual tasks , such as image recognition ( Krizhevsky et al. , 2012 ) , speech recognition ( Hinton et al. , 2012 ) , and machine translation ( Bahdanau et al. , 2015 ) . One crucial aspect for this progress are novel neural architectures ( Szegedy et al. , 2016 ; He et al. , 2016 ; Huang et al. , 2017b ) . Currently employed architectures have mostly been developed manually by human experts , which is a time-consuming and error-prone process . Because of this , there is growing interest in automatic architecture search methods ( Elsken et al. , 2018 ) . Some of the architectures found in an automated way have already outperformed the best manually-designed ones ; however , algorithms such as by Zoph & Le ( 2017 ) ; Zoph et al . ( 2018 ) ; Real et al . ( 2017 ; 2018 ) for finding these architectures require enormous computational resources often in the range of thousands of GPU days . Prior work on architecture search has typically framed the problem as a single-objective optimization problem . However , most applications of deep learning do not only require high predictive performance on unseen data but also low resource-consumption in terms of , e.g. , inference time , model size or energy consumption . Moreover , there is typically an implicit trade-off between predictive performance and consumption of resources . Recently , several architectures have been manually designed that aim at reducing resource-consumption while retaining high predictive performance ( Iandola et al. , 2016 ; Howard et al. , 2017 ; Sandler et al. , 2018 ) . Automatically found neural architectures have also been down-scaled to reduce resource consumption ( Zoph et al. , 2018 ) . However , very little previous work has taken the trade-off between resource-consumption and predictive performance into account during automatic architecture search . In this work , we make the following two main contributions : 1 . To overcome the need for thousands of GPU days ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Real et al. , 2018 ) , we make use of operators acting on the space of neural network architectures that preserve the function a network represents , dubbed network morphisms ( Chen et al. , 2015 ; Wei et al. , 2016 ) , obviating training from scratch and thereby substantially reducing the required training time per network . This mechanism can be interpreted as Lamarckian inheritance in the context of evolutionary algorithms , where Lamarckism refers to a mechanism which allows passing skills acquired during an individual ’ s lifetime ( e.g. , by means of learning ) , on to children by means of inheritance . Since network morphisms are limited to solely increasing a network ’ s size ( and therefore likely also resource consumption ) , we introduce approximate network morphisms ( Section 3.2 ) to also allow shrinking networks , which is essential in the context of multi-objective search . The proposed Lamarckian inheritance mechanism could in principle be combined with any evolutionary algorithm for architecture search , or any other method using ( a combination of ) localized changes in architecture space . 2 . We propose a Lamarckian Evolutionary algorithm for Multi-Objective Neural Architecture DEsign , dubbed LEMONADE , Section 4 , which is suited for the joint optimization of several objectives , such as predictive performance , inference time , or number of parameters . LEMONADE maintains a population of networks on an approximation of the Pareto front of the multiple objectives . In contrast to generic multi-objective algorithms , LEMONADE exploits that evaluating certain objectives ( such as an architecture ’ s number of parameters ) is cheap while evaluating the predictive performance on validation data is expensive ( since it requires training the model first ) . Thus , LEMONADE handles its various objectives differently : it first selects a subset of architectures , assigning higher probability to architectures that would fill gaps on the Pareto front for the “ cheap ” objectives ; then , it trains and evaluates only this subset , further reducing the computational resource requirements during architecture search . In contrast to other multi-objective architecture search methods , LEMONADE ( i ) does not require to define a trade-off between performance and other objectives a-priori ( e.g. , by weighting objectives when using scalarization methods ) but rather returns a set of architectures , which allows the user to select a suitable model a-posteriori ; ( ii ) LEMONADE does not require to be initialized with well performing architectures ; it can be initialized with trivial architectures and hence requires less prior knowledge . Also , LEMONADE can handle various search spaces , including complex topologies with multiple branches and skip connections . We evaluate LEMONADE for up to five objectives on two different search spaces for image classification : ( i ) non-modularized architectures and ( ii ) cells that are used as repeatable building blocks within an architecture ( Zoph et al. , 2018 ; Zhong et al. , 2018 ) and also allow transfer to other data sets . LEMONADE returns a population of CNNs covering architectures with 10 000 to 10 000 000 parameters . Within only 5 days on 16 GPUs , LEMONADE discovers architectures that are competitive in terms of predictive performance and resource consumption with hand-designed networks , such as MobileNet V2 ( Sandler et al. , 2018 ) , as well as architectures that were automatically designed using 40x greater resources ( Zoph et al. , 2018 ) and other multi-objective methods ( Dong et al. , 2018 ) . 2 BACKGROUND AND RELATED WORK . Multi-objective Optimization Multi-objective optimization ( Miettinen , 1999 ) deals with problems that have multiple , complementary objective functions f1 , . . . , fn . Let N be the space of feasible solutions N ( in our case the space of feasible neural architectures ) . In general , multi-objective optimization deals with finding N∗ ∈ N that minimizes the objectives f1 , . . . , fn . However , typically there is no single N∗ that minimizes all objectives at the same time . In contrast , there are multiple Pareto-optimal solutions that are optimal in the sense that one can not reduce any fi without increasing at least one fj . More formally , a solution N ( 1 ) Pareto-dominates another solution N ( 2 ) if ∀i ∈ 1 , . . . , n : fi ( N ( 1 ) ) ≤ fi ( N ( 2 ) ) and ∃ j ∈ 1 , . . . , n : fj ( N ( 1 ) ) < fj ( N ( 2 ) ) . The Pareto-optimal solutions N∗ are exactly those solutions that are not dominated by any other N ∈ N . The set of Pareto optimal N∗ is the so-called Pareto front . Neural Architecture Search We give a short overview of the most relevant works in neural architecture search ( NAS ) and refer to Elsken et al . ( 2018 ) for a comprehensive survey on the topic . It was recently proposed to frame NAS as a reinforcement learning ( RL ) problem , where the reward of the RL agent is based on the validation performance of the trained architecture ( Baker et al. , 2017a ; Zoph & Le , 2017 ; Zhong et al. , 2018 ; Pham et al. , 2018 ) . Zoph & Le ( 2017 ) use a recurrent neural network to generate a string representing the neural architecture . In a follow-up work , Zoph et al . ( 2018 ) search for cells , which are repeated according to a fixed macro architecture to generate the eventual architecture . Defining the architecture based on a cell simplifies the search space . An alternative to using RL are neuro-evolutionary approaches that use genetic algorithms for optimizing the neural architecture ( Stanley & Miikkulainen , 2002 ; Liu et al. , 2018a ; Real et al. , 2018 ; Miikkulainen et al. , 2017 ; Xie & Yuille , 2017 ) . In contrast to these works , our proposed method is applicable for multi-objective optimization and employs Lamarckian inheritance , i.e , learned parameters are passed on to a network ’ s offspring . A related approach to our Lamarckian evolution is population-based training ( Jaderberg et al. , 2017 ) , which , however , focuses on hyperparameter optimization and not on the specific properties of the optimization of neural architectures . We note that it would be possible to also include the evolution of hyperparameters in our work . Unfortunately , most of the aforementioned approaches require vast computational resources since they need to train and validate thousands of neural architectures ; e.g. , Zoph & Le ( 2017 ) trained over 10.000 neural architectures , requiring thousands of GPU days . One way of speeding up evaluation is to predict performance of a ( partially ) trained model ( Domhan et al. , 2015 ; Baker et al. , 2017b ; Klein et al. , 2017 ; Liu et al. , 2017 ) . Works on performance prediction are complementary to our work and could be incorporated in the future . One-Shot Architecture Search is another promising approach for speeding up performance estimation , which treats all architectures as different subgraphs of a supergraph ( the one-shot model ) and shares weights between architectures ( Saxena & Verbeek , 2016 ; Brock et al. , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018b ; Bender et al. , 2018 ) . Only the weights of a single one-shot model need to be trained , and architectures ( which are just subgraphs of the one-shot model ) can then be evaluated without any separate training . However , a general limitation of one-shot NAS is that the supergraph defined a-priori restricts the search space to its subgraphs . Moreover , approaches which require that the entire supergraph resides in GPU memory during architecture search will be restricted to relatively small supergraphs . It is also not obvious how one-shot models could be employed for multi-objective optimization as all subgraphs of the one-shot models are of roughly the same size and it is not clear if weight sharing would work for very different-sized architectures . LEMONADE does not suffer from any of these disadvantages ; it can handle arbitrary large , unconstrained search spaces while still being efficient . Elsken et al . ( 2017 ) ; Cai et al . ( 2018a ) proposed to employ the concept of network morphisms ( see Section 3.1 ) . The basic idea is to initialize weights of newly generated neural architectures based on weights of similar , already trained architectures so that they have the same accuracy . This pretrained initialization allows reducing the large cost of training all architectures from scratch . Our work extends this approach by introducing approximate network morphisms , making the use of such operators suitable for multi-objective optimization . Multi-objective Neural Architecture Search Very recently , there has also been some work on multi-objective neural architecture search ( Kim et al. , 2017 ; Dong et al. , 2018 ; Tan et al. , 2018 ) with the goal of not solely optimizing the accuracy on a given task but also considering resource consumption . Kim et al . ( 2017 ) parameterize an architecture by a fixed-length vector description , which limits the architecture search space drastically . In parallel , independent work to ours , Dong et al . ( 2018 ) extend PNAS ( Liu et al. , 2017 ) by considering multiple objective during the model selection step . However , they employ CondenseNet ( Huang et al. , 2017a ) as a base network and solely optimize building blocks within the network which makes the search less interesting as ( i ) the base network is by default already well performing and ( ii ) the search space is again limited . Tan et al . ( 2018 ) use a weighted product method ( Deb & Kalyanmoy , 2001 ) to obtain a scalarized objective . However , this scalarization comes with the drawback of weighting the objectives a-priori , which might not be suitable for certain applications . In contrast to all mentioned work , LEMONADE ( i ) does not require a complex macro architecture but rather can start from trivial initial networks , ( ii ) can handle arbitrary search spaces , ( iii ) does not require to define hard constraints or weights on objectives a-priori .

Summary: This paper proposes a multi-objective evolutionary algorithm for the neural architecture search. Specifically, this paper employs a Lamarckian inheritance mechanism based on network morphism operations for speeding up the architecture search. The proposed method is evaluated on CIFAR-10 and ImageNet (64*64) datasets and compared with recent neural architecture search methods. In this paper, the proposed method aims at solving the multi-objective problem: validation error rate as a first objective and the number of parameters in a network as a second objective.