Learnable Embedding Space for Efficient Neural Architecture Compression

1 INTRODUCTION . In many application domains , it is common practice to make use of well-known deep network architectures ( e.g. , VGG ( Simonyan & Zisserman , 2014 ) , GoogleNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016 ) ) and to adapt them to a new task without optimizing the architecture for that task . While this process of transfer learning is surprisingly successful , it often results in over-sized networks which have many redundant or unused parameters . Inefficient network architectures can waste computational resources and over-sized networks can prevent them from being used on embedded systems . There is a pressing need to develop algorithms that can take large networks with high accuracy as input and compress their size while maintaining similar performance . In this paper , we focus on the task of compressed architecture search – the automatic discovery of compressed network architectures based on a given large network . One significant bottleneck of compressed architecture search is the need to repeatedly evaluate different compressed network architectures , as each evaluation is extremely costly ( e.g. , backpropagation to learn the parameters of a single deep network can take several days on a single GPU ) . This means that any efficient search algorithm must be judicious when selecting architectures to evaluate . Learning a good embedding space over the domain of compressed network architectures is important because it can be used to define a distribution on the architecture space that can be used to generate a priority ordering of architectures for evaluation . To enable the careful selection of architectures for evaluation , we propose a method to incrementally learn an embedding space over the domain of network architectures . In the network compression paradigm , we are given a teacher network and we aim to search for a compressed network architecture , a student network that contains as few parameters as possible while maintaining similar performance to the teacher network . We address the task of compressed architecture search by using Bayesian Optimization ( BO ) with a kernel function defined over our proposed embedding space to select architectures for evaluation . As modern neural architectures can ∗indicates equal contribution . have multiple layers , multiple branches and multiple skip connections , defining an embedding space over all architectures is non-trivial . In this work , we propose a method for mapping a diverse range of discrete architectures to a continuous embedding space through the use of recurrent neural networks . The learned embedding space allows us to perform BO to efficiently search for compressed student architectures that are also expected to have high accuracy . We demonstrate that our search algorithm can significantly outperform various baseline methods , such as random search and reinforcement learning ( Ashok et al. , 2018 ) . For example , our search algorithm can compress VGG-19 ( Simonyan & Zisserman , 2014 ) by 8× on CIFAR-100 ( Krizhevsky & Hinton , 2009 ) while maintaining accuracy on par with the teacher network . The automatically found compressed architectures can also achieve higher accuracy than the state-of-the-art manuallydesigned compact architecture ShuffleNet ( Zhang et al. , 2018 ) with a similar size . We also demonstrate that the learned embedding space can be transferred to new settings for architecture search , such as a larger teacher network or a teacher network in a different architecture family , without any training . Contributions : ( 1 ) We propose a novel method to incrementally learn an embedding space over the domain of network architectures . Based on the learnable embedding space , we present a framework of searching for compressed network architectures with BO . The learned embedding provides a feature space over which the kernel function of BO is defined . ( 2 ) We propose a set of architecture operators for generating architectures for search . Operators for modifying the teacher network are : layer removal , layer shrinkage and skip connection addition . ( 3 ) We propose a multiple kernel strategy to prevent the premature convergence of the search and encourage the search algorithm to explore more diverse architectures during the search process . 2 RELATED WORK . Computationally Efficient Architecture : There has been great progress in designing computationally efficient network architectures . Representative examples include SqueezeNet ( Iandola et al. , 2016 ) , MobileNet ( Howard et al. , 2017 ) , MobileNetV2 ( Sandler et al. , 2018 ) , CondenseNet ( Huang et al. , 2018 ) and ShuffleNet ( Zhang et al. , 2018 ) . Different from them , we aim to develop an algorithm that can automatically search for an efficient network architecture with minimal human efforts involved in the architecture design . Neural Architecture Search ( NAS ) : NAS has recently been an active research topic ( Zoph & Le , 2016 ; Zoph et al. , 2017 ; Real et al. , 2018 ; Pham et al. , 2018 ; Liu et al. , 2017a ; b ; 2018 ; Luo et al. , 2018 ) . Some existing works in NAS are focused on searching for architectures that not only can achieve high performance but also respect some resource or computation constraints ( Ashok et al. , 2018 ; Tan et al. , 2018 ; Zhou et al. , 2018 ; Dong et al. , 2018 ; Hsu et al. , 2018 ; Elsken et al. , 2018a ) . NAO ( Luo et al. , 2018 ) and our work share the idea of mapping network architectures into a latent continuous embedding space . But NAO and our work have fundamentally different motivations , which further lead to different architecture search frameworks . NAO maps network architectures to a continuous space such that they can perform gradient based optimization to find better architectures . However , our motivation for learning the embedding space is to find a principled way to define a kernel function between architectures with complex skip connections and multiple branches . Our work is also closely related to N2N ( Ashok et al. , 2018 ) , which searches for a compressed architecture based on a given teacher network using reinforcement learning . Our search algorithm is developed based on Bayesian Optimization ( BO ) , which is different from N2N and many other existing works . We will compare our approach to other BO based NAS methods in the next paragraph . Readers can refer to Elsken et al . ( 2018b ) for a more complete literature review of NAS . Bayesian Optimization ( BO ) : BO is a popular method for hyper-parameter optimization in machine learning . BO has been used to tune the number of layers and the size of hidden layers ( Bergstra et al. , 2011 ; Swersky et al. , 2014 ) , the width of a network ( Snoek et al. , 2012 ) or the size of the filter bank ( Bergstra et al. , 2013 ) , along with other hyper-parameters , such as the learning rate , number of iterations . Mendoza et al . ( 2016 ) , Jenatton et al . ( 2017 ) and Zela et al . ( 2018 ) also fall into this category . Our work is also related to Hernández-Lobato et al . ( 2016 ) , which presents a Bayesian method for identifying the Pareto set of multi-objective optimization problems and applies the method to searching for a fast and accurate neural network . However , most existing works on BO for NAS only show results on tuning network architectures where the connections between network layers are fixed , i.e. , most of them do not optimize how the layers are connected to each other . Kandasamy et al . ( 2018 ) proposes a distance metric OTMANN to compare network architectures with complex skip connections and branch layers , based on which NASBOT is developed , a BO based NAS framework , which can tune how the layers are connected . Although the OTMANN distance is designed with thoughtful choices , it is defined based on some empirically identified factors that can influence the performance of a network , rather than the actual performance of networks . Different from OTMANN , the distance metric ( or the embedding ) for network architectures in our algorithm is automatically learned according to the actual performance of network architectures instead of manually designed . Our work can also be viewed as carrying out optimization in the latent space of a high dimensional and structured space , which shares a similar idea with previous literature ( Lu et al. , 2018 ; GómezBombarelli et al. , 2018 ) . For example , Lu et al . ( 2018 ) presents a new variational auto-encoder to map kernel combinations produced by a context-free grammar into a continuous latent space . Deep Kernel Learning : Our work is also related to recent works on deep kernel learning ( Wilson et al. , 2016a ; b ) . They aim to learn more expressive kernels by representing the kernel function as a neural network to incorporate the expressive power of deep networks . The follow-up work ( AlShedivat et al. , 2017 ) extends the kernel representation to recurrent networks to model sequential data . Our work shares a similar motivation with them and tries to learn a kernel function for the neural architecture domain by leveraging the expressive power of deep networks . 3 APPROACH . In this work , we focus on searching for a compressed network architecture based on a given teacher network and our goal is to find a network architecture which contains as few parameters as possible but still can obtain a similar performance to the teacher network . Formally , we aim to solve the following optimization problem : x∗ = argmax x∈X f ( x ) , ( 1 ) where X denotes the domain of neural architectures and the function f ( x ) : X 7→ R evaluates how well the architecture x meets our requirement . We adopt the reward function design in N2N ( Ashok et al. , 2018 ) for the function f , which is defined based on the compression ratio of the architecture x and its validation performance after being trained on the training set . More details about the exact form of f are given in Appendix 6.1 due to space constraints . As evaluating the value of f ( x ) for a specific architecture x is extremely costly , the algorithm must judiciously select the architectures to evaluate . To enable the careful selection of architectures for evaluation , we propose a method to incrementally learn an embedding space over the domain of network architecture that can be used to generate a priority ordering of architectures for evaluation . In particular , we develop the search algorithm based on BO with a kernel function defined over our proposed embedding space . In the following text , we will first introduce the sketch of the BO algorithm and then explain how the proposed embedding space is used in the loop of BO . We adopt the Gaussian process ( GP ) based BO algorithms to maximize the function f ( x ) , which is one of the most popular algorithms in BO . A GP prior is placed on the function f , parameterized by a mean function µ ( · ) : X 7→ R and a covariance function or kernel k ( · , · ) : X × X 7→ R. To search for the solution , we start from an arbitrarily selected architecture x1 . At step t , we evaluate the architecture xt , i.e. , obtaining the value of f ( xt ) . Using the t evaluated architectures up to now , we compute the posterior distribution on the function f : p ( f ( x ) | f ( x1 : t ) ) ∼ N ( µt ( x ) , σ2t ( x ) ) , ( 2 ) where f ( x1 : t ) = [ f ( x1 ) , . . . , f ( xt ) ] and µt ( x ) and σ2t ( x ) can be computed analytically based on the GP prior ( Williams & Rasmussen , 2006 ) . We then use the posterior distribution to decide the next architecture to evaluate . In particular , we obtain xt+1 by maximizing the expected improvement acquisition function EIt ( x ) : X 7→ R , i.e. , xt+1 = argmaxx∈X EIt ( x ) . The expected improvement function EIt ( x ) ( Mockus & Mockus , 1991 ) measures the expected improvement over the current maximum value according to the posterior distribution : EIt ( x ) = Et [ max ( 0 , f ( x ) − f∗t ) ] , ( 3 ) where Et indicates the expectation is taken with respect to the posterior distribution at step t p ( f ( x ) | f ( x1 : t ) ) and f∗t is the maximum value among { f ( x1 ) , . . . , f ( xt ) } . Once obtaining xt+1 , we repeat the above described process until we reach the maximum number of steps . Finally , we return the best evaluated architecture as the solution . The main difficulty in realizing the above optimization procedure is the design of the kernel function k ( · , · ) for X and the maximization of the acquisition function EIt ( x ) over X , since the neural architecture domain X is discrete and highly complex . To overcome these difficulties , we propose to learn an embedding space for the neural architecture domain and define the kernel function based on the learned embedding space . We also propose a search space , a subset of the neural architecture domain , over which maximizing the acquisition function is feasible and sufficient .

Summary: In this work, the authors propose a new strategy to compress a teacher neural network. Briefly, the authors propose using Bayesian optimization (BO) where the accuracy of the networks is modelled using a Gaussian Process function with a squared exponential kernel on continuous neural network (NN) embeddings. Such embeddings are the output of a bidirectional LSTM taking as input the “raw” (discrete) NN representations (when regarded as a covariance function of the “raw” (discrete) NN representations, the kernel is a deep kernel).