Neural Epitome Search for Architecture-Agnostic Network Compression

Traditional compression methods including network pruning , quantization , low rank factorization and knowledge distillation all assume that network architectures and parameters are one-to-one mapped . In this work , we propose a new perspective on network compression , i.e. , network parameters can be disentangled from the architectures . From this viewpoint , we present the Neural Epitome Search ( NES ) , a new neural network compression approach that learns to find compact yet expressive epitomes for weight parameters of a specified network architecture end-to-end . The complete network to compress can be generated from the learned epitome via a novel transformation method that adaptively transforms the epitomes to match weight shapes of the given architecture . Compared with existing compression methods , NES allows the weight tensors to be independent of the architecture design and hence can achieve a good trade-off between model compression rate and performance given a specific model size constraint . Experiments demonstrate that , on ImageNet , when taking MobileNetV2 as backbone , our approach improves the full-model baseline by 1.47 % in top-1 accuracy with 25 % MAdd reduction , and with the same compression ratio , improves AutoML for Model Compression ( AMC ) by 2.5 % in top-1 accuracy . Moreover , taking EfficientNet-B0 as baseline , our NES yields an improvement of 1.2 % but has 10 % less MAdd . In particular , our method achieves a new state-of-the-art results of 77.5 % under mobile settings ( < 350M MAdd ) . Code can be found at https : //github.com/zhoudaquan/NES . 1 INTRODUCTION . Despite the remarkable performance achieved in many applications , powerful deep convolutional neural networks ( CNNs ) typically suffer from high complexity ( Han et al. , 2015a ) . The large model size and computation cost hinders their deployment on resource limited devices , such as mobile phones . Very recently , huge efforts have been made to compress powerful CNNs . Existing compression techniques can be generally categorized into four categories : network pruning ( Han et al. , 2015a ; Collins & Kohli , 2014 ; Han et al. , 2015b ) , low rank factorization ( Jaderberg et al. , 2014 ) , quantization ( Jacob et al. , 2018 ; Hubara et al. , 2017 ; Rastegari et al. , 2016 ) , and knowledge distillation ( Hinton et al. , 2015 ; Papernot et al. , 2016 ) . Network pruning targets on removing unimportant connections or weights to reduce the number of parameters and multiply-adds ( MAdd ) . Low rank factorization decomposes an existing layer into lower-rank and smaller layers to reduce the computation cost . Weights quantization aims to use less number of bits to store the weights and activation maps . Knowledge distillation uses a well trained teacher network to train a lightweight student network . All of those compression methods assume that the model parameters ( weight tensors ) must have one-to-one correspondence to the architectures . As a result , they suffer from performance drop since changing architectures will inevitably lead to loss of informative parameters . ∗correspondence In this paper , we consider the network compression problem from a new perspective where the shape of the weight tensors and the architecture are designed independently . The key insight is that the network parameters can be disentangled from the architecture and can be compactly represented by a small-sized parameter set ( called epitome ) , inspired by success of epitome methods in image/video modeling and data sparse coding ( Jojic et al. , 2003 ; Cheung et al. , 2008 ; Aharon & Elad , 2008 ) . As shown in Figure 1 , unlike conventional convolutional layers that use the architecture tied weight tensors to convolve with the input feature map , our proposed neural epitome search ( NES ) approach first learns a compact yet expressive epitome along with an adaptive transformation function to expand the epitomes . The transformation function is able to generate a variety of parameters from epitomes via a novel learnable transformation function , which also guarantees the representation capacity of the resulting weight tensors to be large . Our transformation function is differentiable and hence enables the NES approach to search for optimal epitome end-to-end , achieving a good trade-off between required model size and performance . In addition , we propose a novel routing map to record the index mapping used for the transformation between the epitome and the convolution kernel . During inference , this routing map enables the model to reuse computations when the expanded weight tensors are formed based on the same set of elements in the epitomes and therefore effectively reduces the computation cost . Benefiting from the learned epitomic network parameters and transformation method , compared to existing compression approaches , NES has less performance drop . To the best of our knowledge , this is the first work that automatically learns compact epitomes of network parameters and the corresponding transformation function for network compression . To sum up , our work offers the following attractive properties : • Our method is flexible . It allows the weight tensors to be independent of the architecture design . We can easily control the model size by defining the size of the epitomes given a specified network architecture . This is especially beneficial in the context of edge devices . • Our method is effective . The learning-based transformation method empowers the epitomes with highly expressive capability and hence incurs less performance drop even with large compression ratios . • Our method is easy to use . It can be encapsulated as a drop in replacement to the current convolutional operator . There is no dependence on specialized platforms/frameworks for NES to conduct compression . To demonstrate the efficacy of the proposed approach , We conduct extensive experiments on CIFAR10 ( Krizhevsky & Hinton , 2009 ) and ImageNet ( Deng et al. , 2009 ) . On CIFAR-10 dataset , our method outperforms the baseline model by 1.3 % . On ImageNet , our method outperforms MobileNetV2 full model by 1.47 % in top-1 accuracy with 25 % MAdd reduction , and MobileNetV2-0.35 baseline by 3.78 % . Regarding MobileNetV2-0.7 backbone , our method improves AMC ( He et al. , 2018 ) by 2.47 % . Additionally , when taking EfficientNet-b0 ( Tan & Le , 2019 ) as baseline , we have an improvement of 1.2 % top-1 accuracy with 10 % MAdd reduction . 2 RELATED WORK . Traditional model compression methods include network pruning ( Collins & Kohli , 2014 ; Han et al. , 2015b ) , low rank factorization ( Jaderberg et al. , 2014 ) , quantization ( Jacob et al. , 2018 ; Hubara et al. , 2017 ; Rastegari et al. , 2016 ) and knowledge distillation ( Hinton et al. , 2015 ; Papernot et al. , 2016 ) . For all of those methods , as mentioned in Section 1 extensive expert knowledge and manual efforts are needed and the process might need to be done iteratively and hence is time consuming . Recently , AutoML based methods have been proposed to reduce the experts efforts for model compression ( He et al. , 2018 ; Zoph et al. , 2018 ; Noy et al. , 2019 ; Li et al. , 2019 ) and efficient convolution architecture design ( Liu et al. , 2018 ; Wu et al. , 2018 ; Tan et al. , 2018 ) . As proposed in AutoML for model compression ( AMC ( He et al. , 2018 ) ) , reinforcement learning can be used as an agent to remove redundant layers by adding resource constraints into the rewards function which however is highly time consuming . Later , gradient based search method such as DARTS ( Liu et al. , 2018 ) is developed for higher search efficiency over basic building blocks . There are also methods that use AutoML based method to search for efficient architectures directly ( Wu et al. , 2018 ; He et al. , 2018 ) . All of those methods are searching for optimized network architecture with an implicit assumption that the weights and the model architecture have one-to-one correspondence . Different from all of the above mentioned methods , our method provides a new search space by separating the model weights from the architecture . The model size can thus be controlled precisely by nature . Our method is also related to the group-theory based network transformation . Based on the group theory proposed in Cohen & Welling ( 2016 ) , recent methods try to design a compact filter to reduce the convolution layer computation cost such as WSNet ( Jin et al. , 2017 ) and CReLU ( Shang et al. , 2016 ) . WSNet tries to reduces the model parameters and computations by allowing overlapping between adjacent filters of 1D convolution kernel . This can be seen as a simplified version of our method as the overlapping can be regarded as a fixed rule transformation . CReLu tries to learn diversified features by concatenating ReLU output of original and negated inputs . However , as the rule is fixed , the design of those schemes are application specific and time consuming . Besides , the performance typically suffers since the scheme is not optimized during the training . In contrast , our method requires negligible human efforts and the transformation rule is learned end-to-end . 3 METHOD . 3.1 OVERVIEW . A convolutional layer is composed of a set of learnable weights ( or kernels ) that are used for feature transformation . The observation of this paper is that the learnable weights in CNNs can be disentangled from the architecture . Inspired by this fact , we provide a new perspective on the network compression problem , i.e. , finding a compact yet expressive parameter set , called epitome , along with a proper transformation function to fully represent the whole network , as illustrated in Figure 1 . Formally , consider a convolutional network with a fixed architecture consisting of a stack of convolutional layers , each of which is associated with a weight tensor θi ( i is layer index ) . Further , let L ( X , Y ; θ ) be the loss function used to train the network , where X and Y are the input data and label respectively and θ = { θi } denotes the parameter set in the network . Our goal is to learn epitomes E = { Ei } which have smaller sizes than θ and the transformation function τ = { τi } to represent the network parameter θ with the compact epitome as τ ( E ) . In this way , network compression is achieved . The objective function of neural epitome search ( NES ) for a given architecture is to learn optimal epitomes E∗ and transformation functions τ∗ : { E∗ , τ∗ } = argmin E , τ L ( X , Y ; τ ( E ) ) , s.t . |E| < |θ| , ( 1 ) where | · | calculates the number of all elements . The above NES approach provides a flexible way to achieve network compression since the epitomes can be defined to be of any size . By learning a proper transformation function , the epitomes of predefined sizes can be adaptively transformed to match and instantiate the specified network architecture . In the following sections , we will elaborate on how to learn the transformation functions τ ( · ) , the epitome ( E ∈ RWE×HE×CEin×CEout ) and how to compress models via NES in an end-to-end manner . In this paper , we use “ a sub-tensor in the epitome ” to describe a patch of the epitome that will be selected to construct the convolution weight tensor . The sub-tensor , Es , is represented with the starting index and the length along each dimension in the epitome as shown below : Es = E [ p : p+ w , q : q + h , cin : cin + β1 , cout : cout + β2 ] , ( 2 ) where ( p , q , cin , cout ) denote the starting index of the sub-tensor and ( w , h , β1 , β2 ) denotes the length of the sub-tensor along each dimension .

Summary: This paper focuses on the problem of  neural network compression, and proposes a new scheme, the neural epitome search. It learns to find compact yet expressive epitomes for weight parameters of a specified network architecture. The learned weight tensors are independent of the architecture design.  It can be encapsulated as a drop in replacement to the current