DyNet: Dynamic Convolution for Accelerating Convolution Neural Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have achieved state-of-the-art performance in many computer vision tasks ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2013 ) , and the neural architectures of CNNs are evolving over the years ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Hu et al. , 2018 ; Zhong et al. , 2018a ; b ) . However , modern high-performance CNNs often require a lot of computation resources to execute large amount of convolution kernel operations . Aside from the accuracy , to make CNNs applicable on mobile devices , building lightweight and efficient deep models has attracting much more attention recently ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Zhang et al. , 2018 ; Ma et al. , 2018 ) . These methods can be roughly categorized into two types : efficient network design and model compression . Representative methods for the former category are MobileNet ( Howard et al. , 2017 ; Sandler et al. , 2018 ) and ShuffleNet ( Ma et al. , 2018 ; Zhang et al. , 2018 ) , which use depth-wise separable convolution and channel-level shuffle techniques to reduce computation cost . On the other hand , model compression based methods tend to obtain a smaller network by compressing a larger network via pruning , factorization or mimic ( Chen et al. , 2015 ; Han et al. , 2015a ; Jaderberg et al. , 2014 ; Lebedev et al. , 2014 ; Ba & Caruana , 2014 ) . Although some handcrafted efficient network structures have been designed , we observe that the significant correlations still exist among convolutional kernels , and introduce large amount of redundant calculations . Moreover , these small networks are hard to compress . For example , Liu et al . ( 2019 ) compress MobileNetV2 to 124M , but the accuracy drops by 5.4 % on ImageNet . We theoretically analyze above observation , and find that this phenomenon is caused by the nature of static convolution , where correlated kernels are cooperated to extract noise-irrelevant features . Thus it is hard to compress the fixed convolution kernels without information loss . We also find that if we linearly fuse several convolution kernels to generate one dynamic kernel based on the input , we can obtain the noise-irrelevant features without the cooperation of multiple kernels , and further reduce the computation cost of convolution layer remarkably . Based on above observations and analysis , in this paper , we propose a novel dynamic convolution method named DyNet . The overall framework of DyNet is shown in Figure 1 , which consists of a coefficient prediction module and a dynamic generation module . The coefficient prediction module is trainable and designed to predict the coefficients of fixed convolution kernels . Then the dynamic generation module further generates a dynamic kernel based on the predicted coefficients . Our proposed dynamic convolution method is simple to implement , and can be used as a drop-in plugin for any convolution layer to reduce computation cost . We evaluate the proposed DyNet on state-of-the-art networks such as MobileNetV2 , ShuffleNetV2 and ResNets . Experiment results show that DyNet reduces 37.0 % FLOPs of ShuffleNetV2 ( 1.0 ) while further improve the Top-1 accuracy on ImageNet by 1.0 % . For MobileNetV2 ( 1.0 ) , ResNet18 and ResNet50 , DyNet reduces 54.7 % , 67.2 % and 71.3 % FLOPs respectively , the Top-1 accuracy on ImageNet changes by −0.27 % , −0.6 % and −0.08 % . Meanwhile , DyNet further accelerates the inference speed of MobileNetV2 ( 1.0 ) , ResNet18 and ResNet50 by 1.87×,1.32×and 1.48× on CPU platform respectively . 2 RELATED WORK . We review related works from three aspects : efficient convolution neural network design , model compression and dynamic convolutional kernels . 2.1 EFFICIENT CONVOLUTION NEURAL NETWORK DESIGN . In many computer vision tasks ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2013 ) , model design plays a key role . The increasing demands of high quality networks on mobile/embedding devices have driven the study on efficient network design ( He & Sun , 2015 ) . For example , GoogleNet ( Szegedy et al. , 2015 ) increases the depth of networks with lower complexity compared to simply stacking convolution layers ; SqueezeNet ( Iandola et al. , 2016 ) deploys a bottleneck approach to design a very small network ; Xception ( Chollet , 2017 ) , MobileNet ( Howard et al. , 2017 ) and MobileNetV2 ( Sandler et al. , 2018 ) use depth-wise separable convolution to reduce computation and model size . ShuffleNet ( Zhang et al. , 2018 ) and ShuffleNetV2 ( Ma et al. , 2018 ) shuffle channels to reduce computation of 1× 1 convolution kernel and improve accuracy . Despite the progress made by these efforts , we find that there still exists redundancy between convolution kernels and cause redundant computation . 2.2 MODEL COMPRESSION . Another trend to obtaining small network is model compression . Factorization based methods ( Jaderberg et al. , 2014 ; Lebedev et al. , 2014 ) try to speed up convolution operation by using tensor decomposition to approximate original convolution operation . Knowledge distillation based methods ( Ba & Caruana , 2014 ; Romero et al. , 2014 ; Hinton et al. , 2015 ) learn a small network to mimic a larger teacher network . Pruning based methods ( Han et al. , 2015a ; b ; Wen et al. , 2016 ; Liu et al. , 2019 ) try to reduce computation by pruning the redundant connections or convolution channels . Compared with those methods , DyNet is more effective especially when the target network is already efficient enough . For example , in ( Liu et al. , 2019 ) , they get a smaller model of 124M FLOPs by pruning the MobileNetV2 , however it drops the accuracy by 5.4 % on ImageNet compared with the model with 291M FLOPs . While in DyNet , we can reduce the FLOPs of MobileNetV2 ( 1.0 ) from 298M to 129M with the accuracy drops only 0.27 % . 2.3 DYNAMIC CONVOLUTION KERNEL . Generating dynamic convolution kernels appears in both computer vision and natural language processing ( NLP ) tasks . In computer vision domain , Klein et al . ( Klein et al. , 2015 ) and Brabandere et al . ( Jia et al. , 2016 ) directly generate convolution kernels via a linear layer based on the feature maps of previous layers . Because convolution kernels has a large amount of parameters , the linear layer will be inefficient on the hardware . Our proposed method solves this problem via merely predicting the coefficients for linearly combining static kernels and achieve real speed up for CNN on hardware . The idea of linearly combining static kernels using predicted coefficients has been proposed by Yang et al . ( Yang et al. , 2019 ) , but they focus on using more parameters to make models more expressive while we focus on reducing redundant calculations in convolution . We make theoretical analysis and conduct correlation experiment to prove that correlations among convolutional kernels can be reduced via dynamically fusing several kernels . In NLP domain , some works ( Shen et al. , 2018 ; Wu et al. , 2019 ; Gong et al. , 2018 ) incorporate context information to generate input-aware convolution filters which can be changed according to input sentences with various lengths . These methods also directly generate convolution kernels via a linear layer , etc . Because the size of CNN in NLP is smaller and the dimension of convolution kernel is one , the inefficiency issue for the linear layer is alleviated . Moreover , Wu et al . ( Wu et al. , 2019 ) alleviate this issue utilizing the depthwise convolution and the strategy of sharing weight across layers . These methods are designed to improve the adaptivity and flexibility of language modeling , while our method aims to cut down the redundant computation cost . 3 DYNET : DYNAMIC CONVOLUTION IN CNNS . In this section , we first describe the motivation of DyNet . Then we explain the proposed dynamic convolution in detail . Finally , we illustrate the DyNet based architectures of our proposed Dy-mobile , Dy-shuffle , Dy-ResNet18 , Dy-ResNet50 . 3.1 MOTIVATION . As illustrated in previous works ( Han et al. , 2015a ; b ; Wen et al. , 2016 ; Liu et al. , 2019 ) , convolutional kernels are naturally correlated in deep models . For some of the well known networks , we plot the distribution of Pearson product-moment correlation coefficient between feature maps in Figure 2 . Most existing works try to reduce correlations by compressing . However , efficient and small networks like MobileNets are harder to prune despite the correlation is still significant . We think these correlations are vital for maintaining the performance because they are cooperated to obtain noiseirrelevant features . We take face recognition as an example , where the pose or the illumination is not supposed to change the classification results . Therefore , the feature maps will gradually become noise-irrelevant when they go deeper . Based on the theoretical analysis in appendix A , we find that if we dynamically fuse several kernels , we can get noise-irrelevant feature without the cooperation of redundant kernels . In this paper , we propose dynamic convolution method , which learns the coefficients to fuse multiple kernels into a dynamic one based on image contents . We give more in depth analysis about our motivation in appendix A . 3.2 DYNAMIC CONVOLUTION . The goal of dynamic convolution is to learn a group of kernel coefficients , which fuse multiple fixed kernels to a dynamic one . We demonstrate the overall framework of dynamic convolution in Figure 1 . We first utilize a trainable coefficient prediction module to predict coefficients . Then we further propose a dynamic generation module to fuse fixed kernels to a dynamic one . We will illustrate the coefficient prediction module and dynamic generation module in detail in the following of this section . Coefficient prediction module Coefficient prediction module is proposed to predict coefficients based on image contents . As shown in Figure 3 , the coefficient prediction module can be composed by a global average pooling layer and a fully connected layer with Sigmoid as activation function . Global average pooling layer aggregates the input feature maps into a 1 × 1 × Cin vector , which serves as a feature extraction layer . Then the fully connected layer further maps the feature into a 1× 1×C vector , which are the coefficients for fixed convolution kernels of several dynamic convolution layers . Dynamic generation module For a dynamic convolution layer with weight [ Cout × gt , Cin , k , k ] , it corresponds with Cout × gt fixed kernels and Cout dynamic kernels , the shape of each kernel is [ Cin , k , k ] . gt denotes the group size , it is a hyperparameter . We denote the fixed kernels as wit , the dynamic kernels as w̃t , the coefficients as η i t , where t = 0 , ... , Cout , i = 0 , ... , gt . After the coefficients are obtained , we generate dynamic kernels as follows : w̃t = gt∑ i=1 ηit · wit ( 1 ) Training algorithm For the training of the proposed dynamic convolution , it is not suitable to use batch based training scheme . It is because the convolution kernel is different for different input images in the same mini-batch . Therefore , we fuse feature maps based on the coefficients rather than kernels during training . They are mathematically equivalent as shown in Eq . 2 : Õt = w̃t ⊗ x = gt∑ i=1 ( ηit · wit ) ⊗ x = gt∑ i=1 ( ηit · wit ⊗ x ) = gt∑ i=1 ( ηit · ( wit ⊗ x ) ) = gt∑ i=1 ( ηit ·Oit ) , ( 2 ) where x denotes the input , Õt denotes the output of dynamic kernel w̃t , Oit denotes the output of fixed kernel wit .

Summary: The authors propose to use dynamic convolutional kernels as a means to reduce the computation cost in static CNNs while maintaining their performance. The dynamic kernels are obtained by a linear combination of static kernels where the weights of the linear combination are input-dependent (they are obtained similarly to the coefficients in squeeze-and-excite). The authors also include a theoretical and experimental study of the correlation.