Rethinking Convolution: Towards an Optimal Efficiency

In this paper , we present our recent research about the computational efficiency in convolution . Convolution operation is the most critical component in recent surge of deep learning research . Conventional 2D convolution takes O ( C2K2HW ) to calculate , where C is the channel size , K is the kernel size , while H and W are the output height and width . Such computation has become really costly considering that these parameters increased over the past few years to meet the needs of demanding applications . Among various implementations of the convolution , separable convolution has been proven to be more efficient in reducing the computational demand . For example , depth separable convolution reduces the complexity to O ( CHW · ( C + K2 ) ) while spatial separable convolution reduces the complexity to O ( C2KHW ) . However , these are considered an ad hoc design which can not ensure that they can in general achieve optimal separation . In this research , we propose a novel operator called optimal separable convolution which can be calculated at O ( C 3 2KHW ) by optimal design for the internal number of groups and kernel sizes for general separable convolutions . When there is no restriction in the number of separated convolutions , an even lower complexity at O ( CHW · log ( CK2 ) ) can be achieved . Experimental results demonstrate that the proposed optimal separable convolution is able to achieve an improved accuracyFLOPs and accuracy- # Params trade-offs over both conventional and depth/spatial separable convolutions . 1 INTRODUCTION . Tremendous progresses have been made in recent years towards more accurate image analysis tasks , such as image classification , with deep convolutional neural networks ( DCNNs ) ( Krizhevsky et al. , 2012 ; Srivastava et al. , 2015 ; He et al. , 2016 ; Real et al. , 2019 ; Tan & Le , 2019 ; Dai et al. , 2020 ) . However , the computational complexity for state-of-the-art DCNN models has also become increasingly high and computationally expensive . This can significantly defer their deployment to real-world applications , such as mobile platforms and robotics , where the resources are highly constrained ( Howard et al. , 2017 ; Dai et al. , 2020 ) . It is very much desired that a DCNN could achieve better performance with less computation and fewer model parameters . The most time-consuming building block of a DCNN is the convolutional layer . There have been many previous works aiming at reducing the amount of computation in the convolutional layer . Historically , researchers apply Fast Fourier Transform ( FFT ) ( Nussbaumer , 1981 ; Quarteroni et al. , 2010 ) to implement convolution and they gain great speed up for large convolutional kernels . For small convolutional kernels , a direct application is often still cheaper ( Podlozhnyuk , 2007 ) . Researchers also explore low rank approximation ( Jaderberg et al. , 2014 ; Ioannou et al. , 2015 ) to implement convolutions . However , most of the existing methods start from a pre-trained model and mainly focus on network pruning and compression . In this research , we study how to design a separable convolution to achieve an optimal implementation in terms of computational complexity . Enabling convolution separable has been proven to be an efficient way to reduce the computational complexity ( Sifre & Mallat , 2014 ; Howard et al. , 2017 ; Szegedy et al. , 2016 ) . Comparing to the FFT and low rank approximation approaches , a welldesigned separable convolution shall be efficient for both small and large kernel sizes and shall not require a pre-trained model to operate on . In the DCNN research , two most well-known separable convolutions are depth separable ( Sifre & Mallat , 2014 ) and spatial separable ( Szegedy et al. , 2016 ) convolutions . Both are able to reduce the computational complexity of a convolution . The complexity of a conventional 2D convolution is quadratic with three hyper-parameters : number of channels ( C ) , kernel size ( K ) , and spatial dimensions ( H or W ) , and its computational complexity is actually O ( C2K2HW ) . Depth separable convolution is constructed as a depth-wise convolution followed by a point-wise convolution , where depth-wise convolution is a group convolution with its number of groups g = C and point-wise convolution is a 1× 1 convolution . Spatial separable convolution replaces a K ×K kernel to K × 1 and 1 ×K kernels . Different types of convolutions and their computational costs are summarized in Table 1 . From this table , we can easily verify that depth separable convolution has a complexity of O ( CHW · ( C +K2 ) ) and spatial separable convolution has a complexity of O ( C2KHW ) . Both depth separable and spatial separable convolutions follow an ad hoc design . They are able to reduce the computational cost to some degree but normally will not achieve an optimal separation . A separable convolution in general has three sets of parameters : the internal number of groups , channel size , and kernel size of each separated convolution . Instead of setting these parameters in an ad hoc fashion , we design a scheme to achieve an optimal separation . The resulting separable convolution is called optimal separable convolution in this research . To prevent the proposed optimal separable convolution from being degenerated , we assume that the internal channel size is in an order of O ( C ) and propose the following volumetric receptive field condition . As illustrated in Fig . 1a , similar to the receptive field ( RF ) of a convolution which is defined as the region in the input space that a particular CNN ’ s feature is looking at ( or affected by ) ( Lindeberg , 2013 ) , we define the volumetric RF of a convolution to be the volume in the input space that affects CNN ’ s output . The volumetric RF condition requires that a properly decomposed separable convolution maintains the same volumetric RF as the original convolution before decomposition . Hence , the proposed optimal separable convolution will be equivalent to optimizing the internal number of groups and kernel sizes to achieve the computational objective ( measured in FLOPs1 ) while satisfying the volumetric RF condition . Formally , the objective function is defined by Equation ( 2 ) under the constraints defined by Equations ( 3 ) - ( 6 ) . The solution to this optimization problem will be described in detail in Section 2 . 1In this research , similar to ( He et al. , 2016 ) , FLOPs are measured in number of multiply-adds . We shall show that the proposed optimal separable convolution can be calculated at the order of O ( C 3 2KHW ) . This is at least a factor of √ C more efficient than the depth separable and spatial separable convolutions . The proposed optimal separable convolution is able to be easily generalized into an N -separable case , where the number of separated convolutions N can be optimized further . In such a generalized case , an even lower complexity at O ( CHW · log ( CK2 ) ) can be achieved . Extensive experiments are carried out to demonstrate the effectiveness of the proposed optimal separable convolution . As illustrated in Fig . 3 , on the CIFAR10 dataset ( Krizhevsky et al. , 2009 ) , the proposed optimal separable convolution achieves a better Pareto-frontier2 than both conventional and depth separable convolutions using the ResNet ( He et al. , 2016 ) architecture . To demonstrate that the proposed optimal separable convolution generalizes well to other DCNN architectures , we adopt the DARTS ( Liu et al. , 2018 ) architecture by replacing the depth separable convolution with the proposed optimal separable convolution . The accuracy is improved from 97.24 % to 97.67 % with fewer parameters . On the ImageNet dataset ( Deng et al. , 2009 ) , the proposed optimal separable convolution also achieves an improved performance . For the DARTS architecture , the proposed approach achieves 74.2 % top1 accuracy with only 4.5 million parameters . 2 THE PROPOSED APPROACH . 2.1 CONVOLUTION AND ITS COMPUTATIONAL COMPLEXITY . A convolutional layer takes an input tensor Bl−1 of shape ( Cl−1 , Hl−1 , Wl−1 ) and produces an output tensor Bl of shape ( Cl , Hl , Wl ) , where C∗ , H∗ , W∗ are input and output channels , feature heights and widths . The convolutional layer is parameterized with a convolutional kernel of shape ( Cl , Cl−1 , K H l , K W l ) , where K ∗ l are the kernel sizes , and the superscript indicates whether it is aligned with the features in height or width . In this research , we take C∗ = O ( C ) , H∗ = O ( H ) , W∗ = O ( W ) , and K H|W ∗ = O ( K ) for complexity analysis . Formally , we have Bl ( cl , hl , wl ) = ∑ cl−1 ∑ kHl ∑ kWl Bl−1 ( cl−1 , hl−1 , wl−1 ) · Fl ( cl , cl−1 , kHl , kWl ) , ( 1 ) where hl = hl−1 + kHl and wl = wl−1 + k W l . Hence , the number of FLOPs ( multiply-adds ) for convolution is ClHlWl · Cl−1KHl KWl = O ( C2K2HW ) and the number of parameters is ClCl−1K H l K W l = O ( C 2K2 ) . For a group convolution , we have g convolutions with kernels of shape ( Cl/g , Cl−1/g , KHl , K W l ) . Hence , it has O ( C2K2HW/g ) FLOPs and O ( C2K2/g ) parameters , where g is the number of groups . A depth-wise convolution is equivalent to a group convolution with g = C∗ = C. A pointwise convolution is a 1×1 convolution . A depth separable convolution is composed of a depth-wise convolution and a point-wise convolution . A spatial separable convolution replaces a K ×K kernel with K × 1 and 1 ×K kernels . Different types of convolutions are summarized in Table 1 . From this table , their FLOPs and number of parameters can be easily verified . 2.2 RETHINKING CONVOLUTION AND THE VOLUMETRIC RECEPTIVE FIELD CONDITION . Separable convolution has been proven to be efficient in reducing the computational demand in convolution . However , existing approaches including both depth separable and spatial separable convolutions follow an ad hoc design . They are able to reduce the computational cost to some extent but will not normally achieve an optimal separation . In this research , we shall design an efficient convolution operator achieving the computational objective by optimal design of its internal hyperparameters . The resulting operator is called optimal separable convolution . One difficulty is that if we do not pose any restriction to a separable convolution , optimizing the FLOPs target will resulting in a separable convolution being equivalent to a degenerated channel scaling operator3 . Hence , we propose the following volumetric receptive field condition . 2In multi-objective optimization , a Pareto-frontier is the set of parameterizations ( allocations ) that are all Pareto-optimal . An allocation is Pareto-optimal if there is no alternative allocation where improvement can be made to one participant ’ s well-being without sacrificing any other ’ s . Here , Pareto-frontier represents the curve of the accuracies we are able to achieve for different FLOPs/ # Params . 3From Table 1 , let g = C and K = 1 , a convolution will have C parameters and CHW FLOPs . This is in fact a channel scaling operator . Composition of such operators is not meaningful because the composition itself is equivalent to a single channel scaling operator . As illustrated in Fig . 1a , the receptive field ( RF ) of a convolution is defined to be the region in the input space that a particular CNN ’ s feature is affected by ( Lindeberg , 2013 ) . We define the channel RF to be the channels that affect CNN ’ s output and define the volumetric RF to be the Cartesian product of the RF and channel RF of this convolution . The volumetric RF of a convolution is actually the volume in the input space that affects CNN ’ s output . The volumetric RF condition requires that a properly decomposed separable convolution maintains ( at least ) the same volumetric RF as the original convolution before decomposition . Hence , the proposed optimal separable convolution will be equivalent to optimizing its internal parameters while satisfying the volumetric RF condition .

Summary: This paper proposes a novel analysis for optimal separable convolution considering the number of parameters and FLOPs. The idea is to constraint the input information consumed stationery throughout the optimization of the parameters for separable convolutions. More specifically, this paper proposes the notion of volumetric receptive field and by holding it constant throughout optimization, one can arrive at a constrained optimization problem for solving the parameters for the optimal separable convolution. Empirical results of replacing common convolution with optimal convolution in various modern CNNs have demonstrated the effectiveness of the proposed optimal separable convolution.