Optimized Separable Convolution: Yet Another Efficient Convolution Operator

The convolution operation is the most critical component in recent surge of deep learning research . Conventional 2D convolution needs O ( C2K2 ) parameters to represent , where C is the channel size and K is the kernel size . The amount of parameters has become really costly considering that these parameters increased tremendously recently to meet the needs of demanding applications . Among various implementations of the convolution , separable convolution has been proven to be more efficient in reducing the model size . For example , depth separable convolution reduces the complexity to O ( C · ( C + K2 ) ) while spatial separable convolution reduces the complexity to O ( C2K ) . However , these are considered ad hoc designs which can not ensure that they can in general achieve optimal separation . In this research , we propose a novel and principled operator called optimized separable convolution by optimal design for the internal number of groups and kernel sizes for general separable convolutions can achieve the complexity of O ( C 3 2K ) . When the restriction in the number of separated convolutions can be lifted , an even lower complexity at O ( C · log ( CK2 ) ) can be achieved . Experimental results demonstrate that the proposed optimized separable convolution is able to achieve an improved performance in terms of accuracy- # Params trade-offs over both conventional , depth-wise , and depth/spatial separable convolutions . 1 INTRODUCTION . Tremendous progresses have been made in recent years towards more accurate image analysis tasks , such as image classification , with deep convolutional neural networks ( DCNNs ) ( Krizhevsky et al. , 2012 ; Srivastava et al. , 2015 ; He et al. , 2016 ; Real et al. , 2019 ; Tan & Le , 2019 ; Dai et al. , 2020 ) . However , the complexity of state-of-the-art DCNN models has also become increasingly high . This can significantly deter their deployment to real-world applications , such as mobile platforms and robotics , where the resources and networks are highly constrained ( Howard et al. , 2017 ; Dai et al. , 2020 ) . The most resource-consuming building block of a DCNN is the convolutional layer . There have been many previous works aiming at reducing the amount of parameters in the convolutional layer . Network pruning ( Han et al. , 2015 ) strategies are developed to reduce redundant parameters that are not sensitive to performances . Quantization and binarization ( Gong et al. , 2014 ; Courbariaux et al. , 2016 ) techniques are introduced to compress the original network by reducing the number of bits required to represent each parameter . Low-rank factorization methods ( Jaderberg et al. , 2014 ; Ioannou et al. , 2015 ) are designed to approximate the original weights using matrix decomposition . Knowledge distillation ( Hinton et al. , 2015 ) is applied to train a compact network with distilled knowledge from a large ensemble model . However , all these existing methods start from a pre-trained model . Besides , they mainly focus on network compression and have limited or no improvements in terms of network acceleration . In this research , we study how to design a separable convolution to achieve an optimal implementation in terms of model size ( representational complexity ) . Enabling convolution to be separable has been proven to be an efficient way to reduce the representational complexity ( Sifre & Mallat , 2014 ; Howard et al. , 2017 ; Szegedy et al. , 2016 ) . Comparing to the network compression related approaches , a well-designed separable convolution shall be more efficient in both storage and computation and shall not require a pre-trained model to begin with . In the DCNN research , the two most well-known separable convolutions are depth separable ( Sifre & Mallat , 2014 ) and spatial separable ( Szegedy et al. , 2016 ) convolutions . Both are able to reduce the complexity of a convolution . The representational complexity of a conventional 2D convolution is quadratic with two hyper-parameters : number of channels ( C ) and kernel size ( K ) , and its representational complexity is actually O ( C2K2 ) . Depth separable convolution is constructed as a depth-wise convolution followed by a point-wise convolution , where depth-wise convolution is a group convolution with its number of groups g = C and point-wise convolution is a 1 × 1 convolution . Spatial separable convolution replaces a K × K kernel with a K × 1 and a 1 × K kernel . Different types of convolutions and their complexities are summarized in Table 1 . From this table , we can see that , for all convolutions , their computational complexities equal to the corresponding representational complexity times a constant . We can also verify that depth separable convolution has a complexity ofO ( C · ( C+K2 ) ) and spatial separable convolution has a complexity ofO ( C2K ) . Both depth and spatial separable convolutions follow an ad hoc design mode and are non-principled . They are able to reduce the complexity to some degree but normally can not achieve an optimal separation . A separable convolution in general has three sets of hyperparameters : the internal number of groups , channel size , and kernel size of each separated convolution . Instead of setting these hyperparameters in an ad hoc ( manual ) fashion , we design a novel and principled ( auto ) scheme to achieve an optimal separation . The resulting separable convolution is called optimized separable convolution in this research . The proposed scheme in general performs better than the other convolution operator counterparts and it also enriches the separable convolution family . To prevent the proposed optimized separable convolution from being degenerated , we assume that the internal channel size is in an order of O ( C ) and propose the following volumetric receptive field condition . As illustrated in Fig . 1a , similar to the receptive field ( RF ) of a convolution which is defined as the region in the input space that a particular CNN ’ s feature is looking at ( or affected by ) ( Lindeberg , 2013 ) , we define the volumetric RF of a convolution to be the volume in the input space that affects CNN ’ s output . The volumetric RF condition requires a properly decomposed separable convolution to maintain the same volumetric RF as the original convolution before decomposition . Hence , the proposed optimized separable convolution will be equivalent to optimizing the internal number of groups and kernel sizes to achieve the target objective ( measured in # Params ) while satisfying the proposed volumetric RF condition . Formally , the objective function is defined by Equation ( 2 ) under the constraints defined by Equations ( 3 ) - ( 6 ) . The solution to this optimization problem will be elaborated in Section 2 . We shall show that the proposed optimized separable convolution can be represented with the order of O ( C 3 2K ) . This is at least a factor of √ C more efficient than the depth and spatial separable convolutions . The proposed optimized separable convolution is able to be generalized into an N - separable case , where the number of separated convolutions N can be optimized further . In such a generalized case , an even lower complexity at O ( C · log ( CK2 ) ) may be achieved . Extensive experiments have been carried out to demonstrate the effectiveness of the proposed optimized separable convolution over other alternatives , including conventional , depth-wise , depth and spatial separable convolutions ( Fig . 3 ( c ) and Fig . 4 ( c ) ) . As further illustrated in Fig . 3 and Fig . 4 , on the CIFAR10 and CIFAR100 datasets ( Krizhevsky et al. , 2009 ) , the proposed optimized separable convolution achieves a better Pareto-frontier1 than both conventional and depth separable convolutions using the ResNet ( He et al. , 2016 ) architecture . To demonstrate that the proposed optimized separable convolution generalizes well to other DCNN architectures , we adopt the DARTS ( Liu et al. , 2018 ) architecture by replacing the depth separable convolution with the proposed optimized separable convolution . The accuracy is improved from 97.24 % to 97.67 % with reduced representational complexity . On the ImageNet dataset ( Deng et al. , 2009 ) , the proposed optimized separable convolution also achieves improved performance . For the DARTS architecture , the proposed approach achieves 74.2 % top1 accuracy with only 4.5 million parameters . For MobileNet , the proposed approach achieves 71.1 % top1 accuracy with only 3.0 million parameters . 2 THE PROPOSED APPROACH . 2.1 CONVOLUTION AND ITS COMPLEXITY . A convolutional layer takes an input tensor Bl−1 of shape ( Cl−1 , Hl−1 , Wl−1 ) and produces an output tensor Bl of shape ( Cl , Hl , Wl ) , where C∗ , H∗ , W∗ are input and output channels , feature heights and widths . The convolutional layer is parameterized with a convolutional kernel of shape ( Cl , Cl−1 , K H l , K W l ) , where K ∗ l are the kernel sizes , and the superscript indicates whether it is aligned with the features in height or width . In this research , we take C∗ = O ( C ) , H∗ = O ( H ) , W∗ = O ( W ) , and K H|W ∗ = O ( K ) for complexity analysis . Formally , we have Bl ( cl , hl , wl ) = ∑ cl−1 ∑ kHl ∑ kWl Bl−1 ( cl−1 , hl−1 , wl−1 ) · Fl ( cl , cl−1 , kHl , kWl ) , ( 1 ) where hl = hl−1 + kHl and wl = wl−1 + k W l . Hence , the number of parameters for convolution is ClCl−1K H l K W l and its representational complexity isO ( C 2K2 ) . The number of FLOPs ( multiplyadds ) for convolution isClHlWl ·Cl−1KHl KWl and its computational complexity isO ( C2K2HW ) . For a group convolution , we have g convolutions with kernels of shape ( Cl/g , Cl−1/g , KHl , K W l ) . Hence , it has O ( C2K2/g ) parameters and O ( C2K2HW/g ) FLOPs , where g is the number of groups . A depth-wise convolution is equivalent to a group convolution with g = C∗ = C. A pointwise convolution is a 1×1 convolution . A depth separable convolution is composed of a depth-wise convolution and a point-wise convolution . A spatial separable convolution replaces a K ×K kernel with K × 1 and 1 ×K kernels . Different types of convolutions are summarized in Table 1 . From this table , their number of parameters and FLOPs can be easily verified . It can also be seen that , for a convolution , its representational complexity is equivalent to its computational complexity for up to a constant ( HW ) . 2.2 RETHINKING CONVOLUTION AND THE VOLUMETRIC RECEPTIVE FIELD CONDITION . Separable convolution has been proven to be efficient in reducing the representational demand in convolution . However , existing approaches including both depth and spatial separable convolutions follow an ad hoc design and are non-principled . They are able to reduce the complexity to some extent but will not normally achieve an optimal separation . In this research , we shall design an 1In multi-objective optimization , a Pareto-frontier is the set of parameterizations ( allocations ) that are all Pareto-optimal . An allocation is Pareto-optimal if there is no alternative allocation where improvement can be made to one participant ’ s well-being without sacrificing any other ’ s . Here , Pareto-frontier represents the curve of the accuracies we are able to achieve for different # Params ( or FLOPs ) . efficient convolution operator capable of achieving the representational objective by optimal design of its internal hyper-parameters . The resulting operator is called optimized separable convolution . The proposed optimized separable convolution is called principled as it optimizes the representational complexity under the following volumetric receptive field condition . As illustrated in Fig . 1a , the receptive field ( RF ) of a convolution is defined to be the region in the input space that a particular CNN ’ s feature is affected by ( Lindeberg , 2013 ) . We define the channel RF to be the channels that affect CNN ’ s output and define the volumetric RF to be the Cartesian product of the RF and channel RF of this convolution . The volumetric RF of a convolution actually represents the volume in the input space that affects CNN ’ s output . The volumetric RF condition requires that a properly decomposed separable convolution at least maintains the same volumetric RF as the original convolution before decomposition . Hence , the proposed optimized separable convolution will be equivalent to optimizing its internal parameters while satisfying the volumetric RF condition . Formally , we shall have the objective function defined by Equation ( 2 ) and the volumetric RF constraints defined by Equations ( 3 ) - ( 6 ) . The volumetric RF of a convolution needs to be maintained for technical , conceptual , and experimental reasons . Technically , if we do not pose any restriction to a separable convolution , optimizing the representational complexity will resulting in a separable convolution being equivalent to a degenerated channel scaling operator2 . The composition of such operators is not meaningful because the composition itself is equivalent to a single channel scaling operator . Conceptually , maintaining the volumetric RF encourages the fusion of channel information , which shall contribute to the good performance of a DCNN . In fact , all modern DCNNs are designed following this rule . Without this channel information exchange , the performance of a DCNN shall be significantly degraded ( depth-wise vs depth separable convolutions in Section 3 ) . Finally , the necessity of maintaining the volumetric RF is experimentally verified . We shall quantize the degree of necessity as overlap coefficient ( γ ) in Section 2.3 and elaborate the experimental results in Section 3 .

Summary: The convolution operator is the fundamental unit of most modern DNNs. This paper summarizes the existing (efficient) convolution operators and formulates an optimization problem to choose a few important parameters for the convolution. The authors show that their convolution under certain constraints requires fewer parameters and operations compared to the commonly used operators. They evaluate their proposed operator on CIFAR10, CIFAR100, and on ImageNet.