FALCON: Fast and Lightweight Convolution for Compressing and Accelerating CNN

1 INTRODUCTION . How can we efficiently reduce size and energy consumption of Convolutional Neural Networks ( CNN ) while maintaining their accuracy on classification tasks ? Nowadays , CNN is widely used in various areas including computer vision ( Krizhevsky et al . ( 2012 ) ; Simonyan & Zisserman ( 2014 ) ; Szegedy et al . ( 2017 ) ) , natural language processing ( Yin et al . ( 2016 ) ) , recommendation system ( Kim et al . ( 2016a ) ) , etc . In addition , model compression has become an important technique due to an increase in the model capacity and the number of parameters in CNN . One recent and promising direction for compressing CNN is depthwise separable convolution ( Sifre ( 2014 ) ) which replaces standard convolution with depthwise and pointwise convolutions . The depthwise convolution applies a separate 2D convolution kernel for each input channel , and the pointwise convolution changes the channel size using 1×1 convolution ( details in Section 2.1 ) . Several recent methods ( Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Zhang et al . ( 2017 ) ) based on depthwise separable convolution show reasonable performances in terms of compression and computation reduction . However , existing approaches based on depthwise separable convolution have several crucial limitations . First , they are heuristic methods , and their relation to the standard convolution is not clearly identified . Second , due to their heuristic nature , generalizing the methods is difficult . Third , although they give reasonable compression and computation reduction , their accuracy is not sufficient compared to that of standard-convolution-based models . In this paper , we propose FALCON , an accurate and lightweight method for compressing CNN . FALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas . First , we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP ( Extended Hadamard Product ) , which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel . We then design FALCON by fine-tuning and reordering the results of EHP to improve the accuracy of convolution operations . Second , based on the precise definition , we generalize the FALCON to design rank-k FALCON , which further improves accuracy while sacrificing a bit of compression and computation reduction rates . We also propose FALCON-branch by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy . As a result , FALCON and FALCON-branch provide a superior accuracy compared to other methods based on depthwise separable convolution , with similar compression and computation reduction rates , and rank-k FALCON further improves accuracy , outperforming even the original convolution in many cases . Our contributions are summarized as follows : • Generalization . We analyze and generalize depthwise separable convolution to our proposed EHP ( Extended Hadamard Product ) operation . This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution . Furthermore , with fine-tuning operations , it leads to our proposed method FALCON . • Algorithm . We propose FALCON , a CNN compression method based on depthwise separable convolution . FALCON is carefully designed to compress CNN with little accuracy loss . We also propose rank-k FALCON to further improve the accuracy with a little sacrifice in compression and computation reduction rates . FALCON can be easily integrated into other architectures , and we propose FALCON-branch which combines FALCON with a branch architecture for a better performance . We theoretically analyze the compression and computation reduction rates of FALCON and other competitors . • Experiments . We perform extensive experiments and show that FALCON 1 ) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN , and 2 ) provides up to 8× compression and computation reduction compared to the standard convolution while giving similar accuracies . Furthermore , we show that rank-k FALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and floating-point operations . The rest of this paper is organized as follows . Section 2 explains preliminaries . Section 3 describes our proposed method FALCON . Section 4 presents experimental results . After discussing related works in Section 5 , we conclude in Section 6 . 2 PRELIMINARY . We describe preliminaries on depthwise separable convolution and methods based on depthwise separable convolution . Symbols used in this paper are described in Table 4 of Appendix . 2.1 DEPTHWISE SEPARABLE CONVOLUTION . Depthwise Separable Convolution ( DSConv ) consists of two sub-layers : depthwise convolution and pointwise convolution . The architecture of each convolution layer in DSConv is illustrated in Figure 5 ( a ) . Depthwise convolution ( DWConv ) kernel consists of several D×D 2-dimensional filters . The number of 2-dimension filters is the same as that of input feature maps . Each filter is applied on the corresponding input feature map , and produces an output feature map . Pointwise convolution ( PWConv ) , known as 1× 1 convolution , is a standard convolution with kernel size 1 . DSConv is defined as follows : O′h′ , w′ , m = D∑ i=1 D∑ j=1 Di , j , m · Ihi , wj , m ( 1 ) Oh′ , w′ , n = M∑ m=1 Pm , n ·O′h′ , w′ , m ( 2 ) where Di , j , m and Pm , n are depthwise convolution kernel and pointwise convolution kernel , respectively . O′h′ , w′ , m ∈ RH ′×W ′×M denotes intermediate feature maps . DSConv performs DWConv on input feature maps Ihi , wj , m using equation 1 , and generates intermediate feature maps O ′ h′ , w′ , m . Then , DSConv performs PWConv on O′h′ , w′ , m using equation 2 , and generates output feature maps Oh′ , w′ , n . 2.2 METHODS BASED ON DEPTHWISE SEPARABLE CONVOLUTION . Several CNN methods have been proposed based on Depthwise Separable Convolution ( DSConv ) recently . DSConv was first introduced by Sifre ( 2014 ) . Chollet & Franois ( 2016 ) built Xception module using DSConv in a few layers . Howard et al . ( 2017 ) built Mobilenet with all convolution layers replaced by DSConv . Sandler et al . ( 2018 ) built MobileNetV2 with inverted bottleneck block , denoted as MobileConvV2 in this paper . Zhang et al . ( 2017 ) built a CNN model with Shufflenet Unit , denoted as ShuffleUnit in this paper . Ma et al . ( 2018 ) improved Shufflenet by designing ShufflenetV2 Unit , denoted as ShuffleUnitV2 in this paper . The architecture and detailed descriptions are in Figure 5 and Appendix D . 3 PROPOSED METHOD . We describe FALCON , our proposed method for compressing CNN . We first define Extended Hadamard Product ( EHP ) , a key mathematical formulation to generalize depthwise separable convolution , in Section 3.1 . We interpret depthwise separable convolution used in Mobilenet using EHP in Section 3.2 . We propose FALCON in Section 3.3 and explain why FALCON can replace standard convolution . Then , we propose rank-k FALCON , which extends the basic FALCON , in Section 3.4 . We show that FALCON can be easily integrated into a branch architecture to compress it with little sacrifice of accuracy in Section 3.5 . Finally , we theoretically analyze the performance of FALCON in Appendix C . 3.1 EXTENDED HADAMARD PRODUCT ( EHP ) . We define Extended Hadamard Product ( EHP ) , a generalized elementwise product for two operands of different shapes , to generalize the formulation of the relation between standard convolution and depthwise separable convolution . Before generalizing the formulation , we give an example of formulating the relation between standard convolution and depthwise separable convolution . Suppose we have a 4-order standard convolution kernel K ∈ RI×J×M×N , a 3-order depthwise convolution kernel D ∈ RI×J×M , and a pointwise convolution kernel P ∈ RM×N . Let Ki , j , m , n be ( i , j , m , n ) -th element of K , Di , j , m be ( i , j , m ) -th element of D , and Pm , n be ( m , n ) -th element of P. Then , it can be shown that applying depthwise convolution with D and pointwise convolution with P is equivalent to applying standard convolution kernel K where Ki , j , m , n = Di , j , m · Pm , n ( see Section 3.2 for detailed proof ) . To formally express this relation , we define Extended Hadamard Product ( EHP ) as follows . Definition 1 ( Extended Hadamard Product ) . Given p-order tensor D ∈ RI1×···×Ip−1×M and qorder tensor P ∈ RM×J1×···×Jq−1 , the Extended Hadamard Product D E P of D and P is defined to be the tensor K ∈RI1× ... ×Ip−1×M×J1× ... ×Jq−1 where the last axis of D and the first axis of P are the common axes such that Ki1 , ... , ip−1 , m , j1 , ... , jq−1 = Di1 , ... , ip−1 , m ·Pm , j1 , ... , jq−1 for all elements of K. Contrary to Hadamard Product which is defined only if the shapes of the two operands are the same , Extended Hadamard Product ( EHP ) deals with tensors of different shapes . Now , we define a special case of Extended Hadamard Product ( EHP ) for a third-order tensor and a matrix . Definition 2 ( Extended Hadamard Product for a third order tensor and a matrix ) . Given a thirdorder tensor D ∈ RI×J×M and a matrix P ∈ RM×N , the Extended Hadamard Product D E P is defined to be the tensor K ∈RI×J×M×N where the third axis of the tensor D and the first axis of the matrix P are the common axes such that Ki , j , m , n = Di , j , m ·Pm , n . for all elements of K. We will see that the depthwise separable convolution in Mobilenet can be easily expressed with EHP in Section 3.2 ; we also propose a new architecture FALCON based on EHP in Section 3.3 . EHP is also a core operation that helps us understand other convolution architectures including MobilenetV2 and Shufflenet ( see Appendix E ) . 3.2 DEPTHWISE SEPARABLE CONVOLUTION AND EHP . In this section , we discuss how to represent the convolution layer of Mobilenet as Extended Hadamard Product ( EHP ) described in Section 3.1 . We interpret the depthwise separable convolution , which is the convolution of Mobilenet , as an application of EHP . This interpretation leads to designing a better convolution architecture FALCON in Section 3.3 . We represent the relationship between standard convolution kernel K ∈ RD×D×M×N and depthwise separable convolution consisting of depthwise convolution kernel D ∈ RD×D×M and pointwise convolution kernel P ∈ RM×N using one EHP operation . Figure 2 ( a ) illustrates the relationship between standard convolution and depthwise separable convolution used in Mobilenet . We show that applying depthwise separable convolution with D and P is equivalent to applying standard convolution with a kernel K which is constructed from D and P. Theorem 1 . Applying depthwise separable convolution with depthwise convolution kernel D ∈ RD×D×M and pointwise convolution kernel P ∈ RM×N is equivalent to applying standard convolution with kernel K = D E P. Proof . See Appendix B.1 .

Summary: The paper proposes a CNN compression method, based on the so called EHP operation, which can be used to  analyze and generalize depthwise separable convolution. Based on EHP, the paper develops depthwise separable convolution to compress CNNs, and extend it to a rank-k approach with further improved accuracy. Some analysis is provided about the operation equivalence. The experiments on standard benchmark datasets show the effectiveness of the method.