FALCON: Fast and Lightweight Convolution for Compressing and Accelerating CNN

1 INTRODUCTION . How can we efficiently reduce size and energy consumption of Convolutional Neural Networks ( CNN ) while maintaining their accuracy on classification tasks ? Nowadays , CNN is widely used in various areas including computer vision ( Krizhevsky et al . ( 2012 ) ; Simonyan & Zisserman ( 2014 ) ; Szegedy et al . ( 2017 ) ) , natural language processing ( Yin et al . ( 2016 ) ) , recommendation system ( Kim et al . ( 2016a ) ) , etc . In addition , model compression has become an important technique due to an increase in the model capacity and the number of parameters in CNN . One recent and promising direction for compressing CNN is depthwise separable convolution ( Sifre ( 2014 ) ) which replaces standard convolution with depthwise and pointwise convolutions . The depthwise convolution applies a separate 2D convolution kernel for each input channel , and the pointwise convolution changes the channel size using 1×1 convolution ( details in Section 2.1 ) . Several recent methods ( Howard et al . ( 2017 ) ; Sandler et al . ( 2018 ) ; Zhang et al . ( 2017 ) ) based on depthwise separable convolution show reasonable performances in terms of compression and computation reduction . However , existing approaches based on depthwise separable convolution have several crucial limitations . First , they are heuristic methods , and their relation to the standard convolution is not clearly identified . Second , due to their heuristic nature , generalizing the methods is difficult . Third , although they give reasonable compression and computation reduction , their accuracy is not sufficient compared to that of standard-convolution-based models . In this paper , we propose FALCON , an accurate and lightweight method for compressing CNN . FALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas . First , we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP ( Extended Hadamard Product ) , which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel . We then design FALCON by fine-tuning and reordering the results of EHP to improve the accuracy of convolution operations . Second , based on the precise definition , we generalize the FALCON to design rank-k FALCON , which further improves accuracy while sacrificing a bit of compression and computation reduction rates . We also propose FALCON-branch by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy . As a result , FALCON and FALCON-branch provide a superior accuracy compared to other methods based on depthwise separable convolution , with similar compression and computation reduction rates , and rank-k FALCON further improves accuracy , outperforming even the original convolution in many cases . Our contributions are summarized as follows : • Generalization . We analyze and generalize depthwise separable convolution to our proposed EHP ( Extended Hadamard Product ) operation . This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution . Furthermore , with fine-tuning operations , it leads to our proposed method FALCON . • Algorithm . We propose FALCON , a CNN compression method based on depthwise separable convolution . FALCON is carefully designed to compress CNN with little accuracy loss . We also propose rank-k FALCON to further improve the accuracy with a little sacrifice in compression and computation reduction rates . FALCON can be easily integrated into other architectures , and we propose FALCON-branch which combines FALCON with a branch architecture for a better performance . We theoretically analyze the compression and computation reduction rates of FALCON and other competitors . • Experiments . We perform extensive experiments and show that FALCON 1 ) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN , and 2 ) provides up to 8× compression and computation reduction compared to the standard convolution while giving similar accuracies . Furthermore , we show that rank-k FALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and floating-point operations . The rest of this paper is organized as follows . Section 2 explains preliminaries . Section 3 describes our proposed method FALCON . Section 4 presents experimental results . After discussing related works in Section 5 , we conclude in Section 6 . 2 PRELIMINARY . We describe preliminaries on depthwise separable convolution and methods based on depthwise separable convolution . Symbols used in this paper are described in Table 4 of Appendix . 2.1 DEPTHWISE SEPARABLE CONVOLUTION . Depthwise Separable Convolution ( DSConv ) consists of two sub-layers : depthwise convolution and pointwise convolution . The architecture of each convolution layer in DSConv is illustrated in Figure 5 ( a ) . Depthwise convolution ( DWConv ) kernel consists of several D×D 2-dimensional filters . The number of 2-dimension filters is the same as that of input feature maps . Each filter is applied on the corresponding input feature map , and produces an output feature map . Pointwise convolution ( PWConv ) , known as 1× 1 convolution , is a standard convolution with kernel size 1 . DSConv is defined as follows : O′h′ , w′ , m = D∑ i=1 D∑ j=1 Di , j , m · Ihi , wj , m ( 1 ) Oh′ , w′ , n = M∑ m=1 Pm , n ·O′h′ , w′ , m ( 2 ) where Di , j , m and Pm , n are depthwise convolution kernel and pointwise convolution kernel , respectively . O′h′ , w′ , m ∈ RH ′×W ′×M denotes intermediate feature maps . DSConv performs DWConv on input feature maps Ihi , wj , m using equation 1 , and generates intermediate feature maps O ′ h′ , w′ , m . Then , DSConv performs PWConv on O′h′ , w′ , m using equation 2 , and generates output feature maps Oh′ , w′ , n . 2.2 METHODS BASED ON DEPTHWISE SEPARABLE CONVOLUTION . Several CNN methods have been proposed based on Depthwise Separable Convolution ( DSConv ) recently . DSConv was first introduced by Sifre ( 2014 ) . Chollet & Franois ( 2016 ) built Xception module using DSConv in a few layers . Howard et al . ( 2017 ) built Mobilenet with all convolution layers replaced by DSConv . Sandler et al . ( 2018 ) built MobileNetV2 with inverted bottleneck block , denoted as MobileConvV2 in this paper . Zhang et al . ( 2017 ) built a CNN model with Shufflenet Unit , denoted as ShuffleUnit in this paper . Ma et al . ( 2018 ) improved Shufflenet by designing ShufflenetV2 Unit , denoted as ShuffleUnitV2 in this paper . The architecture and detailed descriptions are in Figure 5 and Appendix D . 3 PROPOSED METHOD . We describe FALCON , our proposed method for compressing CNN . We first define Extended Hadamard Product ( EHP ) , a key mathematical formulation to generalize depthwise separable convolution , in Section 3.1 . We interpret depthwise separable convolution used in Mobilenet using EHP in Section 3.2 . We propose FALCON in Section 3.3 and explain why FALCON can replace standard convolution . Then , we propose rank-k FALCON , which extends the basic FALCON , in Section 3.4 . We show that FALCON can be easily integrated into a branch architecture to compress it with little sacrifice of accuracy in Section 3.5 . Finally , we theoretically analyze the performance of FALCON in Appendix C . 3.1 EXTENDED HADAMARD PRODUCT ( EHP ) . We define Extended Hadamard Product ( EHP ) , a generalized elementwise product for two operands of different shapes , to generalize the formulation of the relation between standard convolution and depthwise separable convolution . Before generalizing the formulation , we give an example of formulating the relation between standard convolution and depthwise separable convolution . Suppose we have a 4-order standard convolution kernel K ∈ RI×J×M×N , a 3-order depthwise convolution kernel D ∈ RI×J×M , and a pointwise convolution kernel P ∈ RM×N . Let Ki , j , m , n be ( i , j , m , n ) -th element of K , Di , j , m be ( i , j , m ) -th element of D , and Pm , n be ( m , n ) -th element of P. Then , it can be shown that applying depthwise convolution with D and pointwise convolution with P is equivalent to applying standard convolution kernel K where Ki , j , m , n = Di , j , m · Pm , n ( see Section 3.2 for detailed proof ) . To formally express this relation , we define Extended Hadamard Product ( EHP ) as follows . Definition 1 ( Extended Hadamard Product ) . Given p-order tensor D ∈ RI1×···×Ip−1×M and qorder tensor P ∈ RM×J1×···×Jq−1 , the Extended Hadamard Product D E P of D and P is defined to be the tensor K ∈RI1× ... ×Ip−1×M×J1× ... ×Jq−1 where the last axis of D and the first axis of P are the common axes such that Ki1 , ... , ip−1 , m , j1 , ... , jq−1 = Di1 , ... , ip−1 , m ·Pm , j1 , ... , jq−1 for all elements of K. Contrary to Hadamard Product which is defined only if the shapes of the two operands are the same , Extended Hadamard Product ( EHP ) deals with tensors of different shapes . Now , we define a special case of Extended Hadamard Product ( EHP ) for a third-order tensor and a matrix . Definition 2 ( Extended Hadamard Product for a third order tensor and a matrix ) . Given a thirdorder tensor D ∈ RI×J×M and a matrix P ∈ RM×N , the Extended Hadamard Product D E P is defined to be the tensor K ∈RI×J×M×N where the third axis of the tensor D and the first axis of the matrix P are the common axes such that Ki , j , m , n = Di , j , m ·Pm , n . for all elements of K. We will see that the depthwise separable convolution in Mobilenet can be easily expressed with EHP in Section 3.2 ; we also propose a new architecture FALCON based on EHP in Section 3.3 . EHP is also a core operation that helps us understand other convolution architectures including MobilenetV2 and Shufflenet ( see Appendix E ) . 3.2 DEPTHWISE SEPARABLE CONVOLUTION AND EHP . In this section , we discuss how to represent the convolution layer of Mobilenet as Extended Hadamard Product ( EHP ) described in Section 3.1 . We interpret the depthwise separable convolution , which is the convolution of Mobilenet , as an application of EHP . This interpretation leads to designing a better convolution architecture FALCON in Section 3.3 . We represent the relationship between standard convolution kernel K ∈ RD×D×M×N and depthwise separable convolution consisting of depthwise convolution kernel D ∈ RD×D×M and pointwise convolution kernel P ∈ RM×N using one EHP operation . Figure 2 ( a ) illustrates the relationship between standard convolution and depthwise separable convolution used in Mobilenet . We show that applying depthwise separable convolution with D and P is equivalent to applying standard convolution with a kernel K which is constructed from D and P. Theorem 1 . Applying depthwise separable convolution with depthwise convolution kernel D ∈ RD×D×M and pointwise convolution kernel P ∈ RM×N is equivalent to applying standard convolution with kernel K = D E P. Proof . See Appendix B.1 .

Summary: This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon's main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P's memory is  D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read.