Title: FALCON: FAST AND LIGHTWEIGHT CONVOLUTION FOR COMPRESSING AND ACCELERATING CNN

Abstract: How can we efficiently compress Convolutional Neural Networks (CNN) while retaining their accuracy on classification tasks? A promising direction is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and a pointwise convolution. However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution. In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads to developing a generalized version rank-k FALCON which further improves the accuracy while sacrificing a bit of compression and computation reduction rates. In addition, we propose FALCON-branch by fitting FALCON into the previous state-of-the-art convolution unit ShuffleUnitV2 which gives even better accuracy. Experiments show that FALCON and FALCON-branch outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8× compression and 8× computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accuracy than standard convolution in many cases, while using a smaller number of parameters and floating-point operations.

1. INTRODUCTION:

How can we efficiently reduce size and energy consumption of Convolutional Neural Networks (CNN) while maintaining their accuracy on classification tasks? Nowadays, CNN is widely used in various areas including computer vision (Krizhevsky et al. (2012) ; Simonyan & Zisserman (2014) ; Szegedy et al. (2017) ), natural language processing (Yin et al. (2016) ), recommendation system (Kim et al. (2016a) ), etc. In addition, model compression has become an important technique due to an increase in the model capacity and the number of parameters in CNN. One recent and promising direction for compressing CNN is depthwise separable convolution (Sifre (2014) ) which replaces standard convolution with depthwise and pointwise convolutions. The depthwise convolution applies a separate 2D convolution kernel for each input channel, and the pointwise convolution changes the channel size using 1×1 convolution (details in Section 2.1). Several recent methods (Howard et al. (2017) ; Sandler et al. (2018) ; Zhang et al. (2017) ) based on depthwise separable convolution show reasonable performances in terms of compression and computation reduction.

However, existing approaches based on depthwise separable convolution have several crucial limitations. First, they are heuristic methods, and their relation to the standard convolution is not clearly identified. Second, due to their heuristic nature, generalizing the methods is difficult. Third, although they give reasonable compression and computation reduction, their accuracy is not sufficient compared to that of standard-convolution-based models.

In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON overcomes the limitations of the previous methods based on the depthwise separable convolution using the following two main ideas. First, we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP (Extended Hadamard Product), which is our proposed mathematical formulation to correlate the standard convolution kernel with the depthwise convolution kernel and the pointwise convolution kernel. We then design FALCON by fine-tuning and reordering the results of EHP to improve the accuracy of convolution operations. Second, based on the precise definition, we generalize the FALCON to design rank-k FALCON, which further improves accuracy while sacrificing a bit of compression and computation reduction rates. We also propose FALCON-branch by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy. As a result, FALCON and FAL-CON-branch provide a superior accuracy compared to other methods based on depthwise separable convolution, with similar compression and computation reduction rates, and rank-k FALCON further improves accuracy, outperforming even the original convolution in many cases. Our contributions are summarized as follows:

• Generalization. We analyze and generalize depthwise separable convolution to our proposed EHP (Extended Hadamard Product) operation. This generalization enables a precise understanding of the relationship between depthwise separable convolution and standard convolution. Furthermore, with fine-tuning operations, it leads to our proposed method FALCON. • Algorithm. We propose FALCON, a CNN compression method based on depthwise separable convolution. FALCON is carefully designed to compress CNN with little accuracy loss. We also propose rank-k FALCON to further improve the accuracy with a little sacrifice in compression and computation reduction rates. FALCON can be easily integrated into other architectures, and we propose FALCON-branch which combines FALCON with a branch architecture for a better performance. We theoretically analyze the compression and computation reduction rates of FALCON and other competitors. • Experiments. We perform extensive experiments and show that FALCON 1) outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN, and 2) provides up to 8× compression and computation reduction compared to the standard convolution while giving similar accuracies. Furthermore, we show that rank-k FALCON provides even better accuracy than the standard convolution in many cases while using a smaller number of parameters and floating-point operations.

The rest of this paper is organized as follows. Section 2 explains preliminaries. Section 3 describes our proposed method FALCON. Section 4 presents experimental results. After discussing related works in Section 5, we conclude in Section 6.

2. PRELIMINARY:

We describe preliminaries on depthwise separable convolution and methods based on depthwise separable convolution. Symbols used in this paper are described in Table 4 of Appendix. The number of 2-dimension filters is the same as that of input feature maps. Each filter is applied on the corresponding input feature map, and produces an output feature map. Pointwise convolution (PWConv), known as 1 × 1 convolution, is a standard convolution with kernel size 1.

DSConv is defined as follows: 

EQUATION

EQUATION

3. PROPOSED METHOD:

We describe FALCON, our proposed method for compressing CNN. We first define Extended Hadamard Product (EHP), a key mathematical formulation to generalize depthwise separable convolution, in Section 3.1. We interpret depthwise separable convolution used in Mobilenet using EHP in Section 3.2. We propose FALCON in Section 3.3 and explain why FALCON can replace standard convolution. Then, we propose rank-k FALCON, which extends the basic FALCON, in Section 3.4. We show that FALCON can be easily integrated into a branch architecture to compress it with little sacrifice of accuracy in Section 3.5. Finally, we theoretically analyze the performance of FALCON in Appendix C.

3.1. EXTENDED HADAMARD PRODUCT (EHP):

We define Extended Hadamard Product (EHP), a generalized elementwise product for two operands of different shapes, to generalize the formulation of the relation between standard convolution and depthwise separable convolution. Before generalizing the formulation, we give an example of formulating the relation between standard convolution and depthwise separable convolution. Suppose we have a 4-order standard convolution kernel K ∈ R I×J×M ×N , a 3-order depthwise convolution kernel D ∈ R I×J×M , and a pointwise convolution kernel P ∈ R M ×N . Let K i,j,m,n be (i, j, m, n)-th element of K, D i,j,m be (i, j, m)-th element of D, and P m,n be (m, n)-th element of P. Then, it can be shown that applying depthwise convolution with D and pointwise convolution with P is equivalent to applying standard convolution kernel K where K i,j,m,n = D i,j,m • P m,n (see Section 3.2 for detailed proof). To formally express this relation, we define Extended Hadamard Product (EHP) as follows.

Definition 1 (Extended Hadamard Product). Given p-order tensor D ∈ R I1×•••×Ip-1×M and qorder tensor P ∈ R M ×J1×•••×Jq-1 , the Extended Hadamard Product D E P of D and P is defined to be the tensor K ∈R I1×...×Ip-1×M ×J1×...×Jq-1 where the last axis of D and the first axis of P are the common axes such that K i1,...,ip-1,m,j1,...,jq-1 = D i1,...,ip-1,m • P m,j1,...,jq-1 for all elements of K.

Contrary to Hadamard Product which is defined only if the shapes of the two operands are the same, Extended Hadamard Product (EHP) deals with tensors of different shapes. Now, we define a special case of Extended Hadamard Product (EHP) for a third-order tensor and a matrix. Definition 2 (Extended Hadamard Product for a third order tensor and a matrix). Given a thirdorder tensor D ∈ R I×J×M and a matrix P ∈ R M ×N , the Extended Hadamard Product D E P is defined to be the tensor K ∈R I×J×M ×N where the third axis of the tensor D and the first axis of the matrix P are the common axes such that K i,j,m,n = D i,j,m • P m,n . for all elements of K.

We will see that the depthwise separable convolution in Mobilenet can be easily expressed with EHP in Section 3.2; we also propose a new architecture FALCON based on EHP in Section 3.3. EHP is also a core operation that helps us understand other convolution architectures including MobilenetV2 and Shufflenet (see Appendix E).

3.2. DEPTHWISE SEPARABLE CONVOLUTION AND EHP:

In this section, we discuss how to represent the convolution layer of Mobilenet as Extended Hadamard Product (EHP) described in Section 3.1. We interpret the depthwise separable convolution, which is the convolution of Mobilenet, as an application of EHP. This interpretation leads to designing a better convolution architecture FALCON in Section 3.3.

We represent the relationship between standard convolution kernel K ∈ R D×D×M ×N and depthwise separable convolution consisting of depthwise convolution kernel D ∈ R D×D×M and pointwise convolution kernel P ∈ R M ×N using one EHP operation. Figure 2 (a) illustrates the relationship between standard convolution and depthwise separable convolution used in Mobilenet. We show that applying depthwise separable convolution with D and P is equivalent to applying standard convolution with a kernel K which is constructed from D and P.

Theorem 1. Applying depthwise separable convolution with depthwise convolution kernel D ∈ R D×D×M and pointwise convolution kernel P ∈ R M ×N is equivalent to applying standard convolution with kernel K = D E P.

Proof. See Appendix B.1.

3.3. FAST AND LIGHTWEIGHT CONVOLUTION (FALCON):

We propose FALCON (FAst and Lightweight CONvolution), a novel lightweight convolution that replaces standard convolution. FALCON is an efficient method with fewer parameters and computations than those that the standard convolution requires. In addition, FALCON has better accuracy than competitors while having similar compression and computation reduction rates. The main idea of FALCON is 1) to carefully align depthwise and pointwise convolutions, and 2) initialize kernels using the convolution kernels of the trained standard model. We observe that a typical convolution has more output channels than input channels. In such a setting, performing depthwise convolution after pointwise convolution would allow the depthwise convolution to extract more features from richer feature space; on the other hand, performing pointwise convolution after depthwise convolution as in Mobilenet only combines features extracted from a limited feature space. Based on the observation, FALCON first applies pointwise convolution to generate an intermediate tensor O ∈ R H×W ×N and then applies depthwise convolution.

We represent the relationship between standard convolution kernel K ∈ R D×D×M ×N and FALCON by applying an EHP operation on pointwise convolution kernel P ∈ R N ×M and depthwise convolution kernel D ∈ R D×D×N in Figure 2 (b). In FALCON, the kernel K is represented by EHP of D and P as follows:

K = T T (1,2,4,3) (D E P) s.t. K i,j,m,n = P n,m • D i,j,n .

where T T (1,2,4,3) indicates tensor transpose operation to permute the third and the fourth dimensions of a tensor. Note that the common axis is the output channel axis of the standard convolution, unlike EHP for depthwise separable convolution where the common axis is the input channel axis of the standard convolution.

As in Section 3.2, we show that applying FALCON is equivalent to applying standard convolution with a specially constructed kernel.

Theorem 2. FALCON which applies pointwise convolution with kernel P ∈ R N ×M and then depthwise convolution with kernel D ∈ R D×D×N is equivalent to applying standard convolution with kernel

K = T T (1,2,4,3) (D E P). Proof. See Appendix B.2.

Based on the equivalence, we initialize pointwise convolution and depthwise convolution kernels D and P of FALCON by fitting them to the convolution kernels of the trained standard model; i.e., D, P = arg min D ,P ||K -T T (1,2,4,3) (D E P )|| F . After pointwise convolution and depthwise convolution, we add batch-normalization and ReLU activation function as shown in Figure 1(b) . We note that FALCON significantly reduces the numbers of parameters and FLOPs compared to standard convolution, which we discuss at Appendix C.

3.4. RANK-k FALCON:

We propose rank-k FALCON, an extended version of FALCON that improves accuracy while sacrificing a bit of compression and computation reduction rates. The main idea is to perform k independent FALCON operations and sum up the result. Then, we apply batch-normalization (BN) and ReLU activation function to the summed result. Since each FALCON operation requires independent parameters for pointwise convolution and depthwise convolution, the number of parameters increases and thus the compression and the computation reduction rates decrease; however, it improves accuracy by enlarging the model capacity. We formally define the rank-k FALCON with EHP as follows.

Definition 3 (Rank-k FALCON with Extended Hadamard Product). Rank-k FALCON expresses standard convolution kernel K ∈ R D×D×M ×N as EHP of depthwise convolution kernel D (i) ∈ R D×D×N and pointwise convolution kernel

P (i) ∈ R N ×M for i = 1, 2, ..., k such that K = k i=1 T T (1,2,4,3) (D (i) E P (i) ) s.t. K i,j,m,n = k i=1 P (i) n,m • D (i) i,j,n

Figure 2(c) illustrates the relation between standard convolution and rank-k FALCON. For each i = 1, 2, ..., k, we construct the tensor K (i) using EHP of the depthwise convolution kernel D (i) and the pointwise convolution kernel P (i) . Then, we construct the standard kernel K by the elementwise sum of the tensors K (i) for all i.

3.5. FALCON-BRANCH:

FALCON can be easily integrated into a CNN architecture called standard convolution operation with a branch (StConv-branch), which consists of two branches: standard convolution on the left branch and a residual connection on the right branch (see Figure 1 (c)). Ma et al. (2018) improved the performance of CNN by applying depthwise and pointwise convolutions on the left branch of StConv-branch. Since FALCON replaces standard convolution, we observe that StConv-branch can be easily compressed by applying FALCON on the left branch.

StConv-branch first splits an input in half along the depth dimension. A standard convolution operation is applied to one half, and no operation to the other half. The two are concatenated along the depth dimension, and an output is produced by shuffling the channels of the concatenated tensor. FALCON-branch (see Figure 1(d) ) is constructed by replacing the standard convolution branch (left branch) of StConv-branch with FALCON. Advantages of FALCON-branch are that 1) the branch architecture improves the efficiency since convolutions are applied to only half of input feature maps and that 2) FALCON further compresses the left branch effectively. FALCON-branch is initialized by fitting FALCON to the standard convolution kernel of the left branch of StConv-branch.

4. EXPERIMENTS:

We validate the performance of FALCON through extensive experiments. We aim to answer the following questions:

• Q1. Accuracy vs. Compression (Section 4.3). What are the accuracy and the compression tradeoffs of FALCON, FALCON-branch, and competitors? Which method gives the best accuracy for a given compression rate? • Q2. Accuracy vs. Computation (Section 4.4). What are the accuracy and the computation tradeoffs of FALCON, FALCON-branch, and competitors? Which method gives the best accuracy for a given amount of computation? • Q3. Rank-k FALCON (Section 4.5). How do the accuracy, the number of parameters, and the number of FLOPs change as the rank k increases in FALCON?

4.1. EXPERIMENTAL SETUP:

Datasets. We perform image classification task on four famous datasets -CIFAR10, CIFAR100, SVHN, and ImageNet. Detailed information of these datasets is described in Table 1 . Models. For CIFAR10, CIFAR100, and SVHN datasets, we choose VGG19 and ResNet34 to evaluate the performance. We shrink the sizes of both models since the sizes of these three datasets are smaller than that of Imagenet. In VGG19, we reduce the number of fully connected layers and the number of features in fully connected layers: three large fully connected layers (4096-4096-1000) in VGG19 are replaced with two small fully connected layers (512-10 or 512-100). In ResNet34, we remove the first 7 × 7 convolution layer and max-pooling layer since the input size (32 × 32) of these datasets is smaller than the input size (224 × 224) of ImageNet. On both models, we replace all standard convolution layers (except for the first convolution layer) with those of FALCON or other competitors in order to compress and accelerate the model. For ImageNet, we choose VGG16 BN (VGG16 with batch normalization after every convolution layer) and ResNet18. We use the pretrained model from Pytorch model zoo as the baseline model with standard convolution, and replace the standard convolution with other types of convolutions.

Competitors. We compare FALCON and FALCON-branch with four convolution units consisting of depthwise convolution and pointwise convolution: DSConv, MobileConvV2, ShuffleUnit, and ShuffleUnitV2 (see Figure 5 , Section 2.1, and Appendix D for more details). To evaluate the effectiveness of fitting depthwise and pointwise convolution kernels to standard convolution kernel, we build EHP-in which is DSConv where kernels D and P are fitted from the pretrained standard convolution kernel K; i.e., D, P = arg min D ,P ||K -

D E P || F .

Implementation. We construct all models using Pytorch framework. All the models are trained and tested on GeForce GTX 1080 Ti GPU.

4.2. FITTING CONVOLUTION UNIT INTO MODEL:

We evaluate the performance of FALCON against DSConv, MobileConvV2, ShuffleUnit, and Shuf-fleUnitV2. We take each standard convolution layer (StConv) as a unit, and replace StConv with those from FALCON or other competitors. We evaluate the classification accuracy, the number of parameters in the model, and the number of FLOPs needed for forwarding one image. We only explain how to apply FALCON in this section. The details of how to fit other convolution units into the models are described in Appendix F.

FALCON. When replacing StConv with FALCON, we use the same setting as that of StConv. I.e., if there are BN and ReLU after StConv, we add BN and ReLU at the end of FALCON; if there is only ReLU after StConv, we add only ReLU at the end of FALCON. This is because FALCON is initialized by approximating the StConv kernel using EHP. Using the same setting for BN and ReLU as StConv is more efficient for FALCON to approximate the StConv. We initialize the pointwise convolution kernel and the depthwise convolution kernel of FALCON by approximating the pretrained standard convolution kernel using EHP. The approximation process is as follows: 1) we first initialize the pointwise convolution kernel and the depthwise convolution kernel randomly, and 2) the pointwise convolution kernel and the depthwise convolution kernel are updated using gradient descent such that the mean squared error of their EHP product and the standard convolution kernel is minimized. Rank-k FALCON uses the same initialization method.

4.3. ACCURACY VS. COMPRESSION:

We evaluate the accuracy and the compression rate of FALCON and competitors. Table 2 shows the results on four image datasets. Note that FALCON or FALCON-branch provides the highest accuracy in 7 out of 8 cases while using similar or smaller number of parameters than competitors. Specifically, FALCON and FALCON-branch achieve up to 8× compression rates with less than 1% accuracy drop compared to that of the standard convolution (StConv). Figure 3 shows the tradeoff between accuracy and the number of parameters. Note that FALCON and FALCON-branch show the best tradeoff (closest to the "best" point) between accuracy and compression rate, giving the highest accuracy with similar compression rates.

Ablation study. We perform an ablation study on two components: 1) the order of depthwise and pointwise convolutions, and 2) initialization. We observe that with a similar number of parameters, 1) FALCON and FALCON without initialization always result in better accuracy than EHP-in and DSC, respectively, and 2) EHP-in always results in better accuracy than DSC. Furthermore, FALCON results in better accuracy than FALCON without initialization in 6 out of 8 cases. These observations prove our claims in Section 3.3 that 1) EHP-out (FALCON) is more efficient than EHP-in, and 2) the fitting and the initialization of kernels using EHP improves accuracy. Additionally, we observe that overall performance is more sensitive towards ordering compared to initialization.

4.4. ACCURACY VS. COMPUTATION:

We evaluate the accuracy and the amount of computation of FALCON and competitors. We use the number of multiply-adds floating point operations (FLOPs) needed for forwarding one image to a model as the metric of computation. Table 2 also shows the accuracies and the number of FLOPs of methods on four image datasets. Note that FALCON or FALCON-branch provide the highest accuracy in 7 out of 8 cases while using similar FLOPs as competitors do. Compared to StConv, FALCON and FALCON-branch achieve up to 8× FLOPs reduction across different models on different datasets. Figure 4 shows the tradeoff between accuracy and the number of FLOPs.

Note that FALCON and FALCON-branch show the best tradeoff (closest to the "best" point) between accuracy and computation, giving the highest accuracy with a similar number of FLOPs.

4.5. RANK-K FALCON:

We evaluate the performance of rank-k FALCON by increasing the rank k and monitoring the changes in the numbers of parameters and FLOPs. In Table 3 , we observe three trends as the rank k increases: 1) the accuracy becomes higher than that of rank-1 FALCON, 2) the number of parameters increases, and 3) the number of floating point operations (FLOPs) increases. Although the rank k that gives the best tradeoff of rank and compression/computation reduction varies, rank-k FALCON improves the accuracy of FALCON in all cases. Especially, we note that rank FALCON often gives even higher accuracy than the standard convolution, while using smaller number of parameters and FLOPs. For example, rank-3 FALCON applied to VGG19 on CIFAR100 dataset shows 1.31 percentage point higher accuracy compared to the standard convolution, with 2.8× smaller number of parameters and 2.8× smaller number of FLOPs. Thus, rank-k FALCON is a versatile method to further improve the accuracy of FALCON while sacrificing a bit of compression and computation.

5. RELATED WORK:

Over the past several years, a lot of studies focused on compressing and accelerating DNN to reduce model size, running time, and energy consumption. method which stores only assignments and centroids of weights. While using the model, weights are loaded according to assignments and centroids. Pruning (Han et al. (2014) ; Li et al. ( 2016)) aims at removing useless weights or setting them to zero. Although weight-sharing and pruning can significantly reduce the model size, they are not efficient in reducing the amount of computation.

Quantizing (Courbariaux et al. (2015; 2016) Layer-wise approaches are also employed to efficiently compress models. A typical example of such approaches is low-rank approximation (Lebedev et al. (2015) ; Kim et al. (2016b) ; Novikov et al. (2015) ); it treats the weights as a tensor and uses general tensor approximation methods to compress the tensor. To reduce computation, approximation methods should be carefully chosen, since some of approximation methods may increase computation of the model.

Compressing existing models has limitations since they are originally designed to be deep and large to give high accuracy. A recent trend is to design a brand new architecture that is small and efficient. Mobilenet (Howard et al. (2017) ), MobilenetV2 (Sandler et al. (2018) ), Shufflenet (Zhang et al. (2017) ), and ShufflenetV2 (Ma et al. (2018) ) are the most representative approaches, and they use depthwise convolution and pointwise convolution as building blocks for designing convolution layers. Our proposed FALCON gives a thorough interpretation of depthwise convolution and pointwise convolution, and applies them into model compression, giving the best accuracies with regard to compression and computation.

6. CONCLUSION:

We propose FALCON, an accurate and lightweight convolution method to replace standard convolution. By interpreting existing convolution methods based on depthwise separable convolution using EHP operation, FALCON and its general version rank-k FALCON provide accurate and efficient compression on CNN. We also propose FALCON-branch, a variant of FALCON integrated into a branch architecture of CNN for model compression. Extensive experiments show that FALCON and its variants give the best accuracy for a given number of parameter or computation, outperforming other convolution models based on depthwise separable convolution. Compared to the standard convolution, FALCON and FALCON-branch give up to 8× compression and 8× computation reduction while giving similar accuracy. We also show that rank-k FALCON provides better accuracy than the standard convolution, while using smaller numbers of parameters and computations. CNN uses convolution operation in convolution layers. In the following, we discuss CNN when applied to typical image data with RGB channels.

Each convolution layer has three components: input feature maps, convolution kernel, and output feature maps. The input feature maps I ∈ R H×W ×M and the output feature maps O ∈ R H ×W ×N are 3-dimensional tensors, and the convolution kernel K ∈ R D×D×M ×N is a 4-dimensional tensor.

The convolution operation is defined as:

EQUATION

where the relations between height h i and width w j of input, and height h and width w of output are as follows: h i = (h -1)s + i -p and w j = (w -1)s + j -p (4) where s is the stride size, and p is the padding size. The third and the fourth dimensions of the convolution kernel K must match the number M of input channels, and the number N of output channels, respectively.

Convolution kernel K can be seen as N 3-dimensional filters F n ∈ R D×D×M . Each filter F n in kernel K performs convolution operation while sliding over all spatial locations on input feature maps. Each filter produces one output feature map.

B PROOFS OF THEOREMS B.1 PROOF OF THEOREM 1:

Proof. From the definition of EHP, K i,j,m,n = D i,j,m • P m,n . Based on equation 3, we replace the kernel K i,j,m,n with the depthwise convolution kernel D i,j,m and the pointwise convolution kernel P m,n .

O h ,w ,n = D i=1 D j=1 M m=1 D i,j,m • P m,n • I hi,wj ,m

where I hi,wj ,m is the (h i , w j , m)-th entry of the input. We split the above equation into the following two equations.

EQUATION

EQUATION

where O h ,w ,m ∈ R H ×W ×M is an intermediate tensor. Note that equation 5 and equation 6 correspond to the depthwise convolution and the pointwise convolution, respectively. Therefore, the output O h ,w ,m is equal to the output applying depthwise separable convolution used in Mobilenet.

B.2 PROOF OF THEOREM 2:

Proof. From equation 3, we replace the kernel K i,j,m,n with the pointwise convolution kernel P and the depthwise convolution kernel D. where I hi,wj ,m is the (h i , w j , m)-th entry of the input I. We split the above equation into the following two equations.

EQUATION

EQUATION

where I, O , and O are the input, the intermediate, and the output tensors of convolution layer, respectively. Note that equation 7 and equation 8 correspond to pointwise convolution and depthwise convolution, respectively. Therefore, the output O h ,w ,n is equal to the output applying FALCON.

C QUANTITATIVE ANALYSIS:

In this section, we evaluate the compression and the computation reduction of FALCON and rank-k FALCON. All the analysis is based on one convolution layer. The comparison of the numbers of parameters and FLOPs of FALCON and other competitors is in Appendix G.

C.1 FALCON:

We analyze the compression and the computation reduction rates of FALCON in Theorems 3 and 4. Theorem 3. Compression Rate (CR) of FALCON is given by

CR = # of parameters in standard convolution # of parameters in FALCON = D 2 M N M N + D 2 N

where D 2 is the size of standard kernel, M is the number of input channels, and N is the number of output channels. The number of depthwise convolution channels is 1 4 of output channels N . ShuffleUnit uses group convolution in two pointwise convolution layers to reduce the parameters and FLOPs. However, it is hard to exchange information among groups when group convolutions are stacked. To deal with this problem, ShuffleUnit adds a channel shuffle layer after the first pointwise group convolution. The channel shuffle layer rearranges the order of channels. making it possible to obtain information from different groups. The number of groups is represented as g. ShuffleUnit needs

1 4g M N + 1 4 D 2 N + 1 4g N 2 parameters and 1 4g HW M N + 1 4 H W D 2 N + 1 4g H W N 2 FLOPs.

ShufflenetV2. Ma et al. (2018) proposed a practically efficient CNN architecture ShufflenetV2. As shown in Figure 5 (d), each unit of ShufflenetV2 (we call it ShuffleUnitV2) consists of two branches. The left branch consists of two pointwise convolutions and one depthwise convolution like MobileConvV2, and the right branch is an identity operation. Note that outputs of both branches maintain the number of channels as M/2. The final output is produced by concatenating and shuffling the output tensors from both of the branches. ShuffleUnitV2 needs 1 2 (M 2 + D 2 M ) parameters and 1 2 HW (M 2 + D 2 M ) FLOPs.

E GENERALITY OF EHP:

We show that EHP is a key operation to understand other convolution architectures based on depthwise separable convolution.

MobilenetV2. As shown in Figure 5 (b), MobilenetV2 has an additional pointwise convolution before depthwise convolution in Mobilenet: one layer of MobilenetV2 consists of two pointwise convolutions and one depthwise convolution. In another point of view, MobilenetV2 can be understood as FALCON followed by additional pointwise convolution; i.e., MobilenetV2 performs EHP operation as FALCON does, and performs additional pointwise convolution after that.

Shufflenet. As shown in Figure 5 (c), Shufflenet consists of depthwise convolution and pointwise group convolution which is a variant of pointwise convolution. We represent the convolution layer of Shufflenet using EHP as follows. Let g be the number of groups. We divide the standard convolution kernel K ∈ R D×D×M ×N into g group standard convolution kernels. Then, the relation of g-th group standard convolution kernel K g ∈ R D×D× M g × N g with regard to g-th depthwise convolution kernel D g ∈ R D×D× M g and g-th pointwise group convolution kernel P g ∈ R M g × N g is K g = D g E P g s.t. K g i,j,mg,ng = D g i,j,mg • P g mg,ng

where m g = 1, 2, ..., M g and n g = 1, 2, ..., N g . Each group standard convolution is equivalent to the combination of a depthwise convolution and a pointwise convolution, and thus easily expressed with EHP as in Mobilenet.

Therefore, each layer of Shufflenet is equivalent to the layer consisting of one group convolution followed by standard convolution.

ShufflenetV2. As shown in Figure 5 (d), the left branch of ShufflenetV2 has the same convolutions as in MobilenetV2: it consists of two pointwise convolutions and one depthwise convolution. Like MobilenetV2, the left branch of ShufflenetV2 can be understood as FALCON followed by additional pointwise convolution.

F FITTING OTHER CONVOLUTION UNITS INTO MODELS:

DSConv. DSConv (shown in Figure 5 (a)) has the most similar architecture as FALCON among competitors, and thus DSConv has nearly the same number of parameters as that of FALCON. As in the setting of FALCON, the existence of BN and ReLU at the end of DSConv depends on that of StConv.

MobileConvV2. In MobileConvV2 architecture (shown in Figure 5 (b)), we adjust the numbers of parameters and FLOPs by changing the expansion ratio t as described in Appendix D, which is represented as 'MobileConvV2-t'. We choose t = 0.5 as the baseline MobileConvV2 to compare with FALCON, since two pointwise convolutions bring lots of parameters and FLOPs to MobileConvV2.

ShuffleUnit. In ShuffleUnit (shown in Figure 5 (c)), we adjust the numbers of parameters and FLOPs by changing the width multiplier α (Howard et al. (2017) ) and the number of groups g, which is represented as 'ShuffleUnit α×(g=g)'. Note that the width multiplier is used to adjust the number of input channels M and the number of output channels N of a convolution layer; if the width multiplier is α, the numbers of input and output channels become αM and αN , respectively. While experimenting with ResNet, we find that ShuffleUnit does not cooperate well with ResNet: ResNet34 with ShuffleUnit does not converge. We suspect that residual block and ShuffleUnit may conflict with each other because of redundant residual connections: the gradient may not find the right path towards previous layers. For this reason, we delete the shortcut of all residual blocks in ResNet34 when using ShuffleUnit.

ShuffleUnitV2. In ShuffleUnitV2 (shown in Figure 5(d )),we also adjust the number of parameters and FLOPs by changing the width multiplier α, which is represented as 'ShuffleUnitV2 α×'. Other operations of ShuffleUnitV2 stay the same as in Ma et al. (2018) .

G PARAMETERS AND FLOPS:

We summarize the numbers of parameters and FLOPs for FALCON and competitors in Table 5 .

:

https://www.cs.toronto.edu/ ˜kriz/cifar.html

https://www.cs.toronto.edu/ ˜kriz/cifar.html

http://ufldl.stanford.edu/housenumbers/

http://www.image-net.org

Figures and Tables:

DEPTHWISE SEPARABLE CONVOLUTION Depthwise Separable Convolution (DSConv) consists of two sub-layers: depthwise convolution and pointwise convolution. The architecture of each convolution layer in DSConv is illustrated in Figure 5(a). Depthwise convolution (DWConv) kernel consists of several D × D 2-dimensional filters.

Figure 1: Comparison of architectures. BN denotes batch-normalization. Relu and Relu6 are activation functions. (a) Standard convolution. (b) Our proposed FALCON. (c) Standard convolution with branch (StConv-branch). (d) FALCON-branch which combines FALCON with StConv-branch. input feature maps I hi,wj ,m using equation 1, and generates intermediate feature maps O h ,w ,m . Then, DSConv performs PWConv on O h ,w ,m using equation 2, and generates output feature maps O h ,w ,n .

Figure 2: (a) Relation between standard convolution and depthwise separable convolution expressed with EHP. The common axes correspond to the input channel-axis of standard convolution. (b) Relation between standard convolution and FALCON expressed with EHP. The common axes correspond to the output channel-axis of standard convolution. T T (1,2,4,3) indicates tensor transpose operation to permute the third and the fourth dimensions of a tensor. (c) Relation between the standard convolution and rank-k FALCON expressed with EHP.

Figure3: Accuracy w.r.t. number of parameters on different models and datasets. The three blue circles correspond to rank-1, 2, 3 FALCON (from left to right order), respectively. FALCON provides the best accuracy for a given number of parameters.

It is believed that DNNs are over-parameterized. Weight-sharing (Han et al. (2016); Ullrich et al. (2017); Chen et al. (2015); Choi et al. (2017); Agustsson et al. (2017)) is a common compression

;Hou et al. (2017);Zhu et al. (2017)) the model into binary or ternary weights reduces model size and computation simultaneously: replacing arithmetic operations with bit-wise operations remarkably accelerates the model.

n • D i,j,n • I hi,wj ,m

Figure 5: Comparison of architectures based on depthwise separable convolution. BN denotes batchnormalization, and Relu is an activation function. expansion ratio. The second sub-layer is a D × D depthwise convolution. Since depthwise convolution cannot change the number of channels, the third sub-layer adjusts the number of channels from tM to N . There is a shortcut connection between the input, and the output of the third sub-layer to facilitate flow of gradient across multiple layers. MobileConvV2 needs tM 2 + D 2 tM + tM N parameters and tHW M 2 + tH W D 2 M + tH W M N FLOPs. ShuffleNet. Zhang et al. (2017) proposed a computation-efficient CNN architecture named Shufflenet. As shown in Figure5(c), each unit of Shufflenet (we call it ShuffleUnit) consists of three sublayers, first group pointwise convolution, depthwise convolution, and second group pointwise convolution, as well as a shortcut. The number of depthwise convolution channels is 1 4 of output channels N . ShuffleUnit uses group convolution in two pointwise convolution layers to reduce the parameters and FLOPs. However, it is hard to exchange information among groups when group convolutions are stacked. To deal with this problem, ShuffleUnit adds a channel shuffle layer after the first pointwise group convolution. The channel shuffle layer rearranges the order of channels. making it possible to obtain information from different groups. The number of groups is represented as g. ShuffleUnit needs 1 4g M N + 1 4 D 2 N + 1 4g N 2 parameters and 1 4g HW M N + 1 4 H W D 2 N + 1 4g H W N 2 FLOPs.

Datasets.

Table content: <table><tr><td>dataset</td><td># of classes</td><td colspan="3">input size # of train # of test</td></tr><tr><td>CIFAR-10 1</td><td>10</td><td colspan="2">32 × 32 × 3 10 × 6000</td><td>10000</td></tr><tr><td>CIFAR-100 2</td><td>100</td><td colspan="2">32 × 32 × 3 100 × 600</td><td>10000</td></tr><tr><td>SVHN 3</td><td>10</td><td>32 × 32</td><td>73257</td><td>26032</td></tr><tr><td>ImageNet 4</td><td colspan="3">1000 224 × 224 × 3 1.2 × 10 6</td><td>150000</td></tr></table>

FALCON and FALCON-branch  gives the best accuracy for similar number of parameters and FLOPs. Bold font indicates the best accuracy among competing compression methods.

Table content: <table><tr><td/><td colspan="2">(a) VGG19-CIFAR10</td><td/><td/><td colspan="4">(b) ResNet34-CIFAR10</td><td/></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param # of FLOPs</td><td>ConvType</td><td/><td colspan="4">Accuracy # of param # of FLOPs</td></tr><tr><td>StConv</td><td/><td>93.56%</td><td>20.30M</td><td>398.70M</td><td>StConv</td><td/><td colspan="2">94.01%</td><td>21.29M</td><td>292.52M</td></tr><tr><td>FALCON</td><td/><td>93.40%</td><td>2.56M</td><td>47.23M</td><td>FALCON</td><td/><td colspan="2">92.78%</td><td>2.63M</td><td>46.33M</td></tr><tr><td colspan="2">FALCON without initialization</td><td>92.88%</td><td>2.56M</td><td>47.23M</td><td colspan="2">FALCON without initialization</td><td colspan="2">92.60%</td><td>2.63M</td><td>46.33M</td></tr><tr><td>FALCON-branch 1.75×</td><td/><td>93.14%</td><td>2.64M</td><td>54.17M</td><td>FALCON-branch 1.625×</td><td/><td colspan="2">92.64%</td><td>2.47M</td><td>58.86M</td></tr><tr><td>EHP-in</td><td/><td>92.06%</td><td>2.56M</td><td>46.41M</td><td>EHP-in</td><td/><td colspan="2">91.73%</td><td>2.62M</td><td>38.41M</td></tr><tr><td>DSC</td><td/><td>91.54%</td><td>2.56M</td><td>48.02M</td><td>DSC</td><td/><td colspan="2">91.54%</td><td>2.62M</td><td>38.41M</td></tr><tr><td>MobileConvV2-0.5</td><td/><td>92.65%</td><td>2.67M</td><td>51.80M</td><td>MobileConvV2-0.5</td><td/><td colspan="2">91.34%</td><td>2.55M</td><td>39.78M</td></tr><tr><td>ShuffleUnit 2×(g=2)</td><td/><td>92.75%</td><td>2.74M</td><td>46.66M</td><td>ShuffleUnit 2×(g=2)</td><td/><td colspan="2">91.74%</td><td>3.08M</td><td>49.78M</td></tr><tr><td>ShuffleUnitV2 1.375×</td><td/><td>92.78%</td><td>2.86M</td><td>58.24M</td><td>ShuffleUnit V2 1.375×</td><td/><td colspan="2">92.16%</td><td>2.98M</td><td>51.30M</td></tr><tr><td/><td colspan="2">(c) VGG19-CIFAR100</td><td/><td/><td colspan="4">(d) ResNet34-CIFAR100</td><td/></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param # of FLOPs</td><td>ConvType</td><td/><td colspan="4">Accuracy # of param # of FLOPs</td></tr><tr><td>StConv</td><td/><td>72.10%</td><td>20.35M</td><td>398.75M</td><td>StConv</td><td/><td colspan="2">73.94%</td><td>21.34M</td><td>292.57M</td></tr><tr><td>FALCON</td><td/><td>71.63%</td><td>2.61M</td><td>47.28M</td><td>FALCON</td><td/><td colspan="2">71.83%</td><td>2.67M</td><td>46.38M</td></tr><tr><td colspan="2">FALCON without initialization</td><td>71.80%</td><td>2.61M</td><td>47.28M</td><td colspan="2">FALCON without initialization</td><td colspan="2">71.80%</td><td>2.67M</td><td>46.38M</td></tr><tr><td>FALCON-branch 1.75×</td><td/><td>73.05%</td><td>2.68M</td><td>54.21M</td><td>FALCON-branch 1.625×</td><td/><td colspan="2">70.26%</td><td>2.54M</td><td>58.93M</td></tr><tr><td>EHP-in</td><td/><td>68.29%</td><td>2.61M</td><td>46.46M</td><td>EHP-in</td><td/><td colspan="2">66.88%</td><td>2.67M</td><td>38.45M</td></tr><tr><td>DSC</td><td/><td>68.18%</td><td>2.61M</td><td>48.07M</td><td>DSC</td><td/><td colspan="2">66.30%</td><td>2.67M</td><td>38.45M</td></tr><tr><td>MobileConvV2-0.5</td><td/><td>72.50%</td><td>2.71M</td><td>51.85M</td><td>MobileConvV2-0.5</td><td/><td colspan="2">65.00%</td><td>2.59M</td><td>39.83M</td></tr><tr><td>ShuffleUnit 2×(g=2)</td><td/><td>72.73%</td><td>2.79M</td><td>46.71M</td><td>ShuffleUnit 2×(g=2)</td><td/><td colspan="2">68.97%</td><td>3.17M</td><td>49.88M</td></tr><tr><td>ShuffleUnit V2 1.375×</td><td/><td>72.32%</td><td>2.91M</td><td>58.29M</td><td>ShuffleUnit V2 1.375×</td><td/><td colspan="2">67.38%</td><td>3.04M</td><td>51.36M</td></tr><tr><td/><td colspan="2">(e) VGG19-SVHN</td><td/><td/><td/><td colspan="3">(f) ResNet34-SVHN</td><td/></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param # of FLOPs</td><td>ConvType</td><td/><td colspan="4">Accuracy # of param # of FLOPs</td></tr><tr><td>StConv</td><td/><td>95.28%</td><td>20.30M</td><td>398.70M</td><td>StConv</td><td/><td colspan="2">94.83%</td><td>21.29M</td><td>292.52M</td></tr><tr><td>FALCON</td><td/><td>95.45%</td><td>2.56M</td><td>47.23M</td><td>FALCON</td><td/><td colspan="2">94.83%</td><td>2.63M</td><td>46.33M</td></tr><tr><td colspan="2">FALCON without initialization</td><td>94.51%</td><td>2.56M</td><td>47.23M</td><td colspan="2">FALCON without initialization</td><td colspan="2">94.75%</td><td>2.63M</td><td>46.33M</td></tr><tr><td>FALCON-branch 1.75×</td><td/><td>94.56%</td><td>2.64M</td><td>54.17M</td><td>FALCON-branch 1.625×</td><td/><td colspan="2">94.98%</td><td>2.47M</td><td>58.86M</td></tr><tr><td>EHP-in</td><td/><td>94.99%</td><td>2.56M</td><td>46.41M</td><td>EHP-in</td><td/><td colspan="2">94.06%</td><td>2.62M</td><td>38.41M</td></tr><tr><td>DSC</td><td/><td>94.37%</td><td>2.56M</td><td>48.02M</td><td>DSC</td><td/><td colspan="2">94.03%</td><td>2.62M</td><td>38.41M</td></tr><tr><td>MobileConvV2-0.5</td><td/><td>93.28%</td><td>2.67M</td><td>51.80M</td><td>MobileConvV2-0.5</td><td/><td colspan="2">93.16%</td><td>2.55M</td><td>39.78M</td></tr><tr><td>ShuffleUnit 2×(g=2)</td><td/><td>93.15%</td><td>2.74M</td><td>46.66M</td><td>ShuffleUnit 2×(g=2)</td><td/><td colspan="2">93.68%</td><td>3.08M</td><td>49.78M</td></tr><tr><td>ShuffleUnit V2 1.375×</td><td/><td>94.36%</td><td>2.86M</td><td>58.24M</td><td>ShuffleUnit V2 1.375×</td><td/><td colspan="2">94.35%</td><td>2.98M</td><td>51.30M</td></tr><tr><td colspan="3">(g) VGG16 BN-ImageNet</td><td/><td/><td/><td colspan="3">(h) ResNet18-ImageNet</td><td/></tr><tr><td>ConvType</td><td>Top-1</td><td colspan="3">Top-5 # of param # of FLOPs</td><td>ConvType</td><td/><td>Top-1</td><td colspan="3">Top-5 # of param # of FLOPs</td></tr><tr><td/><td colspan="2">Accuracy Accuracy</td><td/><td/><td/><td colspan="3">Accuracy Accuracy</td><td/></tr><tr><td>StConv</td><td>73.37%</td><td>91.50%</td><td colspan="2">138.37M 15484.82M</td><td>StConv</td><td colspan="2">69.76%</td><td>89.08%</td><td>11.69M</td><td>1814.07M</td></tr><tr><td>FALCON FALCON without initialization FALCON-branch 1.5× EHP-in DSC MobileConvV2-0.5 ShuffleUnit 2×(g=2) ShuffleUnitV2 1.25×</td><td>71.63% 71.65% 68.24% 70.98% 70.34% 67.80% 70.40% 71.34%</td><td>90.57% 90.47% 88.51% 90.19% 89.71% 87.90% 89.84% 90.34%</td><td>125.33M 125.33M 125.30M 125.33M 125.33M 125.44M 125.77M 125.57M</td><td>1950.75M 1950.75M 1898.39M 1910.56M 1989.49M 2180.49M 2014.73M 2180.65M</td><td>FALCON FALCON without initialization FALCON-branch 1.375× EHP-in DSC MobileConvV2-0.5 ShuffleUnit 2×(g=2) ShuffleUnitV2 1.1875×</td><td colspan="2">66.64% 66.19% 64.01% 66.21% 65.30% 58.99% 65.73% 66.29%</td><td>87.09% 86.86% 85.16% 86.93% 86.30% 81.55% 86.75% 87.32%</td><td>1.97M 1.97M 1.91M 1.96M 1.96M 1.90M 2.22M 2.01M</td><td>395.40M 395.40M 434.44M 336.81M 336.81M 340.06M 438.89M 376.15M</td></tr></table>

Rank-k FALCON further improves accuracy while sacrificing a bit of compression and computation.

Table content: <table><tr><td/><td/><td/><td colspan="2">(a) VGG19-CIFAR10</td><td/><td/><td/><td/><td colspan="2">(b) ResNet34-CIFAR10</td></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param</td><td colspan="2"># of FLOPs</td><td>ConvType</td><td colspan="3">Accuracy # of param</td><td># of FLOPs</td></tr><tr><td>StConv</td><td colspan="2">93.56%</td><td colspan="2">20.30M</td><td/><td>398.70M</td><td>StConv</td><td>94.01%</td><td colspan="2">21.29M</td><td>292.52M</td></tr><tr><td>FALCON-k1</td><td colspan="2">93.40%</td><td/><td>2.56M (7.93×)</td><td/><td>47.23M (8.44×)</td><td>FALCON-k1</td><td>92.78%</td><td/><td>2.63M (8.10×)</td><td>46.33M (6.31×)</td></tr><tr><td>FALCON-k2</td><td colspan="2">93.47%</td><td/><td>4.84M (4.19×)</td><td/><td>94.20M (4.23×)</td><td>FALCON-k2</td><td>93.47%</td><td/><td>5.04M (4.22×)</td><td>88.21M (3.32×)</td></tr><tr><td>FALCON-k3</td><td colspan="2">93.71%</td><td/><td>7.11M (2.86×)</td><td/><td>141.16M (2.82×)</td><td>FALCON-k3</td><td>93.59%</td><td/><td>7.45M (2.86×)</td><td>130.08M (2.25×)</td></tr><tr><td/><td/><td/><td colspan="2">(c) VGG19-CIFAR100</td><td/><td/><td/><td/><td colspan="2">(d) ResNet34-CIFAR100</td></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param</td><td colspan="2"># of FLOPs</td><td>ConvType</td><td colspan="3">Accuracy # of param</td><td># of FLOPs</td></tr><tr><td>StConv</td><td colspan="2">72.10%</td><td colspan="2">20.35M</td><td/><td>398.75M</td><td>StConv</td><td>73.94%</td><td colspan="2">21.34M</td><td>292.57M</td></tr><tr><td>FALCON-k1</td><td colspan="2">71.63%</td><td/><td>2.61M (7.80×)</td><td/><td>47.28M (8.43×)</td><td>FALCON-k1</td><td>71.83%</td><td/><td>2.67M (7.99×)</td><td>46.38M (6.31×)</td></tr><tr><td>FALCON-k2</td><td colspan="2">73.71%</td><td/><td>4.88M (4.17×)</td><td/><td>94.24M (4.23×)</td><td>FALCON-k2</td><td>72.94%</td><td/><td>5.08M (4.20×)</td><td>88.25M (3.32×)</td></tr><tr><td>FALCON-k3</td><td colspan="2">73.41%</td><td/><td>7.16M (2.84×)</td><td/><td>141.21M (2.82×)</td><td>FALCON-k3</td><td>72.85%</td><td/><td>7.49M (2.85×)</td><td>130.13M (2.25×)</td></tr><tr><td/><td/><td/><td colspan="2">(e) VGG19-SVHN</td><td/><td/><td/><td/><td colspan="2">(f) ResNet34-SVHN</td></tr><tr><td>ConvType</td><td colspan="4">Accuracy # of param</td><td colspan="2"># of FLOPs</td><td>ConvType</td><td colspan="3">Accuracy # of param</td><td># of FLOPs</td></tr><tr><td>StConv</td><td colspan="2">95.28%</td><td colspan="2">20.30M</td><td/><td>398.70M</td><td>StConv</td><td>94.83%</td><td colspan="2">21.29M</td><td>292.52M</td></tr><tr><td>FALCON-k1</td><td colspan="2">95.45%</td><td/><td>2.56M (7.93×)</td><td/><td>47.23M (8.44×)</td><td>FALCON-k1</td><td>94.83%</td><td/><td>2.63M (8.10×)</td><td>46.33M (6.31×)</td></tr><tr><td>FALCON-k2</td><td colspan="2">95.72%</td><td/><td>4.84M (4.19×)</td><td/><td>94.20M (4.23×)</td><td>FALCON-k2</td><td>95.34%</td><td/><td>5.04M (4.22×)</td><td>88.21M (3.32×)</td></tr><tr><td>FALCON-k3</td><td colspan="2">95.54%</td><td/><td>7.11M (2.86×)</td><td/><td>141.16M (2.82×)</td><td>FALCON-k3</td><td>94.83%</td><td/><td>7.45M (2.86×)</td><td>130.08M (2.25×)</td></tr><tr><td/><td/><td/><td colspan="2">(g) VGG16 BN-ImageNet</td><td/><td/><td/><td/><td colspan="2">(h) ResNet18-ImageNet</td></tr><tr><td>ConvType</td><td>Top-1</td><td/><td colspan="2">Top-5 # of param</td><td/><td># of FLOPs</td><td>ConvType</td><td>Top-1</td><td colspan="2">Top-5 # of param</td><td># of FLOPs</td></tr><tr><td/><td colspan="3">Accuracy Accuracy</td><td/><td/><td/><td/><td colspan="2">Accuracy Accuracy</td></tr><tr><td>StConv</td><td>73.37%</td><td colspan="2">91.50%</td><td>138.37M</td><td/><td>15484.82M</td><td>StConv</td><td>69.76%</td><td>89.08%</td><td>11.69M</td><td>1814.07M</td></tr><tr><td>FALCON-k1</td><td>71.63%</td><td colspan="2">90.57%</td><td colspan="2">125.33M (1.10×)</td><td>1950.75M (7.94×)</td><td>FALCON-k1</td><td>66.64%</td><td>87.09%</td><td>1.97M (5.93×)</td><td>395.40M (4.59×)</td></tr><tr><td>FALCON-k2</td><td>72.88%</td><td colspan="2">91.19%</td><td colspan="2">127.00M (1.09×)</td><td>3777.86M (4.10×)</td><td>FALCON-k2</td><td>68.03%</td><td>88.26%</td><td>3.22M (3.63×)</td><td>653.00M (2.78×)</td></tr><tr><td>FALCON-k3</td><td>73.24%</td><td colspan="2">91.54%</td><td colspan="2">128.68M (1.08×)</td><td>5604.97M (2.76×)</td><td>FALCON-k3</td><td>69.07%</td><td>88.56%</td><td>4.48M (2.61×)</td><td>910.61M (1.99×)</td></tr></table>

Peer review (iclr2020):
This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon*s main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P*s memory is D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read. Below I have several concerns/suggestions for this paper: 1: Novelty. What is the main difference between this method and all the other tensor decomposition based methods for CNN compression? There are so many tensor decomposition based methods for CNN, and seems Falcon belongs to one of them. The one (maybe) special for Falcon is that it only decomposes along one dimension. Why this method could perform better than other tensor based decomposition methods(some of them are having even smaller memory footprint as they decompose more dimensions) or Falcon could be one special case of it? 2: In Section 3.3 and 3.4, the proposed Falcon and rank-k Falcon seems is fully recovering the original K, see equations above Theorem 2 and above Section 3.5, should it be minimizing the reconstruction error as other tensor decomposition methods? And how to find the solutions P and D from K? how the model is getting trained if using Falcon or rank-k Falcon? Do you have retrain step after the decomposition of K? 3: In the experiment, no any other standard compression techniques such as quantization, low-rank, weight-sharing, sparse, etc are compared. This makes us curious about the benefit of the proposed methods over other methods. In summary, I am mostly worried about the novelty of the paper, and wondering how the model is getting trained, and the comparison with other compression techniques.