FALCON: Fast and Lightweight Convolution for Compressing and Accelerating CNN
Authors: Chun Quan
Jun-Gi Jang
Hyun Dong Lee
U Kang
Venue:  (2020)
Keywords: CNN compression
CNN acceleration
model compression

Abstract: How can we efficiently compress Convolutional Neural Networks (CNN) while retaining their accuracy on classification tasks? A promising direction is based on depthwise separable convolution which replaces a standard convolution with a depthwise convolution and a pointwise convolution. However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without a precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution.

In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads to developing a generalized version rank-k FALCON which further improves the accuracy while sacrificing a bit of compression and computation reduction rates. In addition, we propose FALCON-branch by fitting FALCON into the previous state-of-the-art convolution unit ShuffleUnitV2 which gives even better accuracy. Experiments show that FALCON and FALCON-branch outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8x compression and 8x computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accuracy than standard convolution in many cases, while using a smaller number of parameters and floating-point operations.

Under review as a conference paper at ICLR 2020
FALCON: FAST AND LIGHTWEIGHT CONVOLUTION FOR COMPRESSING AND ACCELERATING CNN
Anonymous authors Paper under double-blind review
ABSTRACT
How can we efficiently compress Convolutional Neural Networks (CNN) while retaining their accuracy classification tasks? A promising direction is based on depthwise separable convolution which replaces a standard convolution with depthwise convolution and a pointwise convolution. However, previous works based on depthwise separable convolution are limited since 1) they are mostly heuristic approaches without precise understanding of their relations to standard convolution, and 2) their accuracies do not match that of the standard convolution. In this paper, propose FALCON, an accurate and lightweight method for com- pressing CNN. FALCON is derived by interpreting existing convolution methods based on depthwise separable convolution using EHP, our proposed mathematical formulation to approximate the standard convolution kernel. Such interpretation leads developing generalized version rank-k FALCON which further improves the accuracy while sacrificing bit of compression and computation reduction rates. In addition, we propose FALCON-branee by fitting FALCON into the previ- ous state-of-the art convolution unit ShuffleUnitV2 which gives even better accu- racy. Experiments show that FALCON and FALCON-brande outperform 1) existing methods based on depthwise separable convolution and 2) standard CNN models by up to 8x compression and 8x computation reduction while ensuring similar accuracy. We also demonstrate that rank-k FALCON provides even better accu- racy than standard convolution in many cases, while using a smaller number parameters and floating-pon maaingg-point operations.
INTRODUCTION
How can we efficiently reduce size and energy consumption of Convolutional Neural Networks (CNN) while maintaining their accuracy classification tasks? Nowadays, CNN is widely used in various areas including computer vision (Krizhevsky et al. (2012;; Simonyan & Zisserman (2014); Szegedy et al. (2017)) natural language processing (Yin et al. (2016)), recommendation system (Kim et al. 6a)), etc. In addition, model compression has become an important technique due to an increase in the model capacity and the number of parameters in CNN. One recent and promising direction for compressing CNN is depthwise separable convolution (Sifre (2044) which replaces standard convolution with depthwise and pointwise convolutions. The depthwise convolution ap- plies a separate 2D convolution kernel for each input channel, and the pointwise convolution changes the channel size using 1x1 convolution (details in Section 2.1 Several recent methods (Howard et al. (2017); Sandler et al. (2018); Zhang et al. (2017)) based on depthwise separable convolution show reasonable performances in terms compression and computation reduction.
However, existing approaches based on depthwise separable convolution have several crucial limi- tations. First, they are heuristic methods, and their relation the standard convolution is not clearly identified. Second, due their heuristic nature, generalizing the methods is difficult. Third, al- though they give reasonable compression and computation reduction, their accuracy is not sufficient compared to that of sanddard-oonvoltion-baaa models.
In this paper, we propose FALCON, an accurate and lightweight method for compressing CNN. FALCON overcomes the limitations of the previous methods based the depthwise separable volution using the following two main ideas. First, we precisely define the relationship between the standard convolution and the depthwise separable convolution using EHP (Extended Hadamard

---

Under review as a conference paper at ICLR 2020
Product), which is our proposed mathematical formulation to correlate the standard convolution ker- nel with the depthwise convolution kernel and the pointwise convolution kemel. We then design FALCON by fine-tume tuning and reordering the results of EHP to improve the accuracy of convolution operations. Second, based on the precise definition, we generalize the FALCON to design rank-k FAI.CON, which further improves accuracy while sacrificing bit of compression and computa- tion reduction rates, We also propose FALCON-brane by fitting FALCON into the state-of-the-art convolution unit ShuffleUnitV2 which gives even higher accuracy. As result, FALCON and FAL- CoN-branch provide superior accuracy compared to other methods based on depthwise separable convolution, with similar compression and computation reduction rates, and rank-k FALCON further improves accuracy, outperforming even the original convolution in many cases. Our contributions are summarized as follows:
Generalization. We analyze and generalize depthwise separable convolution to our pro- posed EHP (Extended Hadamard Product) operation. This generalization enables precise understanding of the relationship between depthwise separable convolution and standard convolution. Furthermore, with fine-tuning operations, it leads to our proposed method FALCON.
Algorithm. We propose FALCON, a CNN compression method based on depthwise sep- arable convolution. FALCON is carefully designed to compress CNN with little accuracy loss. We also propose rank-k FAL.CON to further improve the accuracy with little sac- rifice in compression and computation reduction rates. FALCON can be easily integrated into other architectures, and we propose FALcoN-branch which combines FALCON with branch architecture for better performance. We theoretically analyze the compression and computation reduction rates of FALCON and other competitors.
Experiments. We perform extensive experiments and show that FALCON outperforms other state-of-the-art methods based on depthwise separable convolution for compressing CNN, and 2) provides up 8x compression and computation reduction compared the standard convolution while giving similar accuracies. Furthermore, we show that rank-k FALCON provides even better accuracy than the standard convolution many cases while using smaller number of parameters and floating-pon operations.
The rest of this paper is organized follows. Section Derioonneepaissss preliminaries. Section dectionnJescriiees describes our proposed method FALCON. Section 4 presents experimental results. After discussing related works in Section 5 we conclude in Section
PRELIMINARY
We describe preliminaries on depthwise separable convolution and methods based on depthwise separable convolution. Symbols used in this paper are described in Table of Appendix.
2.1 DEPTHWISE SEPARABLE CONVOLUTION
Depthwise Separable Convolution (DSConv) consists of two sub-layers: depthwise convolution and pointwise convolution. The architecture of each convolution layer in DSConv is illustrated in Fig- ure (a) Depthwise convolution (DWConv) kernel consists of several D D 2-dimensional filters. The number of 2-dimension filters is the same as that of input feature maps. Each filter is applied on the corresponding input feature map, and produces an output feature map. Pointwise convolution (PWConv), known as x 1 convolution, is standard convolution with kernel size 1.
DSCony is defined as follows:
ΣΣ
D k.mmm Jh..ey.
(I)
he' ue
i=1 j=1
M Oz w'm
O Out
Σ
PPP.
(2)
m=1
where Di.j.om and P m.n are depthwise convolution kernel and pointwise convolution kernel, respec-
tively. O' E HR xW'x denotes intermediate feature maps. DSConv performs DWConv
2

---

Under review as a conference paper at ICLR 2020
Output
Output
IM Channels
M Channels
Channel Shuffle
Channel Shuffle
Output
Output
[M Channels
[M Channels
Concat
Concat
Relu
N Channels
BN+ Relu
Channels
M/2 Channels
M/2 Channels
Depthwise
BN+Relu
BN-Relu
Convolution
Depthwise Convolution
Standard Convolution
Channels
Convolution
Standard
M/2 Channels
Pointwise Convolution
Pointwise Convolution
M/2
M/2
Channels
Channels
Channels
Channels
Channels
Channels
Channel Split
Channel Split
Channels
M Channels
Input
Input
Input
Input
(a) Standard Convolution
(b) FALCON (proposed)
(c) StConv-branch
(d) FALCON-branch (proposed)
Figure 1: Comparison of architectures. BN denotes batch-normalizationn Relu and Reluó are activa- tion functions. (a) Standard convolution. (b) Our proposed FALCON. (c) Standard convolution with branch (SiConv-branch). (d) FALcON-branch which combines FALCON with SiConv-branch.
input feature maps 9 Jh,,uy. using equation
and generates intermediate feature maps Oz O'.m".m .m |
Then, DSConv performs PWConv on On wf using equation 2 and generates output feature maps Og sef .n.
2.2
METHODS BASED ON DEPTHWISE SEPARABLE CONVOLUTION
Several CNN methods have been proposed based on Depthwise Separable Convolution (DSConv) recently. DSConv was first introduced by Sifre (2014). Chollet & Franois (2016) built Xception module using DSConv in few layers. Howard al. (2017) built Mobilenet with all convolution layers replaced by DSConv. [Sandler et al. (2018) built MobileNetV2 with inverted bottleneck block, denoted as MobileConv in this paper. Zhang et al. (2017) built CNN model with Shufflenet Unit, denoted as ShuffleUnit in this paper. Ma et al. (2018) improved Shufflenet by designing Shuf- f2enetV2 Unit, denoted as ShuffleUniVV2 in this paper. The architecture and detailed descriptions in Figure and
and Appendix
3 PROPOSED METHOD
We describe FALCON, our proposed method for compressing CNN. We first define Extended Hadamard Product (EHP). key mathematical formulation to generalize depthwise separable con- volution, in Section 3.1 We interpret depthwise separable convolution used in Mobilenet using EHP in Section 3.2 We propose FALCON in Section 3.3 and explain why FALCON can replace standard convolution. Then, we propose rank-k FALCON, which extends the basic FALCON, in Section 3.4 We show that FALCON can be easily integrated into a branch architecture to compress it with little sacrifice of accuracy in Section 3.5 in Appendix C
Finally, we theoretically analyze the performance of FALCON
3.1 EXTENDED HADAMARD PRODUCT (EHP)
We define Extended Hadamard Product (EHP), generalized elementwise product for two operands of different shapes, to generalize the formulation of the relation between standard convolution and depthwise separable convolution. Before generalizing the formulation, we give an example for- mulating the relation between standard convolution and depthwise separable convolution. Suppose we have a 4-order standard convolution kernel E | tion kernel D E xx and pointwise convolution kernel P E A Let Ki,jmm,n be (i. j. m, ni)-th element K. DDi... be (i.j, -h element of DD, and Pm,n be (m, 71)-th element P. Then, it can be shown that applying depthwise convolution with D and pointwise convolution with P is equivalent to applying standard convolution kernel K where K. = D Di.j.m Pm.
3-order depthwise convolu-
(see Section 3.2 for detailed proof).
3

---

Under review as a conference paper at ICLR 2020
ga
D B D
IR
1
D
Og
K B. BRDDDNMNN
ThQAA
Oe
Standard Convolution
Depthwise Separable Convolution used in Mobilenet
(a)
D
!
M
E ND.DDNNN
ROXDNN
BRXM M
N
TT0243)
D
D
D
YTazA
O
Standard Convolution
Convolution rank-k FALCON
Standard Convolution
Convolution of FALCON
(b)
Figure 2: (a) Relation between standard convolution and depthwise separable convolution ex- pressed with EHP. The common correspond to the imput channel-axis of standard convolution. (b) Relation between standard convolution and FALCON expressed with EHP. The common axes correspond to the output channel-axis of standard convolution. (..33)) indicates tensor trans- pose operation to permute the third and the fourth dimensions of tensor. (c) Relation between the standard convolution and rank-k FALCON expressed with EHP.
To formally express this relation, define Extended Hadamard Product (EHP) follows.
Definition 1 (Extended Hadamard Product). Given p-order tensor D E - and q- order tensor E RM X X- Jg-1, the Extended Hadamard Product D @g of D and P is defined
to be the tensor K <R 4 1.....1--1110 100-1Xxxy x...X where the last axis of D and the first axis of P are the common axes such that
K in Ji.. = Din Dip..nd--..
-P Pm.ji mm.j
for all elements of K.
Contrary to Hadamard Product which defined only if the shapes of the two operands are the same, Extended Hadamard Product (EHP) deals with tensors of different shapes. Now, we define a special case of Extended Hadamard Product (EHP) for a third-order tensor and a matrix.
Definition 2 (Extended Hadamard Product for third order tensor and a matrix). Given third- order tensor 'D E R I and matrix P E IR M X the Extended Hadamard Product 'D og P is defined to be the tensor Xx xN where the third axis the tensor D and the first axis of the matrix P are the common axes such that
Ki Dij.m Pm,n-
for all elements J.
We will see that the depthwise separable convolution in Mobilenet can be easily expressed with EHP in Section (3.2] we also propose a new architecture FALCON based on EHP in Section 333 EHP is also a core operation that helps understand other convolution architectures including MobilenetV2 and Shufflenet (see Appendix F.
3.2 DEPTHWISE SEPARABLE CONVOLUTION AND EHP
In this section, we discuss how to represent the convolution layer of Mobilenet as Extended Hadamard Product (EHP) described in Section 3.1 We interpret the depthwise separable convo- lution, which the convolution Mobilenet, as an application of EHP. This interpretation leads to designing better convolution architecture FALCON in Section 3.3
We represent the relationship between standard convolution kernel K DDDxXXX and depth- wise separable convolution consisting of depthwise convolution kernel D = DDDXXM and point- wise convolution kernel P E M using one EHP operation. Figure 2(a) illustrates the rela- tionship between standard convolution and depthwise separable convolution used in Mobilenet. We show that applying depthwise separable convolution with and P is equivalent to applying stan- dard convolution with kernel K which is constructed from D and P.

---

Under review as a conference paper at ICLR 2020
Theorem 1. Applying depthwise separable convolution with depthwise convolution kernel D RD DXM and pointwise convolution kernel ER.. MRM is equivalent to applying standard convo-
lution with kernel K = D
Proof. See Appendix B.1
3.3 FAST AND LIGHTWEIGHT CONVOLUTION (FALCON)
We propose FALCON (FAst and Lightweight CONvolution), novel lightweight convolution that replaces standard convolution. FALCON is an efficient method with fewer parameters and computa- tions than those that the standard convolution requires. In addition, FALCON has better accuracy than competitors while having similar compression and computation reduction rates. The main idea of FALCON is 1) carefully align depthwise and pointwise convolutions, and 2) initialize kernels ing the convolution kernels of the trained standard model. We observe that a typical convolution has more output channels than input channels. In such a setting, performing depthwise convolution after pointwise convolution would allow the depthwise convolution to extract more features from richer feature space: on the other hand, performing pointwise convolution after depthwise convolution as in Mobilenet only combines features extracted from a limited feature space. Based on the observation, FALCON first applies pointwise convolution to generate an intermediate tensor O' R N and then applies depthwise convolution.
We represent the relationship between standard convolution kernel K DxDxMX and FALCON by applying an EHP operation on pointwise convolution kernel P E IR N and depthwise convo- lution kernel D DDDNNN in Figure 2(b) FALCON, the kernel K represented by EHP of D
and P as follows:
K = (3.22433))) (D OF P) S.L. K.j.m,n Pn.m Dijjn-
where (1,33 indicates tensor transpose operation to permute the third and the fourth dimensions of a tensor. Note that the common axis is the output channel axis of the standard convolution, unlike EHP for depthwise separable convolution where the common axis is the input channel axis of the standard convolution.
As in Section 3.2 show that applying FALCON is equivalent to applying standard convolution with a specially constructed kernel.
Theorem 2. FALCON which applies pointwise convolution with kemnel P E N and then depth- wise convolution with kernel D DxXN is equivalent to applying standard convolution with kernel K (T.13443)) OF P).
Proof. See Appendix B.2
Based on the equivalence, we initialize pointwise convolution and depthwise convolution kernels D and P of FALCON by fitting them to the convolution kernels of the trained standard model; i.c.. D.P arg min p. ||K- -T. (I' Og P')[[p. After pointwise convolution and depthwise convolution, we add nonaalitrmaiizaiionn and ReLU activation function as shown in Figure [(b) We note that FALCON significantly reduces the numbers of parameters and FLOPs compared standard convolution, which we discuss at Appendix C
3.4 RANK- FALCON
We propose rank-k FALCON, an extended version of FALCON that improves accuracy while sacrific- ing a bit of compression and computation reduction rates. The main idea is to perform k independent FAI.CON operations and sum up the result. Then, we apply batch-normalization (BN) and Rel.U ac- tivation function to the summed result. Since each FALCON operation requires independent param- eters for pointwise convolution and depthwise convolution, the number of parameters increases and thus the compression and the computation reduction rates decrease; however, improves accuracy by enlarging the model capacity. We formally define the rank-k FALCON with EHP as follows.
Definition 3 (Rank-k FALCON with Extended Hadamard Product). Rank-k FALCON expresses standard convolution kernel K. DDDx>MXN EHP of depthwise convolution kernel D (i)
5

---

Under review as a conference paper at ICLR 2020
R DXDXN and pointwise convolution kernel p( RNM for i 1,2,... ...kk such that
K Σ (12233 OF p()) s.t. Kig.m,n Σ p()) D
n.m 1,j.
Figure ((c) Illustrates the relation between standard convolution and rank-k FALCON. For each i = 1, 2, k, we construct the tensor K( using EHP of the depthwise convolution kernel D and the pointwise convolution kernel p()) Then, we construct the standard kernel K by the element- wise sum of the tensors K(( for all i.
3.5 FALCON-BRARE
FALCON can be easily integrated into CNN architecture called standard convolution operation with a branch (StConv-branch), which consists of two branches: standard convolution on the left branch and residual connection on the right branch (see Figure I(c)). Ma et al. (2018) improved the performance of CNN by applying depthwise and pointwise convolutions on the left branch of StConv- Since FALCON replaces standard convolution, we observe that StConv-branch can be easily compressed by applying FALCON on the left branch.
SiConv-branch first splits input in half along the depth dimension. A standard convolution oper- ation is applied to one half, and no operation to the other half. The two are concatenated along the depth dimension, and output is produced by shuffling the channels of the concatenated tensor. MELCON-branche (see Figure ((d) is constructed by replacing the standard convolution branch (left branch) of StCony-branch with FALCON. Advantages of FAL.CON-rracch
that 1) the branch chitecture improves the efficiency since convolutions are applied to only half of input feature maps and that 2) FALCON further compresses the left branch effectively. FALCON-branch is initialized by fitting FALCON to the standard convolution kerne of the left branch of StConv-branch.
4 EXPERIMENTS
We validate the performance of FALCON through extensive experiments. We aim to answer the following questions:
Q1. Accuracy Compression (Section 4.3) What are the accuracy and the compression
tradeoffs of FALCON, FALCON--ranch, and competitors? Which method gives the best accuracy for given compression rate?
Q2. Accuracy vs. Computation (S.c What are the accuracy and the computation tradeoffs of FALCON. baLchN- and competitors? Which method gives the best accuracy for given amount of computation?
Q3. Rank-k FALCON (Section 4.5). How do the accuracy, the number of parameters, and the number of FLOPs change as the rank & increases in FALCON?
4.1 EXPERIMENTAL SETUP
Datasets.
We perform image classification task on four famous datasets CIFARIO, CIFAR100, SVHN, and ImageNet. Detailed information of these datasets is described in Table
Table 1: Datasets.
dataset CIFAR-10
# classes
input size
of train
of
10
32 x 32 x
10 x 6000
10000
CIFAR-
100
32 32 x 3
100 600
10000
SVHN
10
32 x 32
73257
26032
ImageNe
1000
224 x 224 x
1.2 x 108
150000
https:
http://www.n

---

Under review as a conference paper at ICLR 2020
Models. For CIFARIO, CIFARIOO, and SVHN datasets, choose VGG19 and ResNet34 to evaluate the performance. We shrink the sizes of both models since the sizes of these three datasets smaller than that of Imagenet. In VGG19, reduce the number of fully connected layers and the number of features in fully connected layers: three large fully connected layers (4096-4096- 1000) in VGG19 are replaced with two small fully connected layers (512-10 512-100). In ResNet34, we remove the first 7 x 7 convolution layer and max-pooling layer since the input size (32 x 32) of these datasets is smaller than the input size (224 X 224) of ImageNet. On both models, we replace all standard convolution layers (except for the first convolution layer) with those of FALCON or other competitors in order to compress and accelerate the model. For ImageNet, we choose VGGI6 BN (VGGI6 with batch normalization after every convolution layer) and ResNet18. We use the pretrained model from Pytorch model zoo as the baseline model with standard convolution, and replace the standard convolution with other types of convolutions.
Competitors. We compare FALCON and FALCON-branch with four convolution units consisting of depthwise convolution and pointwise convolution: DSConv, MobileConvV2, ShuffleUnit, and ShuffleUnitV2 (see Figure [S] Section 2.1 and Appendix D for more details), To evaluate the ef- fectiveness of fitting depthwise and pointwise convolution kernels to standard convolution kernel, we build EHP-in which is DSConv where kernels 'D and P are fitted from the pretrained standard convolution kernel K; i..., D, = arg minly'
||3C D' Oe P'I|F
Implementation. We construct all models using Pytorch framework. All the models are trained and tested on GeForce GTX 1080 Ti GPU.
4.2 FITTING CONVOLUTION UNIT INTO MODEL
We evaluate the performance of FALCON against DSConv, MobileConvV2, ShuffleUnit, and Shuf- flcUnitV2. We take each standard convolution layer (StConv) as unit, and replace StConv with those from FALCON or other competitors. We evaluate the classification accuracy, the number of parameters in the model, and the number of FLOPs needed for forwarding one image. We only explain how to apply FALCON in this section. The details of how to fit other convolution units into the models are described in Appendix
FALCON. When replacing StConv with FALCON, we the same setting as that of StConv. Le., if there are BN and ReLU after StConv, we add BN and ReLU at the end of FALCON; if there is only ReLU after StConv, we add only ReLU at the end of FALCON. This is because FALCON is initialized by approximating the StConv kernel using EHP. Using the same setting for BN and ReLU as StConv more efficient for FALCON to approximate the StConv. We initialize the pointwise convolution kernel and the depthwise convolution kernel of FALCON by approximating the pretrained standard convolution kernel using EHP. The approximation process as follows: 1) we first initialize the pointwise convolution kernel and the depthwise convolution kernel randomly, and 2) the pointwise convolution kernel and the depthwise convolution kernel are updated using gradient descent such that the mean squared error of their EHP product and the standard convolution kernel is minimized. Rank-k FALCON uses the same initialization method.
4.3 ACCURACY VS. COMPRESSION
We evaluate the accuracy and the compression rate of FALCON and competitors. Table 2 shows the results on four image datasets. Note that FALCON FALcON-branch provides the highest accuracy in 7 out of 8 cases while using similar or smaller number of parameters than competitors. Specifically, FALCON and MLLCON-brand' achieve up 8x compression rates with less than 1% accuracy drop compared to that of the standard convolution (StConv). Figure 3 shows the tradeoff between accuracy and the number of parameters. Note that FALCON and FAL.CN-braach show the best tradeoff (closest to the "best" point) between accuracy and compression rate, giving the highest accuracy with similar compression rates.
Ablation study. We perform ablation study on two components: 1) the order of depthwise and pointwise convolutions, and 2) initialization. We observe that with a similar number of parameters, 1) FALCON and FALCON without initialization always result in better accuracy than EHP-in and

---

Under review as a conference paper at ICLR 2020
Table 2: FALCON and FALCON-brande gives the best accuracy for similar number of parameters and FLOPs. Bold font indicates the best accuracy among competing compression methods.
VG019-CIRARIO
(b) ResNRet4-CIPAR00
ConyTTpe
Accuracy
param
of
ConvType
Accuracy
of param
of
SiCone
93.56%
20.30M
398.70M
SICom
94.01%
21.29M
292.52M
FALCON
93.40%
2.56M
47.23M
FALCON
92.78%
2.63M
46,33M
FALCON without initialization
92.88%
2.56M
47.23M
FALCON without initalization
92.60%
2.63M
46.33M
FALCON- branch
2.61M
51.17M
FALcON-branch 1.625x
2.47M
58.56M
EHP-in
92.06/8
2.56M
46.41M
EHP-in
91.73%
2.62M
38.11M
DSC
91.54%
2.56M
48.07M
DSC
91. 91.54F5
2.62M
38, 38.41M
MahilaCoesV2.05
92.65%
2.7M
51 80M
MohileCon/VV....
91. 91.34%
2.55M
39.78M
ShufffeUnit 2xcg=22
92.75%
2.74M
46.66
LauffleLii 2x1g=2)
91.74%
3.0M
49.78M
ShuffleUiiV22 1.375
92.78%
2866
58.24M
ShuffleUait 1.375x
92.16%
2.98M
51,30M
VGG19-CIFAR.10
RcNNuSS-CIPARUU0
CyneType
Accuracy
of puram
Ao FLOPS
ConeType
Accuracy
of puram
oo FLOPs
SiConv
72.10%
20.35M
398. 75M
SiCom
73,94/3
21,34M
292.57M
FALCON
71.63%
2.61M
47.28M
FALCON
71.83%
2.67M
46.38M
FALCON without initialization
71,80%
2.6M
47.28M
FALCON without initalization
71.80%
2.67M
46.38M
FALCON MuacoN--racchhtc
73.05%
2.68M
51221M
FALCON-branii 1.625x
2.51M
58.91M
EHP-in
68.29%
2.61
46.46M
EHP-in
66.88%
2.67M
38.45M
DSC
68.18%
2.61
48.07M
DSC
66.30%
2.67M
38.45M
MobileCon/V2-40
72.50%
2.7MM
5185M
MobileCon:V2-055
65.00%
2.59M
39.3M
ShuillUUni 2xco=2)
72.73%
2.79M
46.71M
ShulleUnii 2xxg==2
3.17M
ShofleUni V2 1.375x
72.32%
291M
38.99M
ShofTeeUni V2 1.375x
67.38%
3.04M
$1.36M
VGG19-SVHN
ResNe34-SVHN
ConyType
Accuracy
of param
of FLOPs
CanyType
Accuracy
* param
of FLOPs
SiCome
95.28%
20.30M
398. 70M
SiCom
94,83%
21,29M
292.52M
FALCON
95.45%
2.56M
47.23M
FALCON
94,83%
2.63M
46.33M
without inttialization
94.11%
2.56M
47.23M
withour initiaiiation
2.63M
46.33M
FALcON-branch 1.75x
94.16%
2.64M
FALcON-hranch 1.625x
2,47M
EHP-in
94.99%
2.56M
46.1M
EHP-in
94.06%%
2.62M
38.41M
DSC
94.37%
2.56M
48.02M
DSC
94.03%%
2.62M
38.41M
SathileCon/V--3..
93.28%
167M
51.00
MobileCon-V2-055
93.16%
2.55M
39.78M
ShullleUni 2xc0=22)
93.15%
46.66M
ShulleUnii 2xxgz=2)
3.0MM
ShoMleUnit V2 1,375
94.36%
2866
58.24M
ShofTeeUni V2 1.375x
94.35%
2.9MM
$1.30M
VGGI6 B02I6.BNN-aaee IaxggeN
ResNectM ImaageNet
Top
Top
of PLOPs
Can Type
Tup-1
Top-5
jurm
Ausuracy
Ascusscy
SYCOBy
73.37%
91.50%
139,37M
15456.82
SICony
76%
39.08%
11.69M
1814,07M
FALCON
71,65%
90.57%
125.333
1050-7554
FAT DON
66.64%
87.09%
1.97M
305.40M
withour intoibottion
T1.68'
FALCON withou intiiaioation
66. 19%
86.86%
1.97M
305.40M
13.74/8
93 515
10M
1898.
Faccon-bmuch 1,375x
64.01%
85.16%
191M
434.44M
70.98%
98.19%
125.33M
66.21%
86.93%
1966
316.51M
Stuffee Uni12 1.15x
StuffeeUni 3xige=2)
MrbileCannV2.. V2-0.5
71.36%
10.40%
90.33 30.34%
09.80%
$7.90%
125.43
125
125.7M
2014.730
2150.406
3150.65
DSC
Shulletinnl22 1,1875x
Shufleelin 2x(gm=2
66.29%
58.90 9..9%%
65 73/7
9.9
87.32%
86.75%
16.1
588
2.22M
2.01M
196M
199M
438, 50M
340.06M
316.81M
15M
DSC, respectively, and EiP-in always results in better accuracy than DSC. Furthermore, FALCON results in better accuracy than FALCON without initialization in 6 out of 8 cases. These observations prove our claims in Section that that 1) EHP-out (FALCon) more efficient than EHP-in, and 2) the fitting and the initialization of kernels using EHP improves accuracy. Additionally, we observe that overall performance is more sensitive towards ordering compared to initialization.
4.4 ACCURACY VS. COMPUTATION
We evaluate the accuracy and the amount of computation of FALCON and competitors. We use the number of multiply-adds floating point operations (FL.Os)) needed for forwarding one image to model the metric of computation. Table also shows the accuracies and the number of FLOPs of methods on four image datasets. Note that FALCON or FALCON-branch provide the highest accuracy in out of 8 cases while using similar FLOPs as competitors do. Compared to StConv, FALCON and FALCON- -ranchh achieve up to 8x FLOPs reduction across different models on different datasets. Figure shows the tradeoff between accuracy and the number of FLOPs. Note that FAI.CON and FArcoN-braach show the best tradeoff (closest to the "best" point) between accuracy and computation, giving the highest accuracy with a similar number of FLOPPs
4.5 RANK-K FALCON
We evaluate the performance of rank-k FALCON by increasing the rank i and monitoring the changes in the numbers of parameters and FLOPs. In Table we observe three trends as the rank & increases: 1) the accuracy becomes higher than that of rank-1 FAL.OO, 2) the number of parameters increases,
8

---

Under review as a conference paper at ICLR 2020
FALCON
DSConv
ShuffeuUnt
SIConv
FALCON-Branch
MobileConww
ShufflaUnitzz
BEST
94
BEST
74
Accuracy (%)
Accuracy (%)
Accuracy (%)
Accurse (%6)
93
993
82
70
8 &
91
68
0b
16
20
0
20
15
param (M)
# of param (M)
param (M)
# of peram (M)
(a) CIFAR10
(b)
CIFARIO R
(c) VGGI9-CIFARING 100
(d) ResNet34-CIFAR10
CIFAR100
96
96
70 BEST
95
Accuracy (%)
Accuraty (56)
72
Accuracy (%)
68
Accuracy
8
&
8
93
93
10 15
20
10
15
20
66
125
130
135
8 10
12
# of param (M)
(M)
param (M)
of (M)
(e) VGGI9-SVHN
ResNet34-SVHHI
VGG16-
(h) ResNetl8-ImageNet
Figure 3: Accuracy w.r.t number of parameters on different models and datasets. The three blue circles correspond to rank-1, 2,3 FALCON (from left to right order), respectively. FALCON provides the best accuracy for given number of parameters.
FALCON
DSConv
Shufflelini
FALCON Branch
MobileConwW2
ShuffleUnit2
94mBEST
94
BeeST
74warsy
74
Accuracy (%)
Accuraay (%)
Besurra (%6
93
Accuracy
70
92
66
91
91
2
2a
log2(MFLOPs)
log2/MFLOPs)
log2/MFLOPs)
lgg2(MFLPPs)
(a) VGG19-CIFARIU
(b) ResNet34- OIFIRI
(c) VGGI9-CIFARI00
(d) ResNet34 CIFAR100 ReSNe13-CCFAAR100
96 BEST
BEST
74
74
BEST
Accuracy (%)
95
Accuracy (%)
85
72
Acguracy (%)
A
Accuracy
70
x
66.
94
68
93
2²
2
29
2'
2²
66
2
2
2
2
2'
2
2
Mog2(MFLOPs)
(gg2(MLOPS)
lgg2(MFLOPs)
(og2(MFLOPs)
(e) VGGI9-SVHN
ResNe134-SVHN
VGG16-ImageNet
(h) BesNeel8-IImgeNee
Figure 4: Accuracy w.r.t. FLOPs on different models and datasets. The three blue circles correspond
to rank- 1, 2. 3 FALCON (from left to right order), respectively. FALCON provides the best accuracy for given number of FLOPs.
and 3) the number of floating point operations (FLOPs) increases. Although the rank A: that gives reduction varies, rank-k FALCON improves
the best tradeoff of rank and compression/computatie the accuracy of FALCON in all cases. Especially, we note that rank FALCON often gives even higher accuracy than the standard convolution, while using smaller number of parameters and FLOOPs. For example, rank-3 FALCON applied to VGG19 on CIFARION 100 dataset shows 1.31 percentage point higher accuracy compared to the standard convolution, with 2.8x smaller number of parameters and 2.8x smaller number of FLOPs. Thus, rank-k FALCON is a versatile method to further improve the accuracy of FALCON while sacrificing bit of compression and computation.
5 RELATED WORK
Over the past several years, a lot of studies focused on compressing and accelerating DNN to reduce model size, running time, and energy consumption.
believed that DNNs are over parameeeeeeee.... Weight-sharing (Han et al. (011)): Ullrich et al.
2017) Chen et al. 2015 Choi et al. (201)); Agustsson al. (2017) is common compression

---

Under review as a conference paper at ICLR 2020
Table 3: Rank-k FALCON further improves accuracy while sacrificing a bit of compression and computation.
VOGISC
ReNNaSS-CHARIO
ConvTyge
Accuracy
of param
PLOIS
ConcType
Accurazy
of persan
of
90.56%
20.30M
94.01%
292.50M
FALCON-
93.40%
(7,93x)
47.23M
FALCOX-K]
92.78%
2.63M
(8.10x)
PALCIONK2
93.47%
4.84M
(4.19%)
94.20M
(4.23X)
PALCUN-K2
93.4%%
5.4M
(4,22%)
98.20M
(3,22x)
PADCON-K3
93.71%
7.1MM
(2.86x)
141.16M
(2.82x)
PALCOO-K3
93.59%
7,45M
(2.86k)
130.0MM
(2.25%)
Resher34 CIEARIOU
ConyType
Accuracy
param
of PLOPS
Con-Type
Accuracy
param
FLOPs
SeCaow
72.105
20.35M
398.75M
SoCoow
73.94%
21.34M
292.57M
71.63%
(3.00x)
47.28M
(843x)
PALDONNK1
71.83%
2.67M
(7.9%))
(631X)
BALCON-K2 k2
73.71%
4.88M
(1117)
91.21M
(623x)
FALCOX-K2
72.94%
5.08M
(4.20))
8K:25M
(3,52x)
PALCON-KS
73.41%
7.16M
(288x)
141.21M
(282X)
PALCEX-KS
72.85%
7,4MM
(2.85)
130.EM
(2:25x)
VOGI9-VHN
ConvTyge
Accuracy
of porm
of FLOP
ConeType
Accuracy
param
FLOPS
SoCoov
95.28%
20.30M
298.70M
SoCooy
94.83%
21.29M
292.52M
FALCON kl
95.45%%
2.56M
(7,93x)
47.23M
(8.44x)
PALCOX-Ki k]
94.83%
263M
(8.10x)
46.32M
(6.31x)
PALCCON-22
95.72%%
4.8-M
(411%)
94.20M
(4.23x)
PALCDX-K2
95. 34%
5.44M
(4.22%)
88.21M
(3,325)
PALCON- K3
95.54%
7.I1M
(2.86 86x)
14116M
(2.82))
PALCOX-13
94.83%
7.45M
(2,86x)
130.0MM
(2.25%)
BN-asaadNa
Con/Tyye
Top-1
Top
POCELOP
Con/Type
Top-1
Top-5
Acorayy
Accuncy
Accaney
Accuracy
StCans
73.37%
50%
18837M
1548433M
S1Or
89.08%
11.90M
1814,7MM
FALCON
31,65%
90.57%
325
FALDON
64.6
87,09P
197M
15,93K)
295,4004
(4.59k)
72.88%
195
182
(1 (1))))
1177 85M
DON-67
56 01%
88 66.76/2
527M
(561-)
651.00
(2.8k))
Patcoos-3
73.24/
91 549
(1 (1))
9MH49TM
(2.70k)
FALDON 123
59007
88,566
2.3M
(2,61x)
910.11M
(1.9k))
method which stores only assignments and centroids of weights. While using the model, weights are loaded according to assignments and centroids. Pruning (Han et al.] (2014); Li et al. (2016)
aims removing useless weights setting them to zero. Although weight-sharing and pruning can significantly reduce the model size, they are not efficient in reducing the amount of computation.
Quantizing (Courbariaux et al. (2155 2016); [Hou et al.. (2017); /Zhu al. (2017)) the model into
binary or ternary weights reduces model size and computation simultaneoush replacing arithmetic operations with bit-wise operations remarkably accelerates the model.
Layer-wise approaches are also employed to efficiently compress models. A typical example of such approaches is low-rank approximation (Lebedev al. (2013); Kim al. (2016b); Novikov ett al. (2015)); it treats the weights as tensor and uses general tensor approximation methods to compress the tensor. To reduce computation, approximation methods should be carefully chosen, since some of approximation methods may increase computation of the model.
Compressing existing models has limitations since they are originally designed to be deep and large to give high accuracy. A recent trend is to design brand new architecture that is small and effi- cient. Mobilenet (Howard et al. (2117)) MobilenetV2 (Sandler et al. (2018)) Shufflenet (Zhang et al. 2017)) and V2ufffenet (Ma et al. (2018)) are the most representative approaches, and they use depthwise convolution and pointwise convolution as building blocks for designing convolution layers. Our proposed FALCON gives thorough interpretation of depthwise convolution and point- wise convolution, and applics them into model compression, giving the best accuracies with regard to compression and computation.
6 CONCLUSION
We propose FALCON, an accurate and lightweight convolution method to replace standard convolu- tion. By interpreting existing convolution methods based on depthwise separable convolution using EHP operation, FALCON and its general version rank-k FAL.CON provide accurate and efficient com- pression on CNN. We also propose FALCON-branch, variant of FAL.CON integrated into a branch architecture of CNN for model compression. Extensive experiments show that FALCON and its vari- ants give the best accuracy for given number of parameter or computation, outperforming other convolution models based on depthwise separable convolution. Compared to the standard convo- lution, FALCON and FaLcoN-branch give up to 8x compression and 8x computation reduction while giving similar accuracy. We also show that rank-k FALCON provides better accuracy than the standard convolution, while using smaller numbers of parameters and computations.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Lue Van Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. Neural Information Processing Systems (NIPS), 2017.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. International Conference Machine Learning (ICML), 2015.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. International Conference Learning Representations (ICLR), 2017.
Chollet and Franois. Xception: Deep learning with depthwise separable convolutions, 2016. URL
http://arxiv org/ abs/1600. 02357 cite arxiv:1610.02355
Matthieu Courbariaux, Yoshua Bengio, and Jean- David. Binaryconnect: Training deep neu- ral networks with binary weights during propagations. Neural Information Processing Systems (NIPS), 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training neural networks with weights and activations constrained to +1 or -1. Neural Information Processing Systems (NIPS), 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. Neural Information Processing Systems (NIPS), 2014.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Lu Hou, Quamming Yao, and James T. Kwok. Loss-aware binarization of deep networks. Interna- tional Conference on Learning Representations (ICLR), 2017.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/ 048661, 2017. 2, 5, 6. 2017.
Dong Hyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. Convolutional matrix factorization for document context-aware recommendation. In Recsys, 2016, pp. 233-240, 2016a.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of deep convolutional neural networks for fast and low power mobile applications. In- ternational Conference on Learning Representations (ICLR), 2016b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Aunual Conference on Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pp. 110.p...11.1.1111111111111111110 1106-1114, 2012.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. International Conference on Learning Representations (ICLR), 2015.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations (ICLR), 2016.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines
for efficient CNN architecture design. CoRR, abs/1880.111 100.11164,, 2018. URL http::/arxiv.
org/ hatg/ab//ie bbs//807..m 11164]
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. Neural Information Processing Systems (NIPS),
2017.
11

---

Under review as a conference paper at ICLR 2020
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing neural networks. Neural Information Processing Systems (NIPS), 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Laurent Sifre. Rigid-motion scaltering for image classification. PhD thesis, cole Polytechnique, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 1491.1095666 2014.
wwwpsiiivorrraaaao http//bb//1100 1556
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017, pp. 4278- 4284, 2017.
Karen Ullrich, Edward Meeds, and Max Welling, Soft weight-sharing for neural network compres- sion. International Conference on Learning Representations (ICLR), 2017.
Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. ABCNN: attention-basee based convolu- tional neural network for modeling sentence pairs. TACL, 4:259-272, 2016.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient
convolutional neural network for mobile devices, 2017. URL http://arxiv www.xiii.rrgabs// 1.707.0088
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. Interna- tional Conference on Learning Representations (ICIR), 2017.
Table 4: Symbols.
Symbol
Description
K
convolution kernel of size I DXDXMXN input feature maps of size I XXXXM output feature maps of size H''xW">x height and width of kernel (kernel size) number of input feature map (input channels)
O
D
M
number of output feature map (output channels) height of input feature map width of input feature map height of output feature map width of output feature map stride
H
H'
W
padding
p.
Extended Hadamard Product (EHP) expansion ratio in Mobilenet
t
number of groups in Shufflenet
A CONVOLUTIONAL NEURAL NETWORK
Convolutional Neural Network (CNN) is a type of deep neural network used mainly for structured data. CNN uses convolution operation in convolution layers. In the following, discuss CNN when applied to typical image data with RGB channels.
Each convolution layer has three components: input feature maps, convolution kernel, and output feature maps. The input feature maps I E Hx XXM and the output feature maps O E H' x x are 3-dimensional tensors, and the convolution kernel K E R
RD DxxxMM xN is a 4-dimensional tensor.
The convolution operation is defined
D
D
M
O
Σ
K.j.mmnn Jha.wy.m
(3)
1=
m=1
12

---

Under review as a conference paper at ICLR 2020
where the relations between height h, and width wy of input, and height h and width w' of output are as follows:
h, = (A' 1)s + --p and wy = (w' - 1)s --
(4)
where s is the stride size, and p is the padding size. The third and the fourth dimensions of the convolution kernel K must match the number M of input channels, and the number N of output channels, respectively.
Convolution kernel K can be kernel K performs convolution operation while sliding over all spatial locations on input feature maps, Each filter produces one output feature map.
N 3-dimensional filters F, RDXDXM Each filter F,, in
B PROOFS OF THEOREMS
B.1 PROOF OF THEOREM 1
Proof. From the definition of EHP, j.j...m.. = Dij.m P Pm,.- Based on equation we replace the kernel Kijaamn with the depthwise convolution kernel Dij,an and the pointwise convolution kernel P Pm.n-
D
D
M
O w su
ΣΣΣ
D Dig.m
P Pm, Jna.us.
i=1 j=1
m=1
where my. is the (lis, wj, m) entry of the input. We split the above equation into the follow- ing two equations.
O₂ h''
D Dij.m
- Thaaey.m
(5)
M O'. wm.
Ok ar
P m==
Σ Pm..'
(6)
where O/ amm. E x'''X" is an intermediate tensor. Note that equation and equation correspond to the depthwise convolution and the pointwise convolution, respectivery. Therefore, the output O₂ auf,m an is equal to the output applying depthwise separable convolution used in Mobilenet.
B.2 PROOF OF THEOREM 2
Proof. From equation we replace the kernel Ki,..m,n with the pointwise convolution kernel P and the depthwise convolution kernel D.
M D
D
ΣΣΣPET
Pm, Dig.n Jh..
m=1 2=1 j=1
where Jn,,,, is the (he, wj. m)-th entry of the input J. We split the above equation into the following two equations.
O R. in
P mmn Ihires.
D
Oue and
ΣΣ Dijj. Oh.,ue,, 1=1
where J, O', and O are the input, the intermediate, and the output tensors convolution layer, respectively. Note that equation and corresonnoor to pointwise convolution and depthwise convolution, respectively. Therefore, the output O On wn is equal to the output applying FALCON.
13

---

Under review as a conference paper at ICLR 2020
C QUANTITATIVE ANALYSIS
In this section, we evaluate the compression and the computation reduction of FALCON and rank-k FALCON. All the analysis is based on one convolution layer. The comparison of the numbers of parameters and FLOPs of FALCON and other competitors is in Appendix G
C.I FALCON
We analyze the compression and the computation reduction rates of FALCON in Theorems and Theorem 3. Compression Rate (CR) of FALCON is given by
# of parameters standard convolution
DO
CR
# of parameters in FALCON
MN + D2N
where D is the size of standard kernel, M is the number of input channels, and N is the number of output channels.
Proof. Standard convolution kernel has D2 D parameters. FALCON includes pointwise convolu- tion and depthwise convolution which requires M. and D parameters, respectively, Thus, the
D
compression rate of FALCON is CR= MN + D2N
Theorem 4. Computation Reduction Rate (CRR) of FALCON described
# of FLOPs in standard convolution
CRR # of FLOPs in FALCON
D2N
HWMN + H'W' D
where H' and W" are the height and the width of output, respectively, and H and W are the height and the width of input, respectively.
Proof. The standard convolution operation requires H'W'D FLOPs Molchanov et al. (217 FALCON includes pointwise convolution and depthwise convolution. Pointwise convo- lution has kernel size D = with stride s = and no padding, so the intermediate tensor O' has the same height and width as those of the input feature maps. Thus, pointwise convolution needs HW MN FLOPs. Depthwise convolution has the number of input channel M = 1, so it needs H'W' D2 N FLOPS. The total FL.OPs of FALCON is HW N H'W' D2N, thus the computation H'W' D M
reduction rate of FALCON is CRR HWMN H'W D
C.2 RANK-K FALCON
We analyze the compression and computation reduction rates of rank-k FAL.CON in Theorem Theorem' 5. Compression Rate (C'Rg) and Computation Reduction Rate (CRRg) of rank-k FAL- CON are described as:
CR k
CRR = CRR k
CR²y
Proof. The numbers of parameters and FLOPs increase for k times since rank-k FALCON duplicates FALCON for times. Thus, the compression rate and the computation reduction rate are calculated
CR as CR and CRR k
CRR
k
D DESCRIPTION OF RELATED CONVOLUTION UNITS
Mob. Sandler et al. (001 proposed a new convolution architecture which we call as MobileConvVVs V2, their Mobilenet V2 model. MobileConvV2 consists of three sub-layers shown in Figure [(b] The first and the third sub-layers are pointwise convolution for adjusting the num- ber of channels. The first sub-layer expands the number of channels from M to t.M. where & is an
14

---

Under review as a conference paper at ICLR 2020
Output
Output t
I Charnels
Output
Shuffle
Add
Channels
Output
Add
Channels
Concat
BN+ Relu
N Channels
BN Í Channets
Group Convolution
BNHRlll
W2 Channels
Pointwise
Polntwise
BN N44 Channels
Conrelltiio
Pointviie
Convolution
Depthwise Convolution
Convolution
BN+ Relu6 1M Channels
Deythwise
Channels
Depthwise
BN+Relu M2 Channels
Convolution
Depthwise
BN Relu6 IM Channets
Convolution
Channel Shuffle
Convolution
Paintwise
M12
Channels
Convolution
Pointwise
BN+
[N4 Channels
M12 Channels
M Channels
Convolution
Group Convolution
Split
1M Channels
1 Channois
Input
Input
Input
Input
(a) Mobilenet
(b) MobilenetV2
(c) Shufflenet (Zhang (d) Shufflenet22 (Ma al.
Howard et al.
Sandler et al. 2018))
et al. (017))
(208))
(2077))
Figure 5: Comparison of architectures based on depthwise separable convolution. BN denotes batch- normalization, and Relu is an activation function.
expansion ratio. The second sub-layer is D D depthwise convolution. Since depthwise convolu- tion cannot change the number of channels, the third sub-layer adjusts the number of channels from tM to N. There is shorteut connection between the input, and the output of the third sub-layer to facilitate flow of gradient across multiple layers. MobileCony needs LM² + DPLM + (M parameters and LHW M LH'W LH'W' FLOPs.
NhuffleNet. Zhang et al. (2017) proposed a computation-eefficer CNN architecture named Shuf- flenet. As shown in Figure 5(c) each unit Shufflenet (we call it Shufflcnnit consists of three sub- layers, first group pointwise convolution, depthwise convolution, and second group pointwise convo- lution, as well shortcut. The number of depthwise convolution channels is 4 of output channels N. ShuffleUnit uses group convolution in two pointwise convolution layers to reduce the parameters and FLOPs. However, is hard exchange information among groups when group convolutions are stacked. To deal with this problem, ShuffleUnit adds channel shuffle layer after the first pointwise group convolution. The channel shuffle layer rearranges the order of channels. making it possible obtain information from different groups. The number of groups is represented g. ShuffleUnii N parameters and HW MN + 1 H'''
needs ig
MN + D2N
H'W'N FLOPs.
Ag
ShufflenetV2. Ma et al.
2018) proposed practically efficient CNN architecture ShufflenetV2. As shown in Figure dd each unit of ShufflenetV2 (we call it ShufileUnttV2) consists of two branches. The left branch consists of two pointwise convolutions and one depthwise convolution like MobileConv V2, and the right branch is an identity operation. Note that outputs of both branches maintain the number of channels as M/2. The final output is produced by concatenating and shuf- fling the output tensors from both the branches. ShuffleUnitV2 needs &/ {MM² + D²M) parameters and 2 HW (M2 DD²) FLOPs.
E GENERALITY OF EHP
We show that EHP key operation understand other convolution architectures based on depth- wise separable convolution.
MobilenetV2. As shown in Figure 5(b) MobilenetV2 has an additional pointwise convolution before depthwise convolution in Mobilenet: one layer of MobilenetV2 consists of two pointwise convolutions and one depthwise convolution. In another point of view, MobilenetV2 can under- stood as FALCON followed by additional pointwise convolution; i.c., MobilenetV2 performs EHP operation as FALCON does, and performs additional pointwise convolution after that.
Shufflenet. As shown in Figure 5(c) Shufflenet consists of depthwise convolution and pointwise group convolution which is a variant of pointwise convolution. We represent the convolution layer of Shufflenet using EHP as follows. g be the number of groups. We divide the standard convolution
15

---

Under review as a conference paper at ICLR 2020
kernel K DDDDXMX into 9 group standard convolution kernels. Then, the relation of g-th x with regard to g-th depthwise convolution and g-th pointwise group convolution kernel P RR is
group standard convolution kernel K IR kernel DD' E IR Dx M
N DD OF P s.t.
K.
D s.jmy
P9 my.
M.
where mg 1,2, and By = 1,,, Each group standard convolution equivalent to the combination depthwise convolution and a pointwise convolution, and thus easily expressed with EHP as in Mobilenet.
Therefore, each layer of Shufflenet equivalent to the layer consisting one group convolution followed by standard convolution.
ShufflenetV2. As shown in Figure the left branch of ShufflenetV2 has the same convolutions in MobilenetV2: consists of two pointwise convolutions and one depthwise convolution. Like Mobilenet the left branch of Shufflenet can understood as FALCON followed by additional pointwise convolution.
F FITTING OTHER CONVOLUTION UNITS INTO MODELS
DSConv. DSConv (shown Figure 5(a) has the most similar architecture as FALCON among competitors, and thus DSConv has nearly the same number of parameters that of FALCON. As in the setting of FALCON, the existence of BN and ReLU at the end of DSConv depends on that of StConv.
MobileConv V2. In MobileConvV2 architecture (shown in Figure 5(b) we adjust the numbers of parameters and FLOPs by changing the expansion ratio & as described in Appendix D which rep- resented as 'MobileConvV22 FALCON, since two pointwise convolutions bring lots parameters and FLOPs to MobileConvV2
We choose i = 0.5 as the baseline MobileConv V2 to compare with
ShuffleUnit. In ShuffleUnit (shown in Figure 5(c) we adjust the numbers of parameters and FLOPs by changing the width multiplier a (Howard et al. (2017)) and the number of groups g. which is represented as 'ShuffleUnit a xg=g))". Note that the width multiplier is used to adjust the number of input channels M and the number of output channels N of a convolution layer; if the width multiplier is a, the numbers of input and output channels become aM and a.N, respectively. While experimenting with ResNet, we find that ShuffleUnit does not cooperate well with ResNet: ResNet34 with ShuffleUnit does not converge. We suspect that residual block and ShuffleUnit may conflict with each other because of redundant residual connections: the gradient may not find the right path towards previous layers. For this reason, we delete the shortcut of all residual blocks in ResNet34 when using ShuffleUnit.
ShuffleUnitV2. In ShuffleUnitV2 (shown in Figure 5(d) we also adjust the number of parameters and FLOPPs by changing the width multiplier o, which is represented as ShufffeUnitV2 ax"". Other operations of ShuffleUnitV2 stay the same as in Ma et al. (2018).
G PARAMETERS AND FLOPS
We summarize the numbers of parameters and FLOPSs for FAL.CN and competitors in Table
16

---

Under review as a conference paper at ICLR 2020
Table 5: the numbers parameters and FLOPs of FALCON and competitors. Symbols are described in Table
Convolution
# of parameters
# of FLOPS
FALCON
MN + D
HWMN HWWWWW'' DAN .HWM 1HWD'M M HWD'M MN
FALcoN-branch
1 DD'M
DSConv
MN + D'M
MobilenetV2
tM + tDRM tMN
HIW + D + MHWW
Shufflenet
2 JCMN + D&N + )
LOWWM + DAW + WWWNN)
ShufllenetV2
H(M2 + DRM)
HHW DAM)
SiConv-branch
DP
HW MP DN
Standard convolution
DNN
17


Reviews:
[{"review_id": "r1gYSFBpKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a CNN compression method, based on the so called EHP operation, which can be used to  analyze and generalize depthwise separable convolution. Based on EHP, the paper develops depthwise separable convolution to compress CNNs, and extend it to a rank-k approach with further improved accuracy. Some analysis is provided about the operation equivalence. The experiments on standard benchmark datasets show the effectiveness of the method. \n\nProbably the most important contribution of this work is that it proposes a new operation that can summarize/generalize the existing depthwise separable convolution and reveal the relationship with the standard convolution. The work is meaningful to me because a compact representation of CNNs can largely reduce the computational resources and storage.  \nThe experiments are extensive as well. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1eucsM15S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThe paper is dedicated to studying fast and lightweight convolution for efficient compression and retaining original accuracy. In this paper, the authors interpret existing convolution methods based on depthwise separable convolution and derive FALCON. They claim their FALCON mathematically approximate the standard convolution kernel and achieves a better TA/efficiency tradeoff. They conduct extensive experiments to show that FALCON based method 1) outperforms previous state-of-the-art methods; 2) achieve 8X efficiency while ensuring similar TA.\n\nStrength Bullets:\n1. The authors give a detailed interpretation of the depthwise separable convolution method via EHP. Then they propose novel FALCON has better accuracy than competitors while having similar compression and computation reduction rates.\n2. The paper is well organized and easy to read. The author conducts extensive experiments to show FALCON based method not only surpass other depthwise separable convolution models but also give up to 8 times efficiency while giving similar accuracy.\n\nWeakness Bullets:\n1. The authors need to provide more ablation experiments for two components of FALCON: 1) align depthwise and pointwise convolution; 2) initialize kernels. Moreover, for the order of performing depthwise convolution and pointwise convolution, it also needs experimental support.\n2. I would like to see a comparison of the compression rate with previous filter compression methods. i.e, Soft weight-sharing (ICLR'17), Deep K-means (ICML'18) and so on.\n3. [Minor] Even if there is computation reduction (Flops), it is not interesting enough. I am very curious about how much efficiency gains (i.e. latency) in the software implementation of FALCON. \n\nRecommendation:\nAlthough there are a few flaws in experiment design, it still is a novel and good technique. This is a weak accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgHOuxVqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a model compression method: Falcon and rank-k Falcon. Both are used to compress CNN type of models by replacing standard convolution layer with a compact Falcon or rank-k Falcon layer to compress the model. Falcon's main idea is to decompose the traditional convolution kernel K into two smaller tensors, one is depthwise convolution kernel D and pointwise convolution kernel P. And DP will reconstruct the original kernel K. Since D+P's memory is  D*D*M+N*M which is smaller than the original size D*D*M*N, and thus when N is large, the memory saving could be large. The paper is in general in good writing and very easy to read.\n\nBelow I have several concerns/suggestions for this paper:\n\n1: Novelty. What is the main difference between this method and all the other tensor decomposition based methods for CNN compression? There are so many tensor decomposition based methods for CNN, and seems Falcon belongs to one of them. The one (maybe) special for Falcon is that it only decomposes along one dimension. Why this method could perform better than other tensor based decomposition methods(some of them are having even smaller memory footprint as they decompose more dimensions) or Falcon could be one special case of it? \n\n2: In Section 3.3 and 3.4, the proposed Falcon and rank-k Falcon seems is fully recovering the original K, see equations above Theorem 2 and above Section 3.5, should it be minimizing the reconstruction error as other tensor decomposition methods? And how to find the solutions P and D from K? how the model is getting trained if using Falcon or rank-k Falcon? Do you have retrain step after the decomposition of K?\n\n3: In the experiment, no any other standard compression techniques such as quantization, low-rank, weight-sharing, sparse, etc are compared.  This makes us curious about the benefit of the proposed methods over other methods.\n\nIn summary, I am mostly worried about the novelty of the paper, and wondering how the model is getting trained, and the comparison with other compression techniques.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to accelerating convolutional networks. The framework is related to depthwise separable convolutions. The reviews are split. R3 expresses concerns about the experimental evaluation and results. The AC agrees with these concerns. The AC also notes that the submission is 10 pages long. Taking all factors into account, the AC recommends against accepting the paper.
Decision: Reject