Block-wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions
Authors: Yan-Jen Huang
Hsin-Lung Wu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: convolutional neural network
block-wise separable convolution
network architecture search

Abstract: Convolutional neural networks (CNNs) have demonstrated great capability of solving various computer vision tasks with nice prediction performance. Nevertheless, the higher accuracy often comes with an increasing number of model parameters and large computational cost. This raises challenges in deploying them in resource-limited devices. In this paper, we introduce block-wise separable convolutions (BlkSConv) to replace the standard convolutions in order to compress deep CNN models. First, BlkSConv expresses the standard convolutional kernel as an ordered set of block vectors each of which is a linear combination of fixed basis block vectors. Then it eliminates most basis block vectors and their corresponding coefficients to obtain an approximated convolutional kernel. Moreover, the proposed BlkSConv operation can be efficiently realized via a combination of pointwise and group-wise convolutions. Thus the constructed networks have smaller model size and fewer multiply-adds operations while keeping comparable prediction accuracy. However, it is unknown how to search a qualified hyperparameter setting of the block depth and number of basis block vectors. To address this problem, we develop a hyperparameter search framework based on principal component analysis (PCA) to help determine these two hyperparameters such that the corresponding network achieves nice prediction performance while simultaneously satisfying the constraints of model size and model efficiency. Experimental results demonstrate the prediction performance of constructed BlkSConv-based CNNs where several convolutional layers are replaced by BlkSConv layers suggested by the proposed PCA-based hyperparameter search algorithm. Our results show that BlkSConv-based CNNs achieve competitive performance compared with the standard convolutional models for the datasets including ImageNet, CIFAR-10/100, Stanford Dogs, and Oxford Flowers.

Block-wise Separable Convolutions: An Alternative Way to Factorize Standard Convolutions
Anonymous Author(s)
Affiliation
Address
email
Abstract
Convolutional neural networks (CNNs) have demonstrated great capability of solv- ing various computer vision tasks with nice prediction performance. Nevertheless, the higher accuracy often comes with an increasing number of model parameters and large computational cost. This raises challenges in deploying them resource- limited devices. In this paper, we introduce block-wise separable convolutions (BIkSConv) replace the standard convolutions in order to compress deep CNN models. First, BIkSConv expresses the standard convolutional kernel as an ordered set of block vectors each of which is a linear combination of fixed basis block vectors. Then it eliminates most basis block vectors and their corresponding coef- ficients obtain an approximated convolutional kernel. Moreover, the proposed BIkSConv operation can be efficiently realized via combination of pointwise and group-wise convolutions. Thus the constructed networks have smaller model size and fewer multiply-adds operations while keeping comparable prediction accu- racy, However, it is unknown how to search qualified hyperparameter setting of the block depth and number of basis block vectors. To address this problem, we develop a hyperparameter search framework based on principal component analysis (PCA) to help determine these two hyperparameters such that the responding network achieves nice prediction performance while simultaneously satisfying the constraints of model size and model efficiency. Experimental results demonstrate the prediction performance of constructed BikSConv- CNNs where several convolutional layers replaced by BIkSConv layers suggested by the proposed PCA-based hyperparameter search algorithm. Our results show that BikSConv-based CNNs achieve competitive performance compared with the standard convolutional models for the datasets including ImageNet, CIFAR-10/100, Stanford Dogs, and Oxford Flowers.
10
11
12
14
16
18
20
22
23
24
25
Introduction
26
27
In the past decade, Deep Learning (DL) has been the basis of many successes in artificial intelligence, including a variety of applications in computer vision, reinforcement learning, and natural language 29 processing. One the most popular deep neural networks is Convolutional Neural Network (CNN).
28
30 With the help of various techniques such as residual connections and batch normalization, it is easy to
31 train deep CNNs with many layers on powerful GPUs. While large-scale CNN models have achieved great successes, they require huge computational complexity and massive storage. For example,
33 VGG16 (27) has 138 million parameters and requires 1554700 million multiply-add operations (MAdds) to classify an image. It is great challenge to deploy them in real-time applications, especially on devices with limited resources such as mobile phones and embedded systems. Thus, the prediction models are required be compact and fast while keeping acceptable accuracy. The main approach to compact is the model compression which aims at establishing tradeoff between
35
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38
model efficiency and accuracy. In the area of model compression, methods to construct efficient and compact CNNs are mainly divided into two approaches: one approach to compress trained CNNs and the other approach is to design new compact CNNs and train them from scratch. Many works based on the first approach suggested several techniques such as quantization (33) model pruning (6) 24), Huffman coding (6), and low rank factorization (12).
39
40
41
42
43
Studies in the second approach mainly explored many ways for factorizing convolutions. For instance, Szegedy et al. (30) improved GoogLeNet (29) through factorizing convolutions with larger spatial filters by a two-layer convolutional architecture with smaller spatial filters. At present, most factorizing methods are usually performed via a combination of depthwise convolution, pointwise convolution, and groupwise convolution. For example, in (25). the depth- separable convolutions (DSCs) were proposed where the standard convolution is decomposed into a depth-wise convolution and a pointwise convolution. The ShuffleNets (36 18) utilizes pointwise group convolution with channel shuffle to decompose the standard convolution. Moreover, many lightweight models based
44
45
48
47
48
49
50
51
on DSCs or groupwise convolutions such as MobileNets & 23 7
and ShuffleNets (36) [8) were
proposed to greatly reduce computation cost while maintaining accuracy.
53
In this paper, we follow the research path of the second approach and propose block-wise separable convolutions (BIkSConv) to replace standard convolutions. BIkSConv approximates a standard convolution as follows. A standard k x k M convolutional kernel can be represented as an ordered set of block vectors of size k x k x t. Since each block vector can be written as a linear combination of k basis vectors of size l x k t, this standard convolutional kernel can be viewed as an ordered set of block vectors each of which is a linear combination of k-2t. basis block vectors. Then BIkSConv eliminates most basis block vectors and their corresponding coefficients to obtain an approximated convolutional kernel. As shown on the left of Figure [] the extreme version of BIkSConv is called the basic BIkSConv where only one basis block vector is used. When carefully setting the depth of the block vector, that is the parameter f, approximated convolution of fewer parameters can be obtained and the corresponding compact CNN has acceptable prediction performance compared to the standard convolutions. To increase the prediction accuracy of the basic BIKSConv, an enhanced version is proposed by increasing the number of basis block vectors, that is the parameter s. as shown on the right of Figure || However, adding too many basis block vectors will significantly increase the model size and computational cost. Thus there is tradcoff between model efficiency/size and accuracy. To realize the full potential of the enhanced BIkSConv in trading-off model efficiency/size and accuracy,
54
55
58
57
58
59
60
61
63
64
65
66
67
68
69 we propose a framework based on the principal component analysis to search for the hyperparameters t and of each BIkSConv layer for the given standard convolutional network. The proposed search
71 framework suggests a possible setting of parameters and 8 such that the constructed model based on
72 these selected hyperparameters may achieve high prediction accuracy while simultaneously satisfying the constraints of model size and model efficiency in terms of MAdds.
73
74 To summarize, our main contributions are follows. First, we develop new convolutional layer called BIkSConv to approximate the standard convolutional layer. To approximate a standard convo- lutional kernel, BIkSConv divides the kernel into blocks and approximates each block by linear combination of several fixed basis block vectors. The constructed networks have small model size and cost fewer multiply-adds operations while maintaining acceptable prediction accuracy. Then, also develop a search framework to determine the block depth and the number of basis block vectors such that the corresponding networks with selected hyperparameters achieve comparable prediction performance while simultaneously satisfying the constraints of model size and model efficiency. We also present experimental results demonstrate the performance of selected BIkSConv--base CNNs based on our proposed hyperparameter search algorithm. Our results show that selected HikSConv-based CNNs achieve competitive performance compared with the standard convolutional models for the datasets including ImageNet, CIFAR-10/100, Stanford Dogs, and Oxford Flowers.
75
76
77
79
80
81
82
83
84
85
Related Work
86
87
Many efforts have been devoted to improve the efficiency of CNNs which could be roughly divided into three categories. First, model pruning is popular method to improve efficiency of CNNs. In
88
89
( 37,, their methods remove redundancy in the trained CNN model by pruning connection. In
90
20 35. the calculation amount of the trained model is compressed via quantization. In 28), model filters that have small contributions are removed and the corresponding trained model is fine-tuned to preserve the performance.
6 21
91

---

Basic block-wise
Enhanced block-wise separable
Separable convolution layer
convalution layer
Po (3).
Conv. with
a
12
Conv. with
H
M
W
81
Figure 1: The proposed block-wise separable convolution and its enhanced version.
93 Second, many techniques are developed factorize the standard convolutions. In 30 convolutions with larger spatial filters are factorized into two-layer convolutional architectures with smaller spatial filters. Through different combinations depthwise convolution, pointwise convolution, and groupwise convolution, many well-known factorizing frameworks were developed. In (25), the depth- wise separable convolutions (DSCs) were proposed where the standard convolution is decomposed into depth-wist weit convolution and pointwise convolution. The ShuffleNets (36) 18) uses pointwise group convolution with channel shuffle to decompose the standard convolution. Morcover, many lightweight models based DSCs groupwise convolutions such as MobileNets (8) 23 7 and
95
97
99
101
ShuffleNets (36 18) were proposed to greatly reduce computation cost while maintaining accuracy.
102
31 have been proposed to automatically construct network architectures. These methods search over a set of network hyperpa- rameters including different types of convolutional layers and kernel sizes, to find network structure which satisfies optimization constraints such as inference speed. Major search frameworks include genetic-base based methods (34) and reinforcement learning based methods (38). These techniques were
Recently, neural architecture search-based methods (34
32
38
39
103
105
107
used state-of tfrth-rrr CNN architectures such as MnasNet (31) and MobileNetV3 (7).
108
Convolution weights of trained CNNs are also analyzed in ( 26 4). Following their analysis, sev- eral approaches toward reducing redundant weights were proposed. In 12 13, the convolutional
109
110
kernels are approximated via low-rank factorization. In 4 the kernels are analyzed via principal component analysis.
111
3 Blockwwise Separable Convolutions (BIkSConv)
112
2.... n). In standard CNN, each convolutional 114 layer converts an input tensor of size M x x Y into an output tensor O of size N x X x Y
113
For any natural number n, let [n]] denote the {1,
115 by applying the filter kernels Fi, F2...., FN, each of size M x x E with odd such that, for any
116 J, V.j E [X] x x [Y] x [N],
(1-1)/2
(1-1)/2
M
O v.J)
Σ
Σ
Σ
I( + 81, 82, 83) (5 82, 82,
(1)
(2-1))2
(2-1)/22-11
117
During training, the weights of each kernel F, are optimized via backpropagation. The total number of weight parameters to optimized in each kernel F, is 22. M. the subsequent work, we propose framework to reduce the number of parameters of the standard convolutions while preserving its prediction performance. Then, in order to implement our new framework, we adopt combination of pointwise and group-wise convolutions to efficiently realize the reduced convolutions. Combining these ideas, we introduce block-wise separable convolutions, denoted by BlkSConv. However, to generate a BIkSConv-based models, many hyperparameters should be determined for keeping prediction performance, model size, and model efficiency. Thus, we also propose an efficient hyperparameter search algorithm to select hyperparameters satisfying the given model constraints.
118
119
120
122
124
3

---

128 3.1
Expressing standard convolution via linear combination of block vectors
In this section, propose block-wise separable convolutions. First, each convolutional kernel F3 of size M x l x l can be expressed as concatenation of M/t blocks 03") .09²) ..... ,0)
. (M/9 each
128
of size E x x t where 2,ah (z, 2) = Fj(z, U, + (k 1)t) for any 2, V [X] x [X] x [t]. We call & the block depth. Let {B1, B2...... Bre²} be set of basis block vectors. Each Q,a)) can linear combination of B1. .g...., Biea, that is, there exist tl? values plk) (i) E R such that 0,0² = - (i)- B; In practice, tt2 may be large. In order to reduce the model size, we require the number of basis block vectors is fewer than equal a fixed number
129
130
be expressed uniquely
132
133
8 with 8 < Ue2. Now each Q,44) is replaced by the following linear combination of BI...., B,, that
134
is 2 = Ei=1 ((()( (1) B;- The corresponding convolutional kernel F, is the concatenation of
135
M/t blocks p01)) ,...,, 2 Therefore, the corresponding output tensor is
136
11-1)/2 Σ
Ō(z,y.5)
Σ
1(x 31 1 22, -1,55+22,55) SF,(,,
$1.52 (t- 1312
137 By Equation 2 the number of weight parameters in BIkSConv i 8- (t- + 2). To significantly reduce model size, we set 8 = 1. Figure Ieft illustrates the operation of BIkSConv when s = 1. In order to achieve the minimal model size, can be set as /M/ / and the number of parameters becomes 2fv Mf while the parameter number of the standard and x pointwise convolutions are M and M, respectively. Thus, the constructed BIkSConv- based CNNs have smaller model size than existing lightweight CNN models. Take the ResNet34 (10) as an example where, in the last stage of the ResNet-34, the convolutional kernel size is 3 3 and the channel size is 512, that is l 3 and M- = 512. In this case, the ratio between the parameter size of the BIkSConv-based convolutions and the standard convolutions is approximately 0.0295.
138
139
140
142
144
145
146
However, the prediction performance of the BIkSConv- CNN with the smallest model size is usually worse than the standard CNNs. To increase accuracy, the number of basis block vectors should be increased, that is s > 1. Figure rright illustrates the operation of BIkSConv when s > 1. In this case, the number of parameters becomes 2sE/M M. Let us take convolutions in the last stage of R4 as examples. Let us set t 4 in the BIkSConv. Now the ratio between the parameter size of the BIkSConv- based convolutions and the standard convolutions is approximately 0.0356. add at least basis block vectors increase prediction accuracy. In this case, the ratio between the parameter size of the BIkSConv-based convolutions with 5 basis block vectors and the standard convolutions is approximately 078. In the experimental section, demonstrate that the SilsSConv-based convolutions with few basis block vectors have prediction performance as well as the standard convolutions on ImageNet or even outperform the standard convolutions on several datasets when the backbone CNNs are ResNets.
148
149
151
Thus
153
154
155
156
157
The next problem is the computational efficiency of BIkSConv. If we compute the kernel F, first and perform regular convolution according to the kernel F,, then is obvious that the computational cost is larger than the cost for just performing a standard convolution. We will address this problem in the subsequent section.
159
160
161
162
3.2
Implementation of BIkSCony via a combination of pointwise and group-wise convolutions
163
In this section, we propose an efficient implementation method to realize BIkSConv. The flowchart of the proposed implementation illustrated in Figure To derive an efficient implementation for BIkSConv operation, we rewrite Equation 2 as follows.
164

---

Concetenated tenser bessom
Output Tensor (ss/
I
Reshaped tensor J
IY
)
f
M/t
Rearonge
Step a
Polrowbi summation
Stop
groups 538
1
M/t
Figure 2: Flowchart of the block-wise separable convolution where Gconv, means the group-wise convolution.
M/t ΣEΣ
11, 52,2 +k 1)t) 2,th (0))),,,,s0 (81,82,2) (3)
k=1
M/t
ΣΣΣ
+1, 82, + (k - 1)()
(i) B,(81, (4)
k=1
i=1
M/t
ΣΣΣ
B1
Pl (1) T(a + 81,1 82, + (k: 1)t). (5)
f=1
((, (x, v, i): point-wise convolution of T=
Let (z, k) be tensor of size X x Y x M/tt defined by [1 y. k) [i +k- (k- 1)t). We M/t (k)) (i) I((x (r + $1, y + s2, k) which point-wise convolution of T..
186
define (z, y, i) & Ae/
187
Next, we define [z, 2) (z. v. i) and let J be the reshaped tensor which is the concatenation
168
189
of J JJ...., J.. that is [xx y, + (i 1)t) Jz y, 2). Now Equation 5 can be rewritten as
Ō(z,v.))
ΣΣΣ Σ s2, 2) (s + $1 $2 + 1)t).
(6)
61.82211
170
Finally, Equation G just a group-wii wise convolution of the tensor J with s groups.
171
Let us compute the computational cost (MAdds) of the implementation for BIkSConv. By Equation (Step in Figure 2. the computational cost of point-wise convolutions on the concatenation of
172
I,..., T, is sXY M. addition, by Equation (Step in Figure 2). the computational cost of the group-wiss wise convolution on the tensor J is x Finally. the computational cost of the point-wise
173
2
174
175
summation the last step is s.Y. The total MAdds of a BIkSConv operation is sXY (M tpa + 1) while the MAdds of standard convolution is X ME2 Again, let take convolutions in the last stage ResNet-34 as examples. We set s 5 and = as the hyperparameters the BIkSConv- based convolution. Now the ratio between the MAdds of BIkSConv-basee
177
convolution and standard convolution is approximately 0.595. Thus the proposed BIkSConv operation is much more efficient than the standard convolution in practical cases. We remark that the proposed implementation requires much GPU memory due to using many group-wise and pointwise convolutions.
179
181
182
3.3
Hyperparameter search via principal component analysis
183
Designing a BIkSConv- CNN involves hyperparameters including the block depth and the num- ber basis block vectors each convolutional layer that affect the performance of the corresponding CNN model. To realize an efficient BIkSConv-basee CNN, we conduct hyperparameter search algo- rithm based on principal component analysis trained CNNs. Given trained CNN, the algorithm generates the block depth and the number of basis block vectors for each standard convolutional layer
184
186
5

---

188
of the trained CNN in the following way. First, for each individual E x E x M kernel K of the trained CNN where we assume that M 2" for some a N. the kernel K is partitioned into M/t block vectors B1, B2..... ........ Bus/t each of size l l X with E {1, ...... ,2,] } for some integer Next, perform principal component analysis (PCA) on the set {B1, B2, ........ Ba Then, for
189
190
a.
192
fixed integer y and, for each 4 E (1, 7), the algorithm computes the variance Vz., of the kernel K which is explained by the first q principal components PCI,PCC...,PPC
193
In addition, let Cg and MS denote the MAdds and the model size of the BIkSConv under the setting that the block depth is and the number of basis block vectors is q. respectively. Note that the MAdds and the model size of the standard convolution is exactly CCM,1 and MSM11 respectively. After computing
195
196
197
all Ve.g. CC and MS the algorithm generates the feasible set
Ha = {(1,4): Vz. 2 Ohes CCr.g S ore CCM,1, and MS S Sa MMS
(7)
198
for fixed positive constants Czz, Ole; Og E (0, 1). Finally, the algorithm chooses the hyperparameter (f, q) from Ha..o..a. according to the computational cost or the model size.
199
On one hand, note that the goal of BIkSConv is to maintain the prediction performance of the trained standard CNN. In general, the prediction accuracy is proportional to the model size of the constructed
201
CNN. Therefore, in this sense, we choose the hyperparameters (f, q) from Ha..ge such that the
202
203
constructed BIkSConv has the largest parameter size, that is
(1, arg
max
MS..0*
(
(f.4) Hagaaaa
One can expect that the generated BIkSConv-based CNN has nice prediction performance compared to the original CNN with standard convolutions.
205
206
On the other hand, one of the advantage of BIkSConv operations is that BIkSConv can greatly reduce the model size of the original standard CNN. Thus, in this sense, we can select the hyperparameters
207
208
(f, q) from H Ha such that the constructed BIkSConv has the smallest parameter size, that is
(i,4) arg
min (l.))Eoor,ee.0z
MS..q.
However, the prediction performance may degrade when the parameter size of the BikSConv-based model decreases. We will demonstrate the experimental section that the BIkSConv-based CNNs generated according to Equation X also have acceptable prediction accuracy compared to the standard CNNs.
210
212
213
In summary, both Equation 8 and Equation 9 provide ways to determine hyperparameters from the feasible set Ha..a..., such that corresponding BlksConv-based CNNs have smaller model size and fewer multiply-adds operations than the original CNN with standard convolutions.
214
215
216
Finally, let us consider the extreme case that two constants and y are set by 3 = and y 1. Let us further set the search parameter a = O. Under this restricted search condition, the cardinality of the feasible set Ho, is always 1. Thus the outputs of Equation 8 and Equation aree the same. In fact, the resulting BikSConv- CNN is exactly the same as the CNN where the standard convolutions are replaced by the blueprint separable convolutions previously developed in (4).
217
218
219
220
Experiments
221
We evaluate BIkSConv and the proposed hyperparameter architecture search algorithm combining with R0, R8sNet-18, and ResNet-26 (5) on ImageNet (22), Stanford Dogs, (14), and Oxford 223
102 Flowers (19). The proposed methods are also evaluated combining with ResNet-20 and ResNet-56
225
on CIFAR 10/100 (15).
226
4.1
Hyperparameter Search Details
227
We apply the PCA-based hyperparameter search algorithm (HSA) developed in Section 3.3 on several
228
variants of ResNet models. In the first part, we consider the large-scale classification scenarios.
Several standard ResNets are trained on ImageNet first and their architectures are shown in Table [
230
The HSA for searching BIkSConv architectures is only applied to conv3_x, __, conv4_x __x, conv5_ __
layers of these standard ResNets. Next, the search hyperparameters Qe Ccey ag are set as 0.5 or

---

Table 1: ResNet architectures used in the first part of the experiment on ImageNet, Stanford Dogs, and Oxford 102 Flowers. The PCA-based HSA is applied conv3_ x, conv4_ x, conv5 layers and the corresponding convolutional kemne is replaced by the BIkSConv module found by HSA.
ResNet- 10 (L=1), ResNet- 18 (L=2), ResNet-26 (L=3)
Layers Names
Output Size
ResNet
Applying HSA e.g.(ResNet-10)
convl
112 112 x 64
7 7, 64, stride 3 x 3, stride 2
No
max pool
56 x 56 x 64
3 x 3,
64
conv2_x
56 x 56 64
X L
No
3 x 3,
64
x 3,
128
conv3_x
28 28 x 128
Yes
conv-s5t2
3 x 3,
128
3 x 3,
256
conv4_x
14 X 14 x 256
x L
Yes
conv-s512
3,
256
3
512
conv5_x x
7 7 x 512
L
Yes
conv-sltl
3 x 3, 512
average pool fully connected
x 512 1000
7
512 x 1000 fc
Table 2: Performance results for BIKSConv-based ResNet-18 and ResNet-26 on ImageNet.
ResNet-18 18 on ImageNet
ResNet-26 on ImageNet
(av, Oce Qg,, SS)
Accuracy
P_ratio
Accuracy
P_ratio
MA_ratio
(0.50, 0.50, 0.50, max)
69.922
0.4065
0.4614
72.038
0.4241
0.4618
(0.50, 0.75, 0.75, max) (0.50, 0.50, 0.50, min)
69.782
0.6014
0.6597
72.326
0.6308
0.6722
67.572
0.1264
0.2700
69.970
0.1250
0.2668
(0.50, 0.75, 0.75, min)
67.540
0.1246
0.2986
69.922
0.1243
0.2910
Standard (replaced layers)
70.728
10.8M
1213.8M
72.604
17.03M
1907.4M
232
0.75. It is possible that the feasible set Ha..a..a, is empty. this case, the corresponding standard convolutional layer is not replaced and is denoted by conv as shown in Table [ Moreover, the
234
proposed HSA has two selection strategies: one is based on the largest parameter size, denoted by
SS max, and the other is based on the smallest parameter size, denoted by 5S min as shown in Table 2 The selected BIKSConv, sitj, which means i basis block vectors and j depth of the blocks.
236
237
The feasible set Ho.. is likely to be empty when the parameter o is large. In the case that o is large, it often requires many principal components to accumulate enough explained variance and thus this causes large numbers of parameters or MAdds. Therefore, the feasible set Ha..o..a, is probably empty when we further require small ae and a... On the other hand, the parameter a₂ cannot be too small because the prediction performance of the network is highly proportional to the amount of the accumulated variance as discussed in Section 3.3 where we will demonstrate it the ablation study of this section. For the above reason, we only present the results for ResNet- 18 and ResNet-26 on ImageNet under the setting that a, 0.5 which are shown in Table 2
238
239
240
242
243
244
245
In the second part, we consider the small-scale classification on CIFAR10/100, We use the standard 2esNet-20 and ResNer-56 the experimental models where the architectures are slightly modified to suit the small-scalc images. The proposed HSA is only applied to conv4 x layers of these two standard ResNets. More BIkSConv-based architecture search results can be found in the appendix.
246
247
248
249
4.2
Performance on large-scale classification: ImageNet
To evaluate the performance of BikSConv-based models in large-scale recognition, we conduct experiments on (22) Each model takes 3 days to be trained on a single GPU (Nvidia Tesla V100). ImageNet contains nearly 1.3M training images and 50,000 testing images. For
251
7

---

Table 3: Comparison among the BIkSConv-based and Standard ResNet on ImageNet and CIFAR.
Dataset
Models
Accuracy
Parameters
MAdds
ResNet-10 standard
63.386
4.64M
520M
ImageNet
BIKSCon--ReSNet-Ie (0.5, 0.5, 0.5, max)
69.922
4.39M
560M
BIkSCony-ResNet-26 (0.5, 0.5, 0.5, min)
69.970
2.13M
509M
Re standard
67.994
202752
12.97M
CIFAR 100
BIKSConn-ResNet220) (0.5. 0.5, 0.5, max)
67.078
72704
5.04M
BIKSConv-Res (0.5, 0.5, 0.5, min)
69.994
149440
10.61M
Table 4: Performance results for BIkSConv-based ResNet-56 on CIFAR10/100.
ResNet-50 56 on CIFAR 10
ResNet-56 on CIFAR 100
(ap; Ocs ag., SS)
Accuracy
P P_ratio
MA_ratio
Accuracy
P_ratio
MA_ratic
(0.5, 0.5, 0.5, max)
93.372
0.3734
0.3829
70.636
0.3734
0.3829
(0.5, 0.75, 0.75, max)
93.338
0.6196
0.6292
70.668
0.6196
0.6292
(0.5, 0.5, 0.5, min)
93.324
0.2335
0.2462
69.994
0,2316
0,2570
(0.5, 0.75, 0.75, min)
93.324
0.2335
0.2462
69.994
0.2316
0.2570
Standard (replaced layers)
93.218
645120
41,28M
70.998
645120
41,28M
253
the experimental setup, ResNet-10, 10, ResNet-18, and ResNet-26 are trained on ImageNet under the following setting. The number of epochs is 100 and the batch size is 256. SGD is used as the
255
optimizer and the initial learning rate, the momentum, and the weight decay are set to 0. 0, 0.9, and
10 respectively. The learning rate is scheduled to decay by factor of 0.1 at epochs 30, 60, and 90.
257
We augment the data via random resized crop to 224px and random horizontal flip. The performance results are shown in Table More experimental results can be found in the appendix.
258
259
On one hand, let us focus the cases that a, = 0.5 and SS = max in Table 2 The prediction accuracies of the selected BIKSConv-based models and the standard model close within 1%. confirms our expectation that BIkSConv-baid based models have smaller parameter sizes and fewer MAdds than standard models while preserving prediction performance if the proposed HSA adopts selection strategy based on the maximum parameter size.
261
263
264
On the other hand, let consider the case that 0s, = 0.5 and SS = min in Table 2 The parameters and MAdds of the BikSConv- bbsed models are only 12.6% and 29.8% of the standard model while the gap of their prediction accuracies is about 3%. We adopt an interesting way based on restricting the parameter size and MAdds to interpret the advantage the generated BIkSConv-based
265
267
models where the selection strategy S5 is set as min. We also compare the standard ResNet-10, the ResNet-26 in Table 3 where the parameter
269
BIkSConv-biid based ResNet-18 18, and the BIKSConv-bast sizes or MAdds of three given models similar. The HikSConv-baseee ResNet-26 with parameter (0.5, 0.5, 0.5, min) and the BIkSConv-basse ResNet-15 18 with parameter (0.5, 0.5, 0.5, max) greatly outperform the standard ResNet- 10 where both the BIksConv-based models lead to accuracy gain of at least 6.5%. In addition, the BIkSConv-based ResNet-226 with parameter (0.5, 0.5, 0.5, min) only has half the parameter size of the BlkSConv-basel. ResNet-18 with parameter (0.5, 0.5, 0.5, max).
271
273
275
4.3
Performance on small-scale classification: CIFAR 10/100
278
The performance results are shown in Table The BIkSConv based ResNet-56 models have much smaller model sizes and fewer MAdds than the standard model while all BIkSConv-based variants outperform the standard model on CIFAR 10 and have comparable accuracies on CIFAR 100. In the ResNet-20 with (0.5, 0.5, 0.5, max) and the standard ResNet- 20 both have a comparable accuracy while the BIkSConv-bard based ResNet-20 model is compressed
277
278
279
bottom of 3 the BIkSConv-
281
64% the parameter size and MAdds is decreased 61 % compared to the standard ResNet-20 model.
Furthermore, the HiksConv-basee ResNet-56 with (0.5, 0.5, 0.5, min) and the standard ResNet-20
283
model both have similar parameter sizes and MAdds while the BikSConv-based ResNet-56 model
has an accuracy gain of 2%. More experimental results can be found in the appendix.

---

Table 5: Performance comparison among BIkSConv-bused and the standard ResNet-18 models.
Stanford Dogs
Oxford 102 Flowers
(av, Oc, Oss: SS)
Accuracy
P_ ratio
MA_
Accuracy
P ratio
MA_ratio
(0.5, 0.5, 0.5, max)
53.005
0.5327
0.5394
65.546
0.5327
0.5394
(0.5, 0.75, 0.75, max)
53.359
0.7277
0.7377
64.567
0.7277
0.7377
(0.5, 0.5, 0.5, min)
53.159
0.3273
0,4006
65.289
0.4835
0.5179
(0.5, 0.75, 0.75, min)
53.615
0.3171
0.4611
63.217
0.4743
0,5872
Standard (replaced layers)
52.436
10.8M
1213.8M
62.238
10.8M
1213.8M
Table Results on Stanford Dogs for different o, with SS min.
ResNet- 18 on Stanford Dogs (Ap, Og 0.75, a,
0.75, SS min)
Cxy
Accuracy
P P_ratio
Qg
Accuracy
P P_ratio
MA_ratio
0.0
50.918
0.0397
0.1781
0.3
52.839
0.1074
0.2377
0.1
50,499
0.0449
0.1777
0.4
52.035
0.2751
0,4684
0.2
51.040
0.0767
0.2458
0.5
53.615
0.3171
0.4611
Standard
52.436
10.8M
1213.8M
4.4
Performance on fine-grained classification
We conduct experiments for fine-grained recognition on two datasets Stanford Dogs and Oxford 102 Flowers. For the experimental setup, the standard ResNet-18 and BIkSConv-variant are all trained from scratch by augmenting data through random crops, horizontal flips, and random gamma transform. We use SGD as the optimizer and the initial learning rate, the moment, and the weight
287
289
decay are set to 0.1, 0.9, and 10 respectively. The number of epochs is 200, and the learning rate scheduled to decay at epochs 100, 150, and 200 by a factor of 0.1. The proposed BIkSConv- ResNet- 18 models significantly outperform the standard ResNet- 18 model both Stanford Dogs and Oxford 102 Flowers as shown shown Table
291
293
294
4.5
Ablation Study: Necessity to have large explained variance
295
Here, we demonstrate how the variance hyperparameter og affects the prediction accuracy of BlksConv-based CNNs. We use ResNet-18 18 as the experimental model. After training the stan- dard ResNet-18 on Stanford Dogs, the next goal is to find several BIkSConv-variants of ResNet- 18 all of which have different explained variances such that their accuracies
297
be compared. Note that Ha.0.75.0.75 C H5.0075.0.75 for any a, b with a 2 b. Based on this observation, the model in Ha.0.75.0.75 which has the smallest parameter size is likely to have small explained variance as well. Therefore, the selection strategy of the proposed HSA is set by SS = min ResNet-18 models with different explained variances. Now we apply the proposed HSA to the trained ResNet- 18 under six search hyperparameters (a,, 0.75, 0.75, min) : a2 0.0, 0.1, 0.2, ....... .., 0.5). The comparison result shown in Table It
289
300
301
302
order to select several BIkSConv-bris
303
304
305
can be seen that the accuracy of the BIkSConv-based model is greater than that of the standard model only when the variance hyperparameter a, large enough, that ar 2 0.5.
306
Conclusion
307
308
In this paper, we introduce the block- wise separable convolutions (BIkSConv) to replace standard convolutions. An efficient implementation of the BIkSConv operation via combination of pointwise
309
310
and group-wise convolutions is also given. Moreover, we also propose an efficient hyperparameter search algorithm based on principal component analysis order to select an optimal BikSConv-basee convolutional network under certain constraints on model size and model efficiency. Finally, the experimental results demonstrate the advantage of the BlksConv-based CNN models selected by the proposed hyperparameter search algorithm.
311
313

---

References
315
316 [1] M. Denil, B. Shakibi, L. Dinh, M. A. Ranzato, and N. De Freitas, "Predicting parameters in deep learning," in Advances in neural information processing systems (NIPS), pages 2148-2156 2013.
318
[2] E. Denton, W. Zaremba, J. Bruna, Y. Le-Cun, and R. Fergus, "Exploiting linear structure within convolutional networks for efficient evaluation," in Advances in neural information processing systems (NIPS), pages 1269-1277, 2014.
320
[3] J. Guo, Y. Li, W. Lin, Y. Chen, and J. Li, "Network decoupling: From regular to depthwise separable convolutions," arXiv preprint
322
2018
323
[4] D. Haase and M. Amthor, "Rethinking Depthwise Separable Convolutions: How Hernel Correlations Lead to Improved MobileNets," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14600-14609, 2020.
324
326
[5] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. CoRR, abs/ abs/151,.03385, 2015.
327
328
[6] S. Han, H. Mao, and W. J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," " in Proceedings of the International Conference on Learning Representations, 2015.
329
330
331
[7] A. G. Howard, M. Sandler, G. Chu, L.-C. Chen, M. Tan, W. Wang, et al., "Searching for mobilenetv3", in Proceedings of the International Conference on Computer Vision (ICCV), 2019.
332
333
[8] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, et al., "Mobilenets: Efficient convolutional neural networks for mobile vision applications", 2017.
335
99] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. "Soft filter pruning for accelerating deep Convolu- tional Neural Networks," 2018.
337
[10] K. He, X. Zhang, S. Ren and Sun, "Deep Residual Learning for Image Recognition," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
339
[1I] Y. He, X. Zhang, and J. Sun, "Channel pruning for accelerating very deep Neural Networks." in Proceedings of the IEEE Internation Conference on Computer Vision (ICCV), pages 1389-1397, 2017.
341
343
[12] M. Jaderberg, A. Vedaldi, and A. Zisserman, "Speeding up convolutional neural networks with low rank expansions," in Proceedings of the British Machine Vision Conference. (BMVA), 2014.
345
[13] Jin, A. Dundar, and E. Culurciello, "Flattened convolutional neural networks for feedforward acceleration," arXiv preprint LEXiv:1412.5474, 2014
348
[14] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-F. Li. "Novel dataset for fine-grained image categorization," in first Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
348
349
350
[15] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
352
[16] H. Li. A. Kadav, Durdanovic, H. Samet, and H.P. Graf, "Pruning filters for efficient ConvNets", 2016.
354
[17] J.H. Luo, J. Wu, and W. Lin, "Thinet: A filter level pruning method for deep Neural Network compression," in Proceedings of the IEEE Internation Conference Computer Vision (ICCV), pages 5058-5066, 2017.
356
[18] N. Ma, X. Zhang, H.-T. Zheng and J. Sun, "Shufflenet v2: Practical guidelines for efficient cnn architecture design", in Proceedings of the European Conference on Computer Vision (ECCV), pages 116-131, 2018
358
10

---

380
[19] M.-E. Nilsback and A. Zisserman. "Automated flower classification over large number of classes", In 2008 Sixth Indian Conference on Computer Vision, Graphies and Image Processing.
361
pages 722-729, IEEE, 2008
363
[20] E. Park, S. Yoo, and P. Vajda, "Value-awar quantization for training and inference of Neural Networks", in Proceedings of the European Conference on Computer Vision (ECCV), pages 580-595, 2018.
364
385
386
[21] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, "Xnor-Net: ImageNet classification using binary Convolutional Neural Networks", in Proceedings of the European Conference on Computer Vision (ECCV), pages
387
368
2016.
369
[22] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, 7. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):21 2015.
370
371
372
[23] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510 2018.
373
374
375
[24] A. See, M.-T. Luong, and C. D. Manning. "Compression of neural machine translation models via pruning," in Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, pages 291-301, 2016.
376
377
378
[25] L. Sifre and S. Mallat. Rigid-motion scattering for image classification. PhD thesis, 2014.
379
[26] W. Shang, K. Sohn, D. Almeida, and H. Lee, "Understanding and improving convolutional neu- ral networks via concatenated rectified linear units," " in Proceedings of international conference on machine learning (ICML), pages 2217-2225, 2016.
381
382
[27] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proceedings of the International Conference on Learning Representations, 2015.
384
[28] P. Singh, V.K. Verma, P. Rai, and V. Namboodiri, "Leveraging filter correlations for deep model compression," 2018.
385
386
[29] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions" in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
387
388
389
[30] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens and Z. Wojna, "Rethinking the Inception Archi- tecture for Computer Vision," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818 2-22,, 2016
390
391
392
[31] M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le, "Mnasnet: Platform-aware neural architecture search for mobile," arXiv preprint arx:v:1801 11666, 2018.
383
394
[32] B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, "Fhnet: Hardware-aware efficient convnet design via differentiable neural architecture search," arXiv preprint 181.0331, 2018
385
396
397 [33] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, "Quantized convolutional neural networks for mobile devices." in Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4820-4828, 2016
399
400
[34] L. Xie and A. Yuille, "Genetic enn," in Proceedings of the IEEE Internation Conference on Computer Vision (ICCV), pages 1388, 2017.
[35] D. Zhang, J. Yang. D. Yc, and G. Hua, "LQ-Nets: Learned quantization for highly accurate and compact deep Neural Networks," in Proceedings of the European Conference on Computer Vision (ECCV), pages 365-382, 2018
403
405
[36] X. Zhang, X. Zhou, M. Lin and J. Sun, "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6848-6856 2018.
406
11

---

408
[37] L. Zhu, R. Deng, M. Maire, Z. Deng, G. Mori, and P. Tan, "Sparsely aggregated Convolutional Networks," " in Proceedings of the European Conference Computer Vision (ECCV), pages 386-201,211
409
[38] B. Zoph and Q. Le, "Neural architecture search with reinforcement learning," " in Proceedings of the International Conference on Learning Representations, 2017.
412
413
[39] B. Zoph, Vasudevan, J. Shlens, and Q. V. Le, "Learning transferable architectures for scalable image recognition," in Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 8697-8710, 2018
415
12

---

Checklist
418
417
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
419
contributions and scope? [Yes] Section T
420
(b) Did you descrihe the limitations of your work? [Yes] Section 3.2
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
423
424
2. If you are including theoretical results...
425
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
426
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Supplemental material
429
(b) Did you specify all the training details (e.g.. data splits. hyperparameters, how they
were chosen)? [Yes] Section 4.1 Section 4.2, Section 4.4 Appendix
432
433
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
434
435
(d) Did you include the total amount of compute and the type of resources used (c.g., type
436
of GPUs, internal cluster, cloud provider)? [Yes] Section 4.2 Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
438
(a) If your work uses existing assets, did you cite the creators? [Yes] Section 4
439
(b) Did you mention the license of the assets? [Yes] Appendix
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l ro using/curating? [Yes] Appendix
442
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Appendix
444
445
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
447
448
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
449
450
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "oYleYuPF-S_", "reviewer": "Reviewer_B29N", "summary": "This paper introduces a novel convolution factorization for efficient image recognition. The formalization proceeds as follows: First, the authors rewrite the standard convolution (on a single output channel) as the sum of contributions of the application of different kernels to different groups of input channels (t groups). Then, every kernel for every group is formalized as a low-rank approximation achieved by means of a linear combination of s basis kernels. As this approach effectively reduces the number of trainable parameters but does not affect (or even increases) the number of operations, the authors showcase an efficient implementation. Specifically, exploiting the linearity of all parts involved (both the convolution and the decomposition of the kernel into basis vectors), they show the factorized convolution can be espressed in terms of standard operations such as point-wise convolution, reshaping, and group convolutions. Finally, the authors propose a PCA-based technique to automatically search for suitable values of t and s, that enable a good tradeoff between accuracy and computation. Experiments are carried out with ResNets variants on 5 datasets.", "questions": "N/A", "limitations": "- The authors briefly mention memory overhead as a potential limitation. I encourage them to discuss about on time latency measurements as well, in the case the proposed model is not beneficial in that respect.\n- No potential negative societal impact that I can think of.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PROs\n\n- The proposed low-rank formulation of convolutional kernels seems novel to me. Moreover, the authors show how it is easily implementable as a combination of standard neural network operations.\n- The paper is clear for most parts, and the authors succeed in explaining all the operations involved in a clear way. To the best of my judgment, the technical contribution is sound and correct.\n- The experimental section showcases good performances in all the tested settings, with the proposed convolution factorization being able to reach performance within 1-3 points of accuracy of the original model (or even surpassing its performance) at a fraction of the computational cost. Remarkably, Table 5 illustrates how optimizing a ResNet with the proposed factorization is a better strategy that using a smaller version of the architecture (e.g. BlkSConv-ResNet-18 or BlkSConv-ResNet-26 are better than ResNet-10 on ImageNet).\n- The authors release their code as part of the supplementary material, which is considerable.\n\n\nCONs\n\n- The main issue with the paper is that in the experimental section it does not provide any baseline for the proposed factorization. It is true that the block-wise separable convolution provides a much better accuracy/efficiency tradeoff with respect to standard convolutions. However, several other methods have been popularized in the past to save computations. For instance, how would group convolutions, depthwise separable convolutions, low-rank factorization methods [4,5] or channel pruning [6] perform in the same settings the authors use? Without reporting at least a subset of them as baselines, it is very hard to quantify whether the proposed factorization, despite being beneficial, would be something interesting and impactful for the community\n- The paper misses the discussion of model compression prior works that rely on low-rank factorization of weight matrices. For instance, [4,5] are seminal approaches that use SVD decomposition to factorize convolutional kernels as a product of lower rank matrices. Since here a low-rank factorization is also achieved, I think a discussion of those models would improve the paper's quality.\n- Formalizing a convolutional kernel as a linear combination of a set of basis has recently raised a lot of interest, especially in the context of conditional computation. For instance, models such as [1,2,3] predict the combination coefficients based on the input feature itself. Although this is somewhat different with what is proposed here, I think the paper misses a discussion of this branch of related works, for completeness.\n- The proposed implementation described in Sec 3.2 incurs into a higher memory overhead\tand involves some heavy memory-bound operations such as reshaping and copying. I therefore expect that, despite the reduction in MAdds, the actual on device latency of the proposed convolution is similar to (or even higher than) the original one. I think this should be verified and discussed in the manuscript.\n- Fig 1 is helpful but could be made more clear:\n  - Fig. 1 left should specify, for clarity, that it represents the special case for s=1.\n  - The color coding (green) suggests that the kernel is shared among the N output channels. To my understanding, that's not the case.\n  \nTypos:\n- line 138/148: \"top of Figure 1\" / \"bottom of Figure 1\" -> \"Figure 1 left\" / \"Figure 1 right\"\n\nIn summary, I think this is a good paper with the potential of being impactful, yet it needs to report some baseline performances to put the results into perspective.\n\n[1] Chen, Yinpeng, et al. \"Dynamic convolution: Attention over convolution kernels.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[2] Li, Yunsheng, et al. \"Revisiting dynamic convolution via matrix decomposition.\" ICLR. 2021.\n\n[3] Zhang, Mingda, et al. \"Basisnet: Two-stage model synthesis for efficient inference.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Zhang, Xiangyu, et al. \"Accelerating very deep convolutional networks for classification and detection.\" IEEE transactions on pattern analysis and machine intelligence 38.10 (2015): 1943-1955.\n\n[5] Jaderberg, Max, Andrea Vedaldi, and Andrew Zisserman. \"Speeding up convolutional neural networks with low rank expansions.\" arXiv preprint arXiv:1405.3866 (2014).\n\n[6] He, Yihui, Xiangyu Zhang, and Jian Sun. \"Channel pruning for accelerating very deep neural networks.\" Proceedings of the IEEE international conference on computer vision. 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mxDNA14Tp5I", "reviewer": "Reviewer_tB5v", "summary": "This works focuses on CNN-based model compression. It first views the kernel in the standard convolution as an ordered set of vectors, where each vector has smaller dimension than the inputs. Each vector can then be treated as a linear combination of another group of basis vectors. By limiting the number of the basis vectors, it approximates the standard convolutional kernel. The proposed operator is called block-wise separable convolutions (BlkSConv) and can be implemented as a combination of point-wise and group-wise convolutions. The paper also proposes a PCA-based method to determine the dimension and the number of basis vectors. By replacing some standard convolutions with the proposed BlkSConv in CNNs, the authors demonstrate the effectiveness of BlkSConv in CNN-based model compression.", "questions": "Does the fact that BlkSConv can be implemented as a combination of point-wise and group-wise convolutions indicate some insights on how BlkSConv works from another point of view?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper provides a unique view of convolutional kernels and derives the proposed BlkSConv. Meanwhile, the authors find an easy way to implement BlkSConv. It also proposes a systematic way to determine hyperparameters in BlkSConv. All of these make the proposed method easy and practical to use.\n\nWeaknesses:\n1. The notations in figures do not match those in texts. It makes the paper less clear.\n2. The experiments only compare with regular ResNet. Comparisons with other model compression works are necessary.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JXE6SsxVmfo", "reviewer": "Reviewer_J7xD", "summary": "This paper proposes block-wise factorization for approximating convolutions. Given a k x k x M convolutional kernel, it factorizes it into many smaller blocks k x k x t, and then use PCA to select a subset of these blocks to approximate the original more expensive computation.  Overall, kernel factorization is a somewhat common way to reduce ConvNet computations (previous  depthwise/pointwise/group convs are more common), but the new PCA based block selection is kind of new and interesting. Results on CIFAR and ImageNet show the proposed approach can achieve better accuracy-computation trade-offs with various ResNet networks.", "questions": "see above.", "limitations": "No concern on social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "==== Strengths ====\n1. Interesting idea on using PCA to choose the most important top-k sub blocks. To my knowledge, this seems to be quite new and different from previous factorization.\n2. Well written paper with promising results.\n\n==== Weaknesses ====\n1. No latency is reported: Table 3 and 4 show impressive reduction in parameters and MAdds, but it is unclear how fast are these models in real hardwares. Would be nice to include latency measurements.\n2. Limited experiments: most experiments are based on ResNets, would be better to explore other models  (especially those more FLOPs-efficient models such as EfficientNets or MobileNetV3).\n3. Some details are not very clear: for example, it requires both M and t to be the form of 2^k, but what if the original network has irregular channel sizes (such as 40)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Reject