Optimized Separable Convolution: Yet Another Efficient Convolution Operator
Authors: Tao Wei
Yonghong Tian
Yaowei Wang
Yun Liang
Chang Wen Chen
Venue: ICLR 2022 Submitted (2022)
Keywords: Separable Convolution
Volumetric Receptive Field
Optimized

Abstract: The convolution operation is the most critical component in recent surge of deep learning research. Conventional 2D convolution needs O(C^{2}K^{2}) parameters to represent, where C is the channel size and K is the kernel size. The amount of parameters has become really costly considering that these parameters increased tremendously recently to meet the needs of demanding applications. Among various implementations of the convolution, separable convolution has been proven to be more efficient in reducing the model size. For example, depth separable convolution reduces the complexity to O(C\cdot(C+K^{2})) while spatial separable convolution reduces the complexity to O(C^{2}K). However, these are considered ad hoc designs which cannot ensure that they can in general achieve optimal separation. In this research, we propose a novel and principled operator called optimized separable convolution by optimal design for the internal number of groups and kernel sizes for general separable convolutions can achieve the complexity of O(C^{\frac{3}{2}}K). When the restriction in the number of separated convolutions can be lifted, an even lower complexity at O(C\cdot\log(CK^{2})) can be achieved. Experimental results demonstrate that the proposed optimized separable convolution is able to achieve an improved performance in terms of accuracy-#Params trade-offs over both conventional, depth-wise, and depth/spatial separable convolutions. 

Under review as a conference paper at ICLR 2022
OPTIMIZED SEPARABLE CONVOLUTION: YET ANOTHER EFFICIENT CONVOLUTION OPERATOR
Anonymous authors Paper under double-blind review
ABSTRACT
The convolution operation is the most critical component in recent surge of deep learning research. Conventional 2D convolution needs O(C²K²) parameters to represent, where C' is the channel size and K is the kernel size. The amount of parameters has become really costly considering that these parameters increased tremendously recently to meet the needs of demanding applications. Among var- ious implementations of the convolution, separable convolution has been proven be more efficient in reducing the model size. For example, depth separable convolution reduces the complexity to O (C K²)) while spatial separable convolution reduces the complexity to O(²*K). However, these are considered ad hoc designs which cannot ensure that they in general achieve optimal sep- aration. this research, we propose a novel and principled operator called opti- mized separable comvolution by optimal design for the internal number of groups and kernel sizes for general separable convolutions can achieve the complexity OCC K). When the restriction in the number of separated convolutions can be lifted, an even lower complexity OCC' log log(CK²)) can be achieved. Experi- mental results demonstrate that the proposed optimized separable convolution is able achieve an improved performance in terms of eecuraccy#tParaaie trade-offs over both conventional, depth-wise, and depth/spatial separable convolutions.
INTRODUCTION
Tremendous progresses have been made in recent years towards more accurate image analysis tasks, such as image classification, with deep convolutional neural networks (DCNNs) (Kizhevsky al.)
2012 Srivastava et al. 2015 He et al.) 2016) Real et al. 2019 Tan & Le 2019 Dai et al.) 2020) However, the complexity of state-of-the-art DCNN models has also become increasingly high. This can significantly deter their deployment to real-world applications, such as mobile platforms and robotics, where the resources and networks are highly constrained (Howard et al. 2017 Dai et al, 2020).
The most resource-consuming building block of a DCNN is the convolutional layer. There have been many previous works aiming at reducing the amount of parameters in the convolutional layer. Network pruning (Han et al. 2015) strategies are developed to reduce redundant parameters that are sensitive performances. Quantization and binarization (Gong al. [0014] Courbariaux al.) 2016) techniques are introduced to compress the original network by reducing the number of bits re- quired represent each parameter. Low-rank factorization methods (Jaderberg al. 2014 Joannou et al. 2015) are designed to approximate the original weights using matrix decomposition. Knowl- edge distillation (Hinton et al. (0155) applied to train compact network with distilled knowledge from large ensemble model. However, all these existing methods start from pre-trained model. Besides, they mainly focus on network compression and have limited or no improvements in terms of network acceleration.
In this research, we study how to design a separable convolution to achieve an optimal implemen- tation in terms of model size (representational complexity). Enabling convolution to be separable has been proven to be an efficient way to reduce the representational complexity (Sifre & Mallat 2014 Howard et al. 2017 Szegedy et al., 2016). Comparing the network compression related approaches, a well-designed separable convolution shall be more efficient both storage and com- putation and shall not require pre-trained model to begin with.

---

Under review as a conference paper at ICLR 2022
Table 1. A comparison of the number of parameters and computational complexity of the proposed optimized separable convolution and existing approaches. The proposed optimized separable convolution is much more efficient in both #Params and FLOPs. In this table. C' represents the channel size of convolution, K is the kernel size, H and W are the output height and width, 9 the number of groups. "Vol. RF" represents whether the corresponding convolution satisfies the proposed volumetric receptive field condition.
Conventional
Grouped
Deyth-wise
Point-Wise
Depth Sepambbl
Spotial Sepurible
Optimized Separable
Oytimized Separbbl
Comy2D
Conv2D
Com22D
Com
Comr2D
CD
Cum2D (N 2)
Cam-2D (Opcimized N)
#Parants
x²x²
G² k
GK²
C²
C/C + K²)
20 K
20 2 K
GG (Cg(
PLOPS
C² k2
c²122 HW
CK2HW
HW
CHWIC + x2)
2C ICHH K
20 3 KHW
cCHW logiCk3)
RF
Note
C
K
Depth-wise + Point-uise
R² + 2K
2.71828...
Charpee 85
3x3
3x3
1x1
3x3
1x1
Volumeetic
Receptive Ewl (RT)
O x²)
O(C(C K²))
O K)
Conv2D
Depth Separable
Optimal Separable
(a)
(b)
Figure 1. Volumetric receptive field and the proposed optimized separable convolution. (a) The volumetric receptive field (RF) of convolution is the Cartesian product of its (spatial) RF and channel RE (b) Illustrations of the channel connections for conventional, depth separable, and the proposed optimized separable convolu- tions. Optimized separable convolution is sparse- pparse-connecte,, whereas it can be efficiently implemented using a channel shuffic operation.
In the DCNN research, the two most well-known separable convolutions are depth separable (Sifre & Mallat 2014) and spatial separable (Szegedy et al. 2016) convolutions. Both are able to reduce the complexity of convolution. The representational complexity of a conventional 2D convolu- tion is quadratic with two byper-parameters, number of channels (C) and kernel size (K), and its representational complexity is actually O (() Depth separable convolution is constructed as a depth- deit convolution followed by a point-wise convolution, where depth- convolution is group convolution with its number of groups g = C and point-wise convolution is 1 X convo- lution. Spatial separable convolution replaces K x K kernel with K x 1 and 1 K kernel. Different types of convolutions and their complexities are summarized in Table T From this table, see that, for all convolutions, their computational complexities equal to the corresponding representational complexity times a constant. We can also verify that depth separable convolution has a complexity of ((C-(C+K3)) k2)) and spatial separable convolution has a complexity of O(C²K)
Both depth and spatial separable convolutions follow an ad hoc design mode and are non-principled. They are able to reduce the complexity to some degree but normally cannot achieve an optimal sepa- ration. A separable convolution in general has three sets of hyperparameters: the internal number of groups, channel size, and kernel size of each separated convolution. Instead of setting these hyper- parameters in an ad hoc (manual) fashion, we design a novel and principled (auto) scheme to achieve an optimal separation. The resulting separable convolution called optimized separable convolu- tion in this research. The proposed scheme in general performs better than the other convolution operator counterparts and it also enriches the separable convolution family.
To prevent the proposed optimized separable convolution from being degenerated, we assume that the internal channel size is an order of O(C) and propose the following volumetric receptive field condition. As illustrated in Fig. [a] similar to the receptive field (RF) of convolution which is defined as the region in the input space that a particular CNN's feature is looking at (or affected by) (Lindeberg 2013). we define the volumetric RF of convolution to be the volume in the input space that affects CNN's 's output. The volumetric RF condition requires a properly decomposed separable convolution to maintain the same volumetric RF the original convolution before decomposition. Hence, the proposed optimized separable convolution will be equivalent to optimizing the internal number of groups and kernel sizes achieve the target objective (measured in #Params) while
2

---

Under review as a conference paper at ICLR 2022
satisfying the proposed volumetric RF condition. Formally, the objective function is defined by
Equation (2) under the constraints defined by Equations (3-()) The solution to this optimization problem will be elaborated in Section [
We shall show that the proposed optimized separable convolution can be represented with the order of O K). This is at least a factor of vC more efficient than the depth and spatial separable convolutions. The proposed optimized separable convolution is able to be generalized into an N- separable case, where the number of separated convolutions N can be optimized further. In such a generalized case, an even lower complexity at O(C log(C)))) may be achieved.
Extensive experiments have been carried out to demonstrate the effectiveness of the proposed opti- mized separable convolution over other alternatives. including conventional, depth-wise. depth and spatial separable convolutions (Fig. 3(c) and Fig. F))). As further illustrated in Fig. and Fig. on the CIFARIO and CIFAR100 datasets (Krizhevsky al. 2009). the proposed optimized sep- arable convolution achieves a better Pareto-frontied" than both conventional and depth separable convolutions using the ResNet (He et al., (016)) architecture. To demonstrate that the proposed op- timized separable convolution generalizes well to other DCNN architectures, (Liu et al. 2018) architecture by replacing the depth separable convolution with the proposed op- timized separable convolution. The accuracy is improved from 97. 97.24% to 97.67% with reduced representational complexity. On the ImageNet dataset (Deng et al. 2009), the proposed optimized separable convolution also achieves improved performance. For the DARTS architecture, the pro- posed approach achieves 74.2% topl accuracy with only 4.5 million parameters. For MobileNet, the proposed approach achieves 71.1% topl accuracy with only 3.0 million parameters.
adopt the DARTS
2
THE PROPOSED APPROACH
2.1
CONVOLUTION AND ITS COMPLEXITY
A convolutional layer takes an input tensor B of shape (G-11,H-11 Wi-1) and produces an output tensor B₁ of shape (Ci, Hi, Wi), where C., H.. W. are input and output channels, feature heights and widths. The convolutional layer is parameterized with a convolutional kernel of shape (Ci, C1-1, K KW), where Ký are the kernel sizes, and the superscript indicates whether it is aligned with the features in height or width. In this research, we take C. = O(C), H. O(H),
W. O(W), and K = OK) for complexity analysis. Formally, we have
Bi Bi h4, wi) ΣΣΣ Bi 1 14-1, wt-1) k- H kW)
(I)
where h4 = h1-1-111- + kf" and w/ 1 + k.W Hence, the number of parameters for convolution is CCG, K.W and its representational complexity is D(C2k)). The number of FLOPs (multiply- adds) for convolution is C. kW" and its computational complexity is ((CCKKHHW)
For group convolution, we have g convolutions with kemnels of shape (Ci/g. Ct-1g, ,K", Hence, it has O(C*K²/9) parameters and K HW/g) FLOPs, where y is the number of groups. A depth-wise convolution equivalent to group convolution with g = C. = C. A point- wise convolution is 1 x convolution. A depth separable convolution is composed of depth- convolution and a point-wise convolution. A spatial separable convolution replaces a K x K kernel with K x 1 and 1 x K kernels. Different types of convolutions are summarized in Table this table, their number of parameters and FLOPs can be easily verified. It can also be seen that, for convolution, its representational complexity is equivalent to its computational complexity for up to constant WH)).
C.
From
2.2 RETHINKING CONVOLUTIOON AND THE VOLUMETRIC RECEPTIVE FIELD CONDITION
Separable convolution has been proven to be efficient in reducing the representational demand in convolution. However, existing approaches including both depth and spatial separable convolutions follow an ad hoc design and are non-principled. They are able reduce the complexity some extent but will not normally achieve an optimal separation. In this research, we shall design an
In multi objecto optimization, Pareto-optimal. An allocation is Pareto-opting if there is no alternative allocation where improvement can be made to one participant's 's well-being without sacrificing any other's. Here, Pareto-frontier represents the curve of the accuracies able achieve for different #Params (or FL.PPs..
is the set of parameterizations (allocations) that are all
3

---

Under review as a conference paper at ICLR 2022
efficient convolution operator capable of achieving the representational objective by optimal design of its internal ayper-parameters. The resulting operator is called optimized separable convolution.
The proposed optimized separable convolution is called principled as it optimizes the representa- tional complexity under the following volumetric receptive field condition. As illustrated in Fig. Ta the receptive field (RF) of convolution is defined to be the region in the input space that a particular CNN's feature is affected by (Lindeberg] 2013). We define the channel RF to be the channels that affect CNN's output and define the volumetric RF to be the Cartesian product of the RF and channel RF of this convolution. The volumetric RF of convolution actually represents the volume in the input space that affects CNN's output. The volumetric RF condition requires that a properly decom- posed separable convolution at least maintains the same volumetric RF as the original convolution before decomposition. Hence, the proposed optimized separable convolution will be equivalent to optimizing its internal parameters while satisfying the volumetric RF condition. Formally, we shall have the objective function defined by Equation (2) and the volumetric RF constraints defined by Equations (3).) 6
The volumetric RF of a convolution needs to be maintained for technical, conceptual, and experi- mental reasons. Technically, if we do not pose any restriction to separable convolution, optimizing the representational complexity will resulting in a separable convolution being equivalent to a de- generated channel scaling operator The composition of such operators is not meaningful because the composition itself is equivalent to a single channel scaling operator. Conceptually, maintain- ing the volumetric RF encourages the fusion of channel information, which shall contribute to the good performance of a DCNN. In fact, all modern DCNNs are designed following this rule. With- this channel information exchange, the performance of DCNN shall be significantly degraded (depth-wise vs depth separable convolutions in Section 3 Finally, the necessity of maintaining the volumetric RF is experimentally verified. We shall quantize the degree of necessity as overlap coefficient (7) in Section 2.3 and elaborate the experimental results in Section 3
2.3 OPTIMIZED SEPARABLE CONVOLUTION
In this section, for ease of simplicity, we first discuss the case of two-sepaaable convolution. Suppose that the shape of the original convolutional kernel is (Cout; Cin K .W)), where Cin. Cout are the input and output channels, and ( (K", ,K") ) is the kernel size. Let C1 = Cins and C3- = Cout- For the proposed optimized separable convolution, we optimize the representational complexity as objective while maintaining the original convolution's volumetric RF. Formally, the representational demand of the proposed separable convolution is
fgg, (2, 92, C2 KWI = C2C,KWKN C3C2KI
(2)
91
92
In order to satisfy the volumetric RF condition, the following three conditions need to be satisfied:
KH K 1 K#
(Receptive Field Condition)
(3)
kWW.. + K.W kWW - 1 = KW
(4)
C1 C2 2 701
91 01 S C2/7
(Channed Condition)
(5)
91 92
min(Cr (Ct, (41)299
(Group Concolution Condition)
(6)
The channel condition means the product C. C2 needs to occupy each node the input channel Ci = Cin to maintain the volumetric receptive field. This is further explained for the channel condition general case (15) in Section[.44] In order to study the necessity of the proposed volumetric RF condition, an overlap coefficient 7 is introduced to encourage channel information fusion. It can be verified that, if y 2 1. the channel RF shall be maintained, otherwise, it shall be not. By default, we set 7 = in this research unless the behavior of y is particularly concerned.
We have three sets of parameters: the number of groups I1. 92, the internal channel size C2. and the internal kernel sizes WW In this research, we assume that the internal channel size C2 is in an order of (C) and is preset according to given policy. Otherwise, 91 92 C2 = 1 will be trivial solution. This could lead the separable convolution to be over-simplified and not applicable
From T let g= C and K = 1, convolution will have C parameters and CHW FL.PPs. This is in fact channel scaling operator.
4

---

Under review as a conference paper at ICLR 2022
in practice. Typical policies of presetting C2 include C2 min(C1, Cs) (normal architecture), C2 (G1 + C2) (linear architecture), C2 = max Cs) (bottleneck architecture (He et al. 2016), or C2 = 4 min(C1, Ca) (inverted residual architecture (Sandlrr et al. 2018
The solution to the proposed optimized separable prob- lem shall be given in Theorem in Section 2.4 By setting
FLO
N = and 7 1. we shall have
CK KW
91 V C2KEK3
NC, 92 C2/91
(7)
(K K.tt = =(K", or (1,
(8)
(Kl", k22) =(KW ,1) (1,
(9)
and
to 15 ofgross
min f = CCC.CcKH KY K KY O(C K).
(10)
Figure 2. Given channels C₁
C2
Cs 64, and kernel sizes K W
One interesting fact is that if we set 92 = C2/91
in Equation 2 by setting (9) = 0,
f(()1) 0, and (Kr) O, assume that kernel sizes aligned in height and width are equal, one can derive that and Ki K₂ 2
f((K) = 0. The solution 91 = 8, K1 = 3 is saddle point.
9i is the same as Equation
Substituting them into Equation (10). one can get f(g, K() OCC This results in a higher complexity than OCC K). In fact, the solution to (9)) = 0 and f((K) = 0 is a saddle point, which is illustrated in Fig. 2
2.4 OPTIMIZED SEPARABLE CONVOLUTION (GENERAL CASE)
In this section, we shall generalize the proposed optimized separable convolution from N to an optimal N. For ease of analysis, we first introduce the notation channels per group ni C. which simply means: channels per group x number of groups = the number of channels.
gi
Theorem (Optimized Separable Convolution: General Case). Suppose that the shape of the orig- inal convolutional kernel is (Cout, Cin, KH, KW) ). Let C1 Cins and CN Cout. The repre- sentational demand of an N N-separable convolution
N
C41CKK KW
f(gg,), ((g.),(KK111 (.Wl
(11)
1=1 91
or
N
f((n.), (K 1)
Σ
(12)
1=1
Under the proposed volumetric RF condition, we will have: KH + K2 +... K. (N -1)
(Receptive Field Condition)
(13)
KW + KW +...= K. + -1- IN 5
(14)
.N....CN
n1 71...ny NN 2 1 + 91 .I.
(Channel Condition)
(15)
7
C1-1 + I min((i, C1+1). (Group Convolution Condition)
Tu 2 m
(16)
C₁
The solution to this constrained optimization problem (the proposed optimized separable convolution problem) is given by
Assume that C_ OCC and k!llW O(K).
K. K.W
ni
CH1KK" KW
VC
(17)
KH = KH K# 1
(18)
KW kW KWW = (1 (11)
(19)
and its corresponding representational complexity is min f((n.), } (K. }) = ONN cl+bK3)
(00)
5

---

Under review as a conference paper at ICLR 2022
94-
[2 (2.
41-
02 p00.
22
7001
Tep Mot tuadiin
Reahst (saciin) Ind
minuppper
siteshee (e
0.55
0.00
aParams (nd koo)
0.55
Lib0
1.5
0l0
FLOPs (pilkon)
cis 0.2
0.5
Aashe133 Aashvi22
Aasherte
Aatler216
(a)
(b)
(c)
Figure 3. Experimental results on CIFARIO for the ResNet architecture (best viewed in color). The proposed optimized separable convolution (o- (o-ResNet) achieves improved (a) accuracy- -araams and (b) accuracy- -LOPss Pareto-frontier than both the conventional (ResNet) and depth separable (d-ResNet) convolutions. (c) A com-
parison for performances of different convolution schemes.
Table 2. Experimental results on CIFARIO for different overlap coefficients (r). 7 2 1, the volumetric RF maintained, otherwise is not. Each row of o-ResNet channel multiplied. When y 1, the performance hurts due to discourage of channel information fusion.
Overlap Coef. (1)
(Dephhise)
1/16
1/4
4
16
64
+ (Conventional)
Vol. RF
/
/
/
o-ResNei20
90.2
91.45
92.56
93.37
92.84
93.06
92.16
91.25
o-ResNet32
89.82
92.31
93.69
93.65
93.03
93.07
92.49
o-ResNe156
89.88
92.71
92.88
93.81
93.8
93.49
92.73
93.03
o- 10
90.26
94.04
94.85
94.88
94.83
94.41
93.95
93.39
Furthermore, if the number of separated convolutions N can be optimized, we will have N = log()CK3)
(21)
and
min f([n,), # = O(C' log()CK)))
(22)
In Theorem T we keep both notations Ii and ni. This is because, for the channel condition, it is intuitive to see 711 ... NN 2 C1 means that the product of пι -. needs to occupy each node in the input channel C1= = Cin. This is equivalent to the less intuitive condition 91 0.....-... 9NN S CC2... Cy. Similarly, for the group convolution condition, 9r S min(C, Ci-1) means the number of groups can not exceed the imput and output channels of this group convolution. while n, 2 max (--) is less intuitive. A sketch of proof of Theorem T is given in Appendix A
Equations 18 and (19 mean that one of the internal kernel sizes should take K" or KW and the remaining ones take 1. Hence, the proposed optimized separable convolution shall have spatial separable configuration: a single kernel takes (K' KW) " or two kernels take (K" ,1) and (1, K.K.) The implementation details of the proposed optimized separable convolution scheme is described in Algorithm [ (Appendix
B Finally, the proposed optimized separable convolution is sparse con- nected. The hyperparameters of each separated convolution are given by Equations (17-119) and the proposed scheme can be efficiently implemented using a channel shuffle operation (Fig. lb
3 EXPERIMENTAL RESULTS
In this section, we carry out extensive experiments on benchmark datasets to demonstrate the effec- tiveness of the proposed optimized separable convolution scheme. In the proposed experiments, we use prefix dw-, d- s-, or to indicate that the conventional depth separable convolutions in the bascline networks are replaced with depth-wise, depth separable (dsep), spatial separable, the proposed optimized separable (osep), or the proposed spatial optimized separable convolutions. In this research, we set the number of separated convolutions N 2. The details of the training settings for the proposed experiments are described in Appendix C
3.1 EXPERIMENTAL RESULTS ON CIFAR10
CIFARIO Krizhevsky et al. 2009) is a dataset consist of 50,000 training images and 10,000 testing images. These images with resolution of 32 x 32 and are categorized into 10 object classes. In

---

Under review as a conference paper at ICLR 2022
the proposed experiments, we use ResNet (He et al. 2016) baselines and replace the conventional convolutions in ResNet with dsep and osep convolutions, resulting in d-ResNet and o- o-ResNet.
The proposed osep scheme significantly reduce the representa- tional complexity. In Section 2 we state that this reduction factor can be VCK in theory As illustrated by the solid lines in Fig. Ba). the orange solid curve lies in a region with significantly smaller v-values than the blue solid curve. This indi- cates that o-ResNet shall have sig- nificantly less parameters than the ResNet bascline. For example, the 110-layered o-ResNet110 10 has even fewer parameters (0. 180 million 0,270 million) than the 20-layered ResNet20, yet with noticeable higher accuracy (92.15% vs 91.25%). This demonstrates that the proposed osep scheme could significantly reduce the repre- sentational complexity for convolutions. For dsep, this reduction factor is 1/K=41/C. which is bounded by K2. For 3 3 kernels, this reduction can be at most 9. Whereas for the proposed osep scheme, no such bounds exist. The advantage of the proposed osep scheme over dsep is illustrated in Fig. 3aa by the orange and green solid curves. From this, we can see the proposed osep scheme is more efficient with smaller æ-values. We further plot -F.Os curves in Fig. 3bb) for reference, where similar conclusions can be drawn.
Table 3. Experimental results CIFARIO for DARTS. The pro- posed optimized separable convolution (o-DARTS) generalizes well to the DARTS architecture, and achieves improved accuracy with ap- proximately the same FL.OPs and fewer parameters. DARTS uses depth separable convolution and an optional d- is prefixed.
#Params
FLOPs
Accuracy
Eroo Rate
Net Arch
(million)
(billion)
(%)
(%)
(d (d-IDARTS Liu es 2018
3.35
0.528
97.24%
2.76%
o-DARTS
3.25
0.572
97.67%
2.33%
P-DARTS
Caen et al.
2019
3.43
0.532
97.50%
2,50%
PC-DARTS
Nu
2019
3.63
0.557
97.43%
2.57%
GOLD DARTS Ri
3.67
0.546
97.47%
2.53%
The proposed o-ResNets can have 10x-18x fewer parameters than the ResNet baselines in the pro- posed experiments. For fair comparisons, we introduce the channel multiplier in order to approxi- mately match the #Params (or FLOPs We use the suffix "_m<multt aenmullppplerrr> to indicate the channel multiplier. As illustrated in Fig. 3aa, from which we can see, the proposed osep scheme is much more efficient than conventional convolutions. The orange curve, including both solid and Pareto-frontier than the blue curve. Such repre-
dashed parts, achieved a better accuracy-#Parant sentation efficiency could result in a more regularized network with fewer parameters prevent over-fitting and possibly contribute to the final performance. In Fig. Fla), we also present the d- ResNet curves in dashed green by replacing the conventional convolutions with dsep convolutions. As can be seen, d' d-ResNet achieves good accuracy Seuuracy-#Params balances for small networks (e.g. d- ResNet20 and d d-ResNet322,, but performs comparable or no better than conventional convolutions for large ones (e.g. d- d-ResNet56 and d' d-ResNet1 10). In summary, the proposed osep scheme achieves better accuracy-tParra
(and also accuracy-FL.c FL.Ps as illustrated Fig. 3(b)) Pareto-frontiers than both conventional and dsep convolutions.
Besides conventional and depth separable (d-) convolutions, we compare the proposed osep (o-) scheme against the other convolution types, including depth-wise (dw-) and spatial separable (s-) convolutions. In the following, we shall omit the suffix of channel multiplier for simplicity, which shall be clear from the context. From Algorithm (Appendix (B), the proposed osep scheme also has a spatial separable (so-) variant. A comparison of all these convolutions for the ResNet architecture is illustrated Fig. C).. From this figure, we can conclude that the proposed osep scheme is more efficient than all other alternatives (the orange bar is highest).
Other Conv2D Types:
We discuss more about dw-ResNet in Fig. [c). Recall from Table 1 that a depth separable convolution is depth-wise convolution followed by a pointwise convolution. du-ResNet allows no channel information exchange while d-ResNet does. Fig. ((c) demonstrates that dw-ResNet performs much worse than deResNett In fact, du-ResNet is the only one that does not maintain the volumetric RF and performs worst of all these six convolution schemes. This
Channel Information Fusion:
For optimized separable, VCK-
63/2K" For depth separable, 1イK
C2 g²
CIC+KTJ < K?
K2
WWe match both #Params and FLOPs here. If this is not allowed, we approximately match for one and make sure the other not to exceed. In Appendix [] we present experimental results of matching #Params only in Fig.
5 and Fig. 6 The conclusions reached in this Section is the same.
7

---

Under review as a conference paper at ICLR 2022
10,ml
72-
70-
R53
Rea fet foaddins
1030
Reahst Dsec ind]
66-
minuppper
61-
55-
oftesheet mel koy
0.25
8:00
155
150
1.5
0.00
cis FLOPs (pilkon)
0.2
Aashe133 Aashv222
Asshest
Aatlet216
aParams (nd koo)
(a)
(b)
(c)
Figure 4. Experimental results on CIFARI00 for the ResNet architecture (best viewed in color). The proposed optimized separable convolution (o- (o-ResNet) achieves improved (a) accuracy- accuracy-#Parrms #FParams and (b) accuracy- -LOPs Pareto-frontiers than both the conventional (ResNet) and depth separable (d-ResNet) convolutions. (c) A com- parison for performances of different convolution schemes.
Table 4. Experimental results CIFARI00 for different overlap coefficients (7), I 2 1, the volumetric RF maintained, otherwise is not. Each row of ReRsNeet is channel multiplied. When y 1, the performance hurts due to discourage of channel information fusion.
Overlap Coef. (1)
(Dephhise)
1/16
1/4
4
16
64
+ (Conventional)
Vol. RF
/
/
/
o-ResNei20
65.46
68.24
70.96
71.03
71.12
70.8
70.08
67.38
o-ResNet32
66.42
70.59
71.05
72.75
70.89
71.91
70.76
68
o-ResNe156
65.39
69.23
69.87
73.55
72.4
71.98
70.98
69.34
o- 1-ResNet1 10
65.98
73.30
74.00
75.48
74.74
73.62
73.01
71.68
suggests that channel information fusion could be critical for the good performance of DCNN, and hence validates our proposed volumetric receptive field condition.
We carry out an ablation study on the overlap coefficient 7. For 7 2 1, the volumetric RF is maintained, otherwise, it is not. From Table we can see that, a good-performing y takes values 1 S 7 S 4 and 7 1 achieves the best. It is reasonable conjecture that, for 7 < 1, the volumetric RF is not maintained and the channel information fusion is compromised, leading to bad performance. For 7 4. the representation efficiency also slightly lower. We argue that this is because the channel information has already been fused sufficiently. For larger 7, more overlap introduces more cost yet no additional fusion, hence the efficiency has been degraded accordingly. In Table 2 also include the results for de-ResNet and ResNet, as they can be roughly viewed as the limit cases of y to be infinitely small (s) or infinitely large (+00). The ablation study on the overlap coefficient in Table 2 clearly demonstrates that we should satisfy the proposed volumetric RF condition.
Overlap Coefficient:
To demonstrate that the proposed osep scheme generalizes well to other DCNN architectures, we adopt the DARTS (V2) (Liu et al., 2018) network as the bascline. The DARTS evaluation network has 20 cells and 36 initial channels, increase the initial channels to 42 to match the FLOPPs By replacing the dsep convolutions in DARTS with the proposed osep convolutions, as illustrated in Table 3 the resulting o-DARTS improved the accuracy from 97. 97.24% to 97.67%, but with fewer parameters (3.25 million vs 3.35 million). It is worth noting that it is very hard to significantly improve the DARTS search space. In Table we also include three variants of DARTS, i.e. P-DARTS (Chen et al. 2019), PC-DARTS (Xu et al.) 2019). and GOLD-DARTY (Bi jt al. 2020), with more advanced search strategies for comparison. As can be seen, o-DARTS can achieve even higher accuracies than these advanced network architectures.
Generalization to DARTS:
3.2 EXPERIMENTAL RESULTS ON CIFAR 100
CIFAR 100 (Kizhevsky et al. 2009) is a dataset consist of 50,000 training images and 10,000 testing images. These images are with a resolution 32 32 and are categorized into 100 object classes. We carry out similar experiments on CIFAR100 those on CIFAR10 10, from which similar conclusions can be drawn.
From Fig. 4a) and (b), we can conclude that the proposed optimized separable convolution scheme achieves better accuracy-#Param #Params and accuracy-FLOPs frreto-frontiers than both conventional and depth separable convolutions. From Fig. 4 cc., we can that the proposed osep scheme is more
8

---

Under review as a conference paper at ICLR 2022
efficient than the other alternative Conv2D types, including depth- spatial separable, and the proposed spatial optimized separable convolutions. In Fig. 4c), dw-ResNet is the only one that does not maintain the volumetric RF and performs significantly worse than the other counterparts. In Table 4 the experimental results indicate the best overlap coefficient takes value 7 = 1. These latter two observations also demonstrate the validity of the proposed volumetric RF condition.
3.3 EXPERIMENTAL RESULTS ON IMAGENET
We evaluate the proposed optimized separable convolution scheme on the benchmark ImageNet (Deng et al. 2009) dataset, which contains 1.28 million training images and 50,000 testing images.
Table 5. Experimental results on full ImageNet for the DARTS ar- chitecture. The proposed o-DARTS achieves 74.2% topl accuracy with only 4.5 million parameters and the proposed o-MobileNet achieves 70.8% topl accuracy with only 3.0 million parameters.
#Params (million)
FLOPs
Top1 (%)
Topl Error
Net Arch
(billion)
(%%
2018
4.72
73.3%
26.7%
3.3.1 IMAGENET40
o-DARTS
4.50
0.554
74.2%
25.8%
Because carrying out experiments di- rectly on the ImageNet dataset can be resource- and time-consuming, resized all the images into 40 x 40 pixels. Due to space limitations, we present the experimental results ImageNet40 in Appendix and Table 7 We conclude that the proposed o-ResNet achieved 4-5% (e.g. 49.97% vs 44.93% for 56-layer and 50.72% vs 46.74% for 110-layer) perfor- mance gains comparing against the ResNet baselines.
Mo-iilbileNet (lloward.
4.20
70.6%
29.4
o-MobiloNet
3.00
0.564
71.1 %
28.9%
3.3.2 FULL IMAGENET
Similar the experiments CIFAR10, we replace the dsep convolutions in the DARTS (V2) network with the proposed osep convolutions to demonstrate that the proposed approach is able to generalize other network architectures. The experiment is carried out the full ImageNet dataset. The DARTS evaluation network has 14 cells and 48 initial channels, we increase the initial channel size 56 to match the original neural net. The resulting network is called o-DARTS. Experimental results are illustrated in Table 5 It can be seen that, with fewer parameters (4.50 million vs 4.72 million), the proposed o-DART'S network achieved higher accuracies both top1 (74.2% vs 73.3%) and top5 (91.9% vs 91.3%) accuracies than the DARTS baseline. Finally, we replace the dsep convolution in MobileNet (Howard et al., 2017) ) to the proposed osep convolution. Using only 3.0 million parameters, the proposed o-MobileNet is able to achieve 71.1% topl accuracy on the ImageNet dataset. This is a great gain comparing against the original MobileNet with 4.2 million parameters. We can conclude that the proposed osep is able achieve better accuracy-FLOP. and accuracy-#Parae #Params balances than dsep convolutions.
4 CONCLUSIONS
In this paper, we have presented yet another novel convolution scheme called optimized separable convolution to improve efficiency. Conventional convolution took a costly complexity O(C2 The proposed optimized separable convolution scheme is able to achieve its complexity at Ô K), which is even lower than that of depth separable convolution at O (C + K²)). Hence, the pro- posed optimized separable convolution has the full potential replace the usage of depth separable convolutions in a DCNN. Examples considered include ResNet, DARTS, and MobileNet architec- tures. The proposed optimized separable convolution also has a spatial separable configuration. A generalized N --separable case achieve better performance O lCK3))
We believe the proposed optimized separable convolution also has a potential impact on the AutoML community. The proposed novel operator is able to increase the neural architecture search space. In a multi-objective optimization formulation, where both accuracy and #Params (or FLOPs) are optimized, we expect the proposed optimized separable convolution operator. In the future, we also expect to carry out experiments more neural network architectures, e.g. EfficientNet, etc.
efficient network architccture can be discovered in the future using

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kaifeng Bi, Lingxi Xie, Xin Chen, Longhui Wei, and Qi Tian. Gold-nas: Gradual, one-level, differ- entiable. arXiv preprint arXiv:2007, 200,::0007,0333,,, 2020.
3 3.1
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 1294-1303, 2019. C
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1, arXiv preprint arXiv: /60...22....... 2016.
G
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. arXiv preprint Siti:20006.000444 2020. [
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
1 3.3
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv:1412.6115, 2014. G
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv: 150..0266,, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
2.3 3.1
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1xxx::55030025/1 2015.
G
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference Computer Vision, pp. 1314-1324, 2019. G
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /704 1704004611, 2017.
3.3.2 G
Yani loannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi. Training cnns with low-rank filters for efficient image classification. arXiv preprint W.1v:1511.06744
2015. G
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 2014. G
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
3.2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012. 1
Tony Lindeberg. A computational theory of visual receptive fields. Biological cybernetics, 107(6): 589- 635, 2013.
[ 2.2
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint /
2018.
3.
Ningning Ma, Xiangyu Zhang, Hai-Tar Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV). pp. 116-131, 2018.
10

---

Under review as a conference paper at ICLR 2022
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiell Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018. 2.3 G
Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for image classification. Ph. D. thesis, 2014. 0
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in neural information processing systems, pp. 2377-2385, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed. Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016. G
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: /905. 1946, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficiet architecture search. In International Conference on Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2022
Algorithm 1 The Algorithm for Optimized Separable Convolution
Input:
Input channel C1 Cin, output channel CN+1 Cout. kernel size (K* K."). number of sepa-
rated convolutions
Optional Input: internal kernel sizes (optional, preset), internal number of groups (optional, masked values), spatial separable (Truc or False), overlap cocfficient (ry = 1).
Output: internal channel sizes C2 -. CN. internal kernel sizes KJ W.II ," K-H internal number of
groups 91 .... .N
Calculate internal channel sizes C2, Cv Cin, Cous), max(Cinn Cous) 4, mnin(Cin, Cous), etc. according to a preset policy.
W if internal kernel sizes K 1'.., K are not given then
#r|
spatial separable then
Sct K [N/2] = K" K-W
[N/2+1] = K and all other internal kernel sizes to 1.
else
Set K HIW [N/2]
WW and all other internal kernel sizes 1.
end if
end if
Calculate internal channels per group according to 14 V -m c.m K# K
C4418 x Let gi = min([Ci/ (. Cr. C.+1). If Ci/ny < 1 or Ci/n; > min(Cr, C++1) for certain I, re-optimize g with a masked number of groups by pre-setting gi = 1 for E (1: Ci/mz < 1,, gi = min(Ci, Ct+1) for I E (T: Ct/T4 > min(Ci, C(+1)).
D Because ny ~ VC, for large channel sizes, we rarely need to re-optimize. KBW
Return C2,".., CNn; K!
A SKETCH OF PROOF FOR THEOREM
Sketch of Proof for Theorem equality, we can get
For Equation
12 after applying an arithmetic-geometric mean in-
(KHW) D2N 9.9 1....o..0 CR.NN R
(23)
2N KF..KKK..KK..... -KK
(24)
The equality holds and only C2n KM KW
ON+IIN KH KW Let ni
Bmn,, where and
C2K KW Let 3 113;, we can solve 71 1/2G
WK CKTKA
31
K
NT CII. 1= K 1=
KIIT KW VC.
(25)
CLIKHK"
Note that the inequality (24 holds for arbitrary K W We need to further optimize K H-W Again,
from the ceetmmetic-gemmetiie mean inequality again, we can get K" K <
+..4K > K
N
(**+ +N-1 ) and the equality holds if and only K want the inequality reversed, instead of finding the maximum of this product, we expect to find its minimum. This still gives us a hint, the maximum is achieved when the internal kernel sizes are as even possible, so the minimum should be achieved when the internal kernel sizes are as diverse as possible. In the extreme case, one of the internal kernel sizes should take KH and all the rest takes 1, i.c. Equations (18) and (19). A formal proof of this claim can be derived. Hence, we have [((n.), [K B(W
K K +N-1 However, we
N
}) 2 N 7/7, ... CN+1 K KW
(26)
ONN x
(27)
By setting ((N = O, we can derive that
N Log(nCk2),
(28)
and
min f(fnn), (K #l
eCHW log()CK3)
(29)
O(CHW log(nCK3)),
(30)
where e 2.71828.., is the natural logarithm constant.
12

---

Under review as a conference paper at ICLR 2022
Sata
94-
Ier Sepersee BH
Suita Oypiest Srparat
at.mag-
41-
92-
Tept
25
lat Daandinn
Reahet Dane Ind)
00-
50-
(Radker tovle)
00-
c ciaaahi (wiintt kor
Roliii Lnd mulpptrr
0dc
0.55
0.0
2.55
1.5
ch0 1.55
3.00
0.10
n.s.
6.35
5.55
0.50
*Param kniioon)
PLOTS (p4lio))
(a)
(b)
(c)
Figure 5. Experimental results on CIFARIO for the ResNet architecture (best viewed in color, same as Fig.
3 except networks are channel multiplied to match #Params). The proposed optimized separable convolution
(o-ResNet) achieves improved (a) accuracy-#Params and (b) accuracy-FLOOP; Pareto-frontiers than both the conventional (ResNet) and depth separable (d- (d-ResNet) convolutions. (c) comparison for performances of different convolution schemes.
Scate SOMMION 14 BU0
76-
70-
Spata Upoina Desamiittoo
72-
02
68-
06-
StReshett
Roaket Led' Botkerr
citestee
62-
0.00
0.25
0.50
6.75
Lis
150
172
2.00
0.05
0.10
0.15
0.23
050
Aasie123
Rarrvt22
Nashei228
kon)
(pilion)
(a)
(b)
(c)
Figure 6. Experimental results on CIFARIOO the ResNet architecture (best viewed in color, same as Fig. 4 Texcept networks are channel multiplied to match #Params). The proposed optimized separable convolution (o (o-ResNet) achieves improved (a) accuracy-##i -Paraass and (b) aecuracy-FLOP Ptreto-frontier than both the
conventional (ResNet) and depth separable (d-ResNet) convolutions. (c) A comparison for performances of different convolution schemes.
B THE PROPOSED OPTIMIZED SEPARABLE CONVOLUTION ALGORITHM
A detailed implementation of the proposed optimized separable convolution algorithm is illustration
in Algorithm
|
C TRAINING SETTINGS
Experiments on CIFARIO and CIFARI00 for the ResNet architecture The images are padded with 4 pixels and randomly cropped into 32 32 feed into the network. A random horizontal flip with probability of 0.5 is also applied. All the networks are trained with a standard SGD optimizer for 200 epochs. The initial learning rate set to 0.1, with a decay of 0.1 at the 100 and 150 epochs. The batch size is 128. A weight decay of 0.0001 and a momentum of 0.9 are used.
Experiments on CIFARIO for the DARTS architecture We follow the same training settings in (Liu et al. 2018): the network is trained with a standard SGD optimizer for 600 epochs with a batch size of 96. The initial learning rate is set to 0.025 with a cosine learning rate scheduler. A weight decay of 0.0003 and a momentum of 0.9 are used. Additional enhancements include cutout, path dropout of probability 0.2, and auxiliary towers with weight 0.4.
Experiments on ImageNet40 for the ResNet architecture Each network is trained with a stan- dard SGD optimizer for 20 epochs with the initial learning rate set to 0.1, and a decay of 0.1 the 10 and 15 epochs. The batch size is 256, the weight decay is 0.0001 and the momentum is 0.9.
Experiments on full ImageNet for the DARTS architecture We follow the training settings in (Chnn et al. 2019) for multi-GPU training: the images are random resized crop into 224 x 224
13

---

Under review as a conference paper at ICLR 2022
Table Experimental results on CI for the ResNet with inference time on Windows 10 Intel i5-8250 CPU or Nvidia GeForce RTX 2080 Ti GPU.
Arch
Channel Multiplier
iParams (million)
FLOPs (billion)
Accuracy (%)
CPU Infer
GPU Inler Time (s)
Time (8)
ResNer20
0.270
0.04055
91.25
0,0310
0.0057
G ReeNNec20
3.625
0.221
0.04070
93.37
0.0468
0.0120
d-RoaNet20
2.72
0.250
0,0400
92.66
0,0468
0.0060
RosNet32
0.464
0.06886
92.49
0,0469
0.0067
o-ResNet32
3.75
0.367
0,06726
93.69
0.0937
0.0185
<RReNNet32
0.429
0.06565
92.98
0.1154
0.0092
ResNe156
0.853
0.12548
93.03
0.0938
0.0101
o R-ReNNe566
3.875
0.682
0,12574
93.81
0.1562
0.0330
d-RoaNet56
2.75
0,786
0,11890
92.69
0,1875
0.0181
ResNet110
1,728
0.25289
93.39
0,1563
0.0178
3,875
1,337
0,24763
94.88
0,3216
0.0671
10
2.75
1,590
0,23870
93.40
0.3462
0.0317
patches with a random scale in [0.08, 1.0] and a random aspect ratio in [0.75, 1.33]. Random horizontal flip and color jitter are also applied. The network is trained from scratch for 250 epochs with batch size 1024 on 8 GPUs. An SGD optimizer with an initial learning rate of 0.5, a momentum of 0.9, and weight decay of 3e-5. The learning rate is decayed linearly after each epoch. Additional enhancements include label smoothing with weight 0.1 and auxiliary towers with weight 0.4.
Experiments on full ImageNet for the MobileNet architecture The images are random resized crop into 224 x 224 patches with random scale in [0.08, 1.0] and random aspect ratio in [0.75, 1.33]. Random horizontal flip is also applied (no color jitter). The network is trained from scratch for 200 epochs with batch size 1024 on 8 GPUs. An SGD optimizer with an initial learning rate of 0.064, a momentum of 0.9, and a weight decay of le-5. The learning rate is decayed with a rate of 0.973 for every 0.8 epoch.
D EXPERIMENTAL RESULTS FOR MATCHING #PARAMS
In this section, we report experimental results of matching #Params only, instead of matching both #Params and FLOPs. In Fig. and Fig. 6 we illustrate the experimental results on CIFAR1O and CIFAR 100 for the ResNet architecture. As can be seen, the observations in these two figures are consistent with those in Fig. and Fig. 4 Hence, the conclusions we reached in Section 3.1 and Section 3.2 are not affected. Please refer to these two sections for a detailed description of the experimental results.
E INFERENCE TIME FOR THE PROPOSED OPTIMIZED SEPARABLE CONVOLUTION
In this research, we focus on the representational efficiency of the proposed optimized separable scheme. The representational complexity is measured with the number of parameters (#Params) and hardware-indeepedent. For the proposed experiments, we included both #Params and FLOPs. In this section, we further report the wall-clock inference time of the proposed optimized separable convolution scheme for reference reasons. It is important to keep in mind that FLOPs measures the theoretical speed we are able to achieve. The wall-clock time reported in this section is hardware dependent. Slowness can occur due to an inefficient hardware implementation.
From T we can see that, under approximately the same FLOPs, both o-ResNet and d-ResNet take a longer inference time than conventional ResNet. This is because the current implementation of grouped convolution in Torch is not optimized. From Table we can also conclude that, under approximately the same FLOPs, ResNNet is slightly faster than d-ResNet (e.g. N-ResNet32 takes 0.0937s while d-ResNet32 takes 0.1 1545) on CPU and yet the former is about twice slower than the latter (c.g. o-ResNet32 takes 0.0185s while d-ResNet32 takes 0.0092s) on a GPU. The better wall-clock timing of the proposed osep scheme over dsep CPU may suggest that it also has
14

---

Under review as a conference paper at ICLR 2022
Table Experimental results on ImageNet40 the ResNet architecture. The proposed optimized separable convolution (o-ResNet) achieves 4-5% performance gain over the ResNet baseline.
Channel Multiplier
#Params (million)
FLOPS
Accuracy (%)
Error Rate
(billion)
(%)
ResNer20
4.58
0.162
40.28
59.72
o-RexNei20
5.375
5.13
0.160
44.94
55.06
ResNet32
7.68
0,275
42.98
57,02
o-RosNet32
5.75
7.78
0.278
47.88
52.12
ResNet56
13.88
0.502
44.93
55.07
o-ResNer56
6.0
12.55
0.497
49.97
50.03
ResNetl
27.83
1.012
46.74
53.26
R-ResNet1
6.25
23.79
1.027
50.72
advantages for ARM CPU architectures. Hence, the proposed osep scheme could be more efficient for mobile applications.
There are good reasons for the slowness of the proposed osep on a GPU. In fact, there are two extra copies of blobs in our current Python implementation of the proposed osep convolution (one for group convolution if the number of groups does not divide the input output channels, and the other one for the channel shuffle operation). These two extra copics of blobs can actually be avoided for an efficient implementation. However, optimizing this code shall require careful tweak of the CUDNN library. It is known that on a GPU, the memory access cost can dominate over the computational cost (Ma et al. 2018). Hence, the slowness of the proposed osep scheme shall occur. On a CPU, the computational cost dominates over the memory access cost. Hence, the proposed osep faster than dsep. In the future, expect the bottleneck of memory access for GPU could be addressed and the wall-clock timing of the proposed osep scheme could be greatly sped up.
F EXPERIMENTAL RESULTS ON IMAGENET40
Because carrying out experiments directly on the ImageNet dataset can be resource- and time- consuming, we resized all the images into 40 x 40 pixels. A 32 x 32 patch is randomly cropped and a random horizontal flip with probability of 0.5 applied before feeding into the network. No extra data augmentation strategies are used. The baseline ResNet architecture is a modified version of that used on the CIFAR dataset, except that the channel sizes are set to be 4x larger, the features are calculated scales of [16, 8. 4], and the last fully-connected (FC) layer outputs 1000 categories for classification. We make this modification because the ImageNet dataset has sig- nificantly more training samples than the CIFARI0 dataset. Experimental results are illustrated in T as can be seen, by substituting conventional convolutions with the proposed optimized sep- arable convolutions, the resulting o-ResNet achieved 4-5% (e.g. 49.97% vs 44.93% for 56-layer and 50.72% vs 66.74% for 100laaeer performance gains comparing against the ResNet baselines. This demonstrates that the proposed optimized separable convolution scheme is much more efficient. For o-ResNet56 and o-ResNetl 10, they also have fewer parameters which could contribute to a more regularized model. For o-ResNet20 and o-ResNet32, they have slightly more parameters because the last FC layer accounts for great portion of overhead for 1000 classes.
G RELATED WORK
There have been many previous works aiming reducing the amount of parameters in convolution. Network pruning (Han et al. 2013) strategies are developed to reduce redundant parameters that are not sensitive to performances. Quantization and binarization (Giong al., 2014 Courbariaux al.) 2016) techniques are introduced to compress the original network by reducing the number of bits re- quired to represent each parameter. Low-rank factorization methods (Jaderberg et al. 2014 Ioannou et al. 2015) are designed to approximate the original weights using matrix decomposition. Knowl- edge distillation (Hinton et al. (2015) is applied to train a compact network with distilled knowledge from large ensemble model. However, all these existing methods start from a pre-trained model. Besides, they mainly focus on network compression and have limited or no improvements in terms of network acceleration.
15

---

Under review as a conference paper at ICLR 2022
Among various implementations of convolution, separable convolution has been proven to be more efficient in reducing the representational demand. Depth separable convolution is explored exten- sively in modern DCNNs (Howard et al. 2017 Sandler et al. 2018 Howard et al., 2019 Liu al. 2018 Tan & Le 2019h. reduces the representational cost of a conventional convolution from O(C K2) to OCC (C K2)). However, the proposed optimized separable convolution is even more efficient than depth separable convolution. It can be represented at O K) and has the full potential to replace the usage of depth separable convolutions. A second advantage of the proposed optimized separable convolution that it can applied to fully connected layers we view them as 1 x 1 convolutional layers, whereas depth separable convolution cannot. Further, depth separable convolution requires the middle channel size to be equal to the input channel size, whereas for the proposed optimized separable convolution, the middle channel size can be freely set.
Spatial separable convolution was originally developed to speed up image processing operations. For example, Sobel kernel is a x 3 kernel and can be written (1, 1)" (-1,0, 0, 1). Spatial separable will require instead of parameters while doing the same operation. Spatial separable convolution is also adopted in the design of modern DCNNS. For example, in (Szegedy al. (016)) the authors introduce spatial separation to the GoogLeNe LeNe (Szegedy et al., 2015) architecture. For the proposed optimized separable convolution, there also a spatial separable configuration.
In the body of literature, separable convolution is also referred as factorized convolution or convolu- tion decomposition. In this research, the proposed scheme called optimized separable convolution following the naming conventions of depth and spatial separable convolutions.
16


Reviews:
[{"review_id": "zcXYTJJO6_2", "reviewer": "Reviewer_yy2C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n1.\tThis paper introduced the new design in potential by generalizing the depth-separable convolution. The proposed method extend common separable convolutions to more generalized & improved separable convolution.\n2.\tIt demonstrated that generalized separable methods surpassed previous convolution methods in performance with a small number of parameters compared with existing separable methods.\n3.    They have studied concrete analysis for flop's complexities. This supports that their approach is optimal among group separable convolutions.\n\nCons.\n1.\tThe arguments about ad hoc design are self-contradictory. The introduction section mentioned that \"Instead of setting these hyperparameters in an ad hoc fashion, we design a novel and principled scheme.\" However, in the experiment section, it seems that optimal separable methods also require hyperparameter (namely, overlap coefficient) tuning to obtain the best results.\n2.\tThe authors claimed that by introducing the proposed convolution, the model size could be reduced. However, we can observe from Table 3 that model size is reduced by 0.1M (approximately 3 percent). This is a bit incremental even they achieved 1% accuracy up.\n3.\tIn appendix section E, we can see that the GPU inference time of the proposed method is slower than other methods. To explain such a disadvantage, the authors say that PyTorch's implementation of group convolution is inefficient. However, it is believed that whenever group convolution is implemented in an efficient manner when considering the architecture of GPU, one-by-one convolution is the most memory-efficient design choice compared to other design choices, including group convolution.\n", "summary_of_the_paper": "This paper proposed an optimized version of depth separable convolution. Optimal separable convolution reduces the model size by replacing both depth-wise and point-wise convolution with group convolution. Furthermore, they allow overlapped channel between group convolution and this was swept to show ablation comparisons. The authors showed that the combination of group convolution methods outperformed the previous ones when the volumetric receptive field (RF) conditions were met.", "summary_of_the_review": "Overall, I think the authors have done a great job in explaining solid calculation for their optimum. However, their results are not convinced in terms of real elapsed time (slower GPU times) and marginal improvements in gain on number of parameters. The structure and representation of their work is good but empirical evidence is not sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "earnT1L3jh", "reviewer": "Reviewer_VokC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper proposes a new extension of the seperable convolution family that attempts to automatically select internal parameters that optimize model complexity and expressiveness.\n- The paper provides interesting analysis.\n- There's a definite use case in the field of AutoML for a block of this type. It would be interesting to see this evaluated.\n- The work shows improvement in common Cifar and ImageNet benchmarks.\n- Satisfying the volumetric receptive field criteria was an interesting addition.\n\nCons:\n- The paper is overall somewhat difficult to follow and could benefit from improved explanations throughout.\n- The paper should have discussed attributes of efficient models beyond just FLOPS and params, such as memory overhead, FLOP vs memory balancing (which seperable convolutions often suffer from), etc.\n- The paper could benefit from more advanced network evaluations.\n\nWhile I found the proposals by the authors to be appealing, I personally found the paper difficult to read, and would encourage the authors to further improve explanations of concepts and overall paper flow, as it was rather difficult to follow at times.", "summary_of_the_paper": "The authors propose a new convolutional operation designed to optimize the internal number of groups, and kernel sizes for a given model size and performance tradeoff. The authors compare their results with similar seperable convolution methods, and evaluate their methods and show performance gains on common benchmarks.", "summary_of_the_review": "Overall I find the work to propose an appealing convolutional operation, in that it attempts to satisfy optimal design for a given model size and complexity. I believe the paper was difficult to read and would benefit from improved explanations of concept and flow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cbil2Q_Cmg", "reviewer": "Reviewer_a8wG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1) Even if we do not discuss the NAS-net methods, the author should compare with other efficient CNN backbone designs, not limit the scope to separable convolutions and handcrafted optimization, for example, the IGCV series [1],  the Shift operations [2][3], learning the group strategies [4].\n\n(2) The author should consider more criteria for the \"efficient\" architecture design, for example, the exact inference speeds (which are more important but listed in the appendix), the memory and time consumption during training [5]. This paper only shows the theoretical #FLOPs and # parameters, however, there is a huge gap between #FLOPs/parameters and the practical inference speed [6], which may prevent the practical implementation for the CNN community. \n\n(3) The proposed optimization method seems quite heuristic, a description of motivation or theoretical proof would make the paper much more impressive.\n\n[1] Sun, Ke, et al. \"Igcv3: Interleaved low-rank group convolutions for efficient deep neural networks.\" arXiv preprint arXiv:1806.00178 (2018).\n\n[2] Wu, Bichen, et al. \"Shift: A zero flop, zero parameter alternative to spatial convolutions.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[3] Chen, Weijie, et al. \"All you need is a few shifts: Designing efficient convolutional neural networks for image classification.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[4] Huang, Gao, et al. \"Condensenet: An efficient densenet using learned group convolutions.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n\n[5] Wu, Shuang, et al. \"Convolution with even-sized kernels and symmetric padding.\" Advances in Neural Information Processing Systems 32 (2019): 1194-1205.\n\n[6] Ma, Ningning, et al. \"Shufflenet v2: Practical guidelines for efficient cnn architecture design.\" Proceedings of the European conference on computer vision (ECCV). 2018.", "summary_of_the_paper": "This work proposes an optimized separable convolution by optimal designs for the internal number of groups\nand kernel sizes for separable convolutions, thereby achieving better trade-offs between the model complexity and task performances.\nThe authors have done experiments and demonstrated the superiority of proposed methods on typical image classification tasks such as  CIFAR and ImageNet, as well as the generalization scalability on NAS-nets such as DARTs.\n", "summary_of_the_review": "Overall, the idea and content of this paper are quite clear, the main concerns from the reviewer are the novelty of the proposed method and practical implementation for the CNN community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cHrUa4ZV_5H", "reviewer": "Reviewer_bjqr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "==== Weaknesses ====\n\nThis work seems more to be a structured analysis for motivating spatial separable and grouped convolutions and a careful way for determining the groups. \n\nIt seems a bit misleading to call this operator optimal, since the problem described is not completely the general case for convolutions. The channels are fixed in the intermediate convolutions and there are many other factors that could be introduced, such as spatial grouping factors or dilation factors in all dimensions. However, analytically and empirically this does seem like a good guide for setting the hyperparameters for spatial, grouped convolutions.\n\nThere are recent related works that should be included in the paper that deal with determining the groups in convolutions. \"Differential Learning to Group Channels ...\" (ICCV19) discusses an end-to-end way to determine these group factor. It would be interesting to compare how an empirical method like this compares with your analytical method. \"Fully Learnable Group Convolution ... \"(CVPR19) proposes a similar fully learnable group convolution. I believe the CondenseNets (CVPR18) may also have a learned group convolution. I understand that your method is more than just determining the number of groups, but especially in the '-o' case, not '-so', it seems the major contribution is the group number.\n\nComputational complexity does not seem to be the most important metric here, since complexity usually deals with limit behavior. Many of these values, especially g and K, and usually small.\n\nIt would be helpful to fill out the curves more on the CIFAR comparisons, either though more layers or wider networks.\n\nSubfigures a and b on Figure 3 and 4 seem mostly redundant since as mentioned in the paper they should be multiples of HW from each other. Also, the saddle point figure does not add much to the paper.\n\nIt seems most results are '-o', not '-so', which I interpret as not having the factored spatial component. I could be wrong on this, so please correct me. In that case, then then both the first and second convolution would have the full spatial components (e.g., 3x3) and this seems fundamentally different than a 'dsep' convolution since the spatial aggregation is much higher. This is like a higher spatial gamma factor (as opposed to the channel gamma introduced in the conditions).\n\nThe evaluation section especially on ImageNet could use more modern efficient architectures, like MobileNetV2 or EfficientNetV2 or others. Also, the experiments are interesting comparing depthwise to optimized, but it seems like all the of architectures are custom.\n\nThe takeaway from Table 4 and the ablation study about channel fusion seems fairly well-documented already in papers like ShuffleNet (CVPR18), Unitary Group Convolutions (CVPR19), and Interleaved Group Convolutions (CVPR17). This last work actually proposes a two-factored convolution that may serve as a reasonable comparison for this optimized convolution.\n\n\n==== Strengths ====\n\nThis paper is very applicable since most efficient models include some grouped or factored or sparse convolutions. \n\nDiscussion of the trivial or degenerate cases help build better understanding of the problem, e.g. degenerate scaling operation.\n\nTable 1 was a useful summary of existing methods.\n\nThe appendix included useful discussions on inference time.\n\n\n==== Questions ====\n\nWhat are the tradeoffs for these factored convolutions? The more factored convolutions (i.e. as N increases) seem to decrease the representational capacity. When would N > 2 be useful?\n\nI'm a little confused when the results including the spatially factored convolutions in the results and when they don't. This confusion comes from the first paragraph of Section 3, when it discusses 'o-' and '-so'. \n\nAutoML was mentioned in the conclusion. How do these operators compare to those returned by a traditional AutoML procedure where the groups and K factors were learnable?\n\nDo any architectures or previous work already combine spatial and group convolutions?\n", "summary_of_the_paper": "The convolution operator is the fundamental unit of most modern DNNs. This paper summarizes the existing (efficient) convolution operators and formulates an optimization problem to choose a few important parameters for the convolution. The authors show that their convolution under certain constraints requires fewer parameters and operations compared to the commonly used operators. They evaluate their proposed operator on CIFAR10, CIFAR100, and on ImageNet.", "summary_of_the_review": "I believe this paper provides an interesting analysis suggesting how to configure the hyperparameters in efficient convolutional operators. The novelty from my reading of the paper was mostly confined to the analysis and conclusions around group number. I thought there were some key areas of related works missing, and the evaluation section was promising but a bit underwhelming relying mostly on CIFAR and custom ResNet models. Overall, I would argue that this paper should be a borderline reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a convolutional-like operator called optimized separable convolution, which scales well in the number of channels, C. The paper studies the classification performance of ResNet with the optimized separable convolution, and with other choices of the convolutional operators. The paper finds the introduced convolutional  operator to be more parameter efficient than competing operators, however only by a relatively small margin, and this also comes at the cost of computational performance. 

The reviewers appreciated that the proposed convolutional operation is more parameter efficient and that any improvement in convolutional networks potentially benefits a large array of methods and tools. The reviewers  criticize that the operation only offers marginal gains at the cost of slower runtimes, and agree that the contribution is only marginally significant and therefore below the acceptance threshold.

I agree with the reasoning of the reviewers that the extra computational cost is not worth the marginal improvement, and therefore recommend to reject the paper. Also, the authors didn't respond to the comments of the reviewers and their reasonable questions.
Decision: Reject