DyNet: Dynamic Convolution for Accelerating Convolution Neural Networks
Authors: Kane Zhang
Jian Zhang
Qiang Wang
Zhao Zhong
Venue:  (2020)
Keywords: CNNs
dynamic convolution kernel

Abstract: Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost. To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models. Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between convolution kernels. To address this issue, we propose a novel dynamic convolution method named \textbf{DyNet} in this paper, which can adaptively generate convolution kernels based on image contents. To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs. The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged. Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 40.0%, 56.7%, 68.2% and 72.4% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1.0%, -0.27%, -0.6% and -0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87x,1.32x and 1.48x on CPU platform respectively. To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

Under review as a conference paper at ICLR 2020
DYNET: DYNAMIC CONVOLUTION FOR ACCELERAT- ING CONVOLUTION NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost. To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models. Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between con- volution kernels. To address this issue, we propose novel dynamic convolution method named DyNet this paper, which can adaptively generate convolution kernels based on image contents. To demonstrate the effectiveness, we apply DyNet on multiple oftth- CNNs. The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged. Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0). ResNet18 and ResNet50, DyNet reduces 37.0%, 54.7%, 67.2% and 71.3% FLOPs respec- tively while the Top-1 accuracy on ImageNet only changes by +1.0%, -0.27% -0.6% and -0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87 x, 1.32x and 1.48x on CPU platform respectively. To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.
INTRODUCTION
Convolutional neural networks (CNNs) have achieved statc-of-the-art performance in many computer vision tasks (Krizhvssyy et al., 2012, Szegedy et al.) (013)), and the neural architectures of CNNs are evolving over the years (Krizhevsky et al. 2012 Simonyan & Zisserman 2014 Szegedy et al. 2015 al.] 201 Sa b). However, modern nigh-performance CNNs
He et al.] 2016 Hu et al.] 2018|/Zhon often require a lot of computation resources to execute large amount of convolution kernel operations. Aside from the accuracy, to make CNNs applicable on mobile devices, building lightweight and efficient deep models has attracting much more attention recently (Howard et al. [0177] Sandler et al. 2018, Zhang et al. 2018 Ma et al. 2018). These methods can be roughly categorized into two types: efficient network design and model compression. Representative methods for the former category MobileNet (Howard et al. 2017] (andler et al., 2018) and ShuffleNet (Ma et al. 2018 (Zhang et al., 2018). which use depth-wisg wise separable convolution and channel-level shufile techniques reduce computation cost. On the other hand, model compression based methods tend to obtain a smaller network by compressing a larger network via pruning, factorization or mimic (Chen al. 2013 Han
et al.
2015a: Jaderberg et al. 2014 Lebedev et al. 2014 Ba & Caruana 2014).
Although some handcrafted efficient network structures have been designed, we observe that the significant correlations still exist among convolutional kereels, and introduce large amount of re- dundant calculations. Moreover, these small networks are hard to compress. For example, Liu et al. (2019) compress MobileNetV2 to 1 124M, but the accuracy drops by 5.4% on ImageNet. We theoretically analyze above observation, and find that this phenomenon is caused by the nature of static convolution, where correlated kernels are cooperated to extract noise-irrelevant features. Thus it is hard to compress the fixed convolution kernels without information loss. We also find that if we linearly fuse several convolution kernels to generate dynamic kernel based on the input, we obtain the noise-irrelevant features without the cooperation of multiple kernels, and further reduce the computation cost of convolution layer remarkably.

---

Under review as a conference paper at ICLR 2020
Dynamic Generation
Fixed convolution kernlss
Cou
Cj
Cin
k
Coefficient Prediction
11
100
Cin
W
Cout
Convolution
Dynamic convolution kernels
Cout
Figure 1: The overall framework of the proposed dynamic convolution.
Based on above observations and analysis, in this paper, we propose a novel dynamic convolution method named DyNet. The overall framework of DyNet is shown in Figure [ which consists of coefficient prediction module and a dynamic generation module. The coefficient prediction module trainable and designed predict the coefficients of fixed convolution kernels. Then the dynamic generation module further generates dynamic kernel based on the predicted coefficients.
Our proposed dynamic convolution method is simple implement, and can be used as a drop-in plugin for any convolution layer to reduce computation cost, We evaluate the proposed DyNet on state-of-ther the-art networks such as MobileNetV2, ShuffleNetV2 and ResNets. Experiment results show that DyNet reduces 77.0% FLOPs of ShuffleNetV2 (1.0) while further improve the Top-1 accuracy on ImageNet by 1.0%. For MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 5%, 67.2 67.2% and 71.3% FLOPs respectively, the Top-1 I accuracy ImageNet changes by -0.27% -0.6% and -0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87 x.132 and 1.48x CPU platform respectively.
RELATED WORK
We review related works from three aspects: efficient convolution neural network design, model compression and dynamic convolutional kernels.
2.1 EFFICIENT CONVOLUTION NEURAL NETWORK DESIGN
In many computer vision tasks Krizhevsky a key role. The increasing demands of high quality networks on mobile/embedding devices have driven the study efficient network design (He & Sun 2015). For example, GoogleNet (Szegedy) et al. 2015) increases the depth of networks with lower complexity compared to simply stacking convolution layers; SqueezeNet (Tandola et al. 2016) deploys bottleneck approach design very small network; Xception (Chollet 2017). MobileNet (Howard et al. (2017) and MobileNetV2 (Sandler al., 2018) use depth-wiee wise separable convolution to reduce computation and model size. ShuffleNet (Zhang et al. 2018) and ShuffleNerV2 (Ma et al. 2018) shuffle channels to reduce computation of x convolution kernel and improve accuracy. Despite the progress made by these efforts, we find that there still exists redundancy between convolution kernels and cause redundant computation.
2012 Szegedy al. 2013), model design plays
2.2 MODEL COMPRESSION
Another trend to obtaining small network is model compression. Factorization based methods
(Jaderberg et al., 2014 Lebedev et al. 2014) try to speed up convolution operation by using tensor
decomposition to approximate original convolution operation. Knowledge distillation based methods
(Ba & Caruana 2014; Romero et al.) 2014 Hinton et al. 2013) learn a small network to mimic a
larger teacher network. Pruning based methods (Han al. 2015a Wen al. 2016 Liu et al. 2019)
2

---

Under review as a conference paper at ICLR 2020
try to reduce computation by pruming the redundant connections convolution channels. Compared with those methods. DyNet is more effective especially when the target network is already efficient enough. For example, in (Liu et al., 2019), they get smaller model of 124M FLOPs by pruning the MobileNetV2, however it drops the accuracy by 5.4% on ImageNet compared with the model with 291M FLOPs. While in DyNet, we can reduce the FL.OPs of MobileNetV2 (1.0) from 298M to 129M with the accuracy drops only 0.27%.
2.3 DYNAMIC CONVOLUTION KERNEL
Generating dynamic convolution kernels appears in both computer vision and natural language
processing (NLP) tasks.
In computer vision domain, Klein et al. (Klein et al. 2013) and Brabandere et al. (Jia et al. 2016) directly generate convolution kernels via linear layer based the feature maps of previous layers. Because convolution kernels has a large amount of parameters, the linear layer will be inefficient on the hardware. Our proposed method solves this problem via merely predicting the coefficients for linearly combining static kernels and achieve real speed up for CNN on hardware. The idea of linearly combining static kernels using predicted coefficients has been proposed by Yang et al. (Yang) et al. 2019). but they focus on using more parameters to make models more expressive while we Tocus on reducing redundant calculations in convolution. We make theoretical analysis and conduct correlation experiment to prove that correlations among convolutional kernels can be reduced via dynamically fusing several kernels.
In NLP domain, some works (Shen et al. 2018 Wu et al.) 2019 Gong et al., 2018) incorporate context information to generate input-aware convolution filters which can changed according to input sentences with various lengths. These methods also directly generate convolution kernels via a linear layer, etc. Because the size of CNN in NLP smaller and the dimension convolution kernel is one, the inefficiency issue for the linear layer is alleviated. Moreover, Wu et al. (Wu et al., 2019) alleviate this issue utilizing the depthwise convolution and the strategy of sharing weight across layers. These methods designed to improve the adaptivity and flexibility of language modeling, while our method aims to cut down the redundant computation cost.
DYNET: DYNAMIC CONVOLUTION IN CNNS
In this section, we first describe the motivation of DyNet. Then we explain the proposed dynamic convolution in detail. Finally, we illustrate the DyNet based architectures of our proposed Dy-mobile, Dy-shuffle, Dy-Re ResNe18,, Dy-ResNet50.
-03
-022
-011
0.1
0.2
0.1
0.5
0.6
0.7
0.9
Correlation valuee
MobileNerV2
ResNe150
Vgg
Figure 2: Pearson product-moment correlation coefficient between feature maps. S, M. W. N denote strong, middle, weak and correlation respectively.

---

Under review as a conference paper at ICLR 2020
3.1 MOTIVATION
As illustrated in previous works (Han et al. 2015a b. Wen et al., 2016 Liu et al. 2019), convolutional kernels are naturally correlated in deep models. For some of the well known networks, we plot the distribution of Pearson product-moment correlation coefficient between feature maps in Figure 2 Most existing works try to reduce correlations by compressing. However, efficient and small networks like MobileNets are harder to prune despite the correlation is still significant. We think these correlations are vital for maintaining the performance because they are cooperated to obtain noise- irrelevant features. We take face recognition as an example, where the pose or the illumination is not supposed to change the classification results. Therefore, the feature maps will gradually become noise-irrelevant when they go deeper. Based on the theoretical analysis in appendix [A] we find that we dynamically fuse several kemels,, we can get crrise-ireeevant feature without the cooperation of redundant kernels. In this paper, propose dynamic convolution method, which learns the coefficients to fuse multiple kernels into a dynamic one based on image contents. We give more in depth analysis about our motivation in appendix A
3.2 DYNAMIC CONVOLUTION
The goal of dynamic convolution to learn a group of kernel coefficients, which fuse multiple fixed kernels to dynamic one. We demonstrate the overall framework of dynamic convolution in Figure We first utilize a trainable coefficient prediction module to predict coefficients. Then we further propose dynamic generation module fuse fixed kernels dynamic one. We will illustrate the coefficient prediction module and dynamic generation module in detail in the following of this section.
D
Coefficient prediction module Coefficient prediction module is proposed to predict coef- ficients based on image contents. As shown in Figure 3 the coefficient prediction module can be composed by global average pooling layer and a fully connected layer with Sigmoid as activation function. Global average pooling layer aggregates the input feature maps into 1 X 1 X Cin vector, which serves as feature extraction layer. Then the fully connected layer further maps the feature into 1 1 x C' vector, which are the coefficients for fixed convolution kernels of several dynamic convolution layers.
Input fenture map
Gun
Coefficient prediction
GAP
FC
1x 1x Ca.
1x1xc
Figure 3: The coefficient prediction module.
Dynamic generation module For dynamic convolution layer with weight [Cout 9t. 9t. Can. k k)., it corresponds with Cout 9e fixed kernels and C.nt dynamic kernels, the shape of each kernel is [Can: k., k). 9r denotes the group size, it is a hyperparameter. We denote the fixed kernels wf, the dynamic kernlls Wh, the coefficients as ni- where O,... Cout, i O,... It-
After the coefficients are obtained, generate dynamic kernels follows:
gr
Wc
w
Training algorithm For the training of the proposed dynamic convolution, it is not suitable to use batch based training scheme. It is because the convolution kernel different for different input images in the same mini-batch. Therefore, we fuse feature maps based on the coefficients rather than kernels during training. They are mathematically equivalent as shown in Eq. 2
o, w
(TÉ 8
(ni
w 2 x)
(2)
91
Σ
ini (w) 8 x)) = Σ (ni Oi),
4

---

Under review as a conference paper at ICLR 2020
where denotes the input, O denotes the output of dynamic kernel Wy, Oj denotes the output of fixed kernel wy-
Input
Input
Input
Input
Dy-Com
Golbal Pool
Chaunel Sptit
Dy-Cons
Golbal
Dy-Com Golbal Pool
Dy.Coou
PC
FC
Ty-Corn
FC
FC
Py-Com
Dy.Cluv
Dy.Com
Cy-Comy
Dy.Com
Chaunel Slafflfe
(a) Dy-mobile
(b)yyfhuffte
(e)Dy-ResNetls
(d)Dy-ResNet50
Figure 4: Basic building bolcks for Dynamic Network variants of MobileNet (a), shuffleNet Re)eRe (c), and ResNet50 (d).
3.3 DYNAMIC CONVOLUTION NEURAL NETWORKS
We equip the MobilcNetVV2, ShuffleNet22 and ResNets with our proposed dynamic convolution, and propose Dy-mobile, Dy-shuffle, Dy-ResNet18 and Dy-ResNet50 respectively. The building blocks of these network are shown in Figure Based on above dynamic convolution, each dynamic kernel can get woise-irrelevane feature without the cooperation of other kernels. Therefore we can reduce the channels for each layer those base models and remain the performance. We set the hyper-parameter as 6 for all of them, and we give details of these dynamic CNNs below.
Dy-mobile In our proposed Dy-mobile, we replace the original MobileNetV2 block with dy- mobile block, which is shown Figure (a). The input of coefficient prediction module is the input of block, produces the coefficients for all three dynamic convolution layers. Moreover, we further make two adjustments:
We do not expand the channels in the middle layer like MobileNetV2. If we denote the output channels of the block as Cout, then the channels of all the three convolution layers will be Cout-
Since the depth-wise convolution is efficient, we set groups = Casi for the dynamic depth- 6
wise convolution. We will enlarge Cout to make it becomes the multiple of 6 if needed.
After the aforementioned adjustments, the first dynamic convolution layer reduce the FLOPs from 6C2 HW to C2 HW. The second dynamic convolution layer keep the FLOPs as 6CHW 32 unchanged because we reduce the output channels by 6x while setting the groups of convolution 6x smaller, too. For the third dynamic convolution layer, we reduce FLOPSs from 6C HW to C HW as well. The ratio of FLOPs for the original block and our dy-mobile block is:
6C HW 6CHW x 32 + HW 6C + =
135
(3)
=
C2HW + 6CHW x 3 + C2
C 27
C 27
Dy-shuffle In the original ShuffleNet V2, channel split operation will split feature maps right- branch and left-branch, the right branch will go through one pointwise convolution, one depthwise convolution and one pointwise convolution sequentially. We replace conventional convolution with dynamic convolution in the right branch as shown in Figure (b). We feed the input of right branch into coefficient prediction module to produce the coefficients. In our dy-shuffle block, we split channels into left-branch and right-branch with ratio 1. thus reduce the 75% computation cost for two dynamic pointwise convolutuon. Similar with dy-mobile, we adjust the parameter "groups" in dynamic depthwise convolution to keep the FLOPSs unchanged.
5

---

Under review as a conference paper at ICLR 2020
Dy-ResNet18500 In Dy- Dy-ResNet18 and DeResNet50, we simple reduce half of the output channels for dynamic convolution layers of each residual block. Because the input channels of each block is large compared with dy-mobile and dy-shuffle, Figure 4 (d) to reduce the amount of parameters. the input channel is Cin, the output channels of the first linear layer will Si for Dy-ResNet18/50.
two linear layer shown in Figure 4 (c) and
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
For the training of the proposed dynamic neural networks. Each image has data augmentation of randomly cropping and flipping, and is optimized with SGD strategy with cosine learning rate decay. We set batch size, initial learning rate, weight decay and momentum as 2048, 0.8, 5e-5 and 0.9 respectively. We also use the label smoothing with rate 0.1. We evaluate the accuracy on the test images with center crop.
4.2 EXPERIMENT SETTINGS AND COMPARED METHODS
We evaluate DyNet ImageNet RRussakovsyy
itt al.) 2015). which contains 1.28 million training images and 50K validation images collected from 1000 different classes. We train the proposed networks on the training set and report the top-1 error on the valida- tion set. To demonstrate the effectiivnesss, we compare the proposed dynamic convolu- tion with state-of-the-at networks under mo- bile setting, including MobileNetV1 (Howard et al. 2017). MobileNetV2 (Sandler et al.1) ShuffleNet (Zhang et al.] 2018). ShuffleNet
70
MRopps
-Mislelee2
V2 (Ma et 2018)) Xception (Chollet 2017).
Figure 5: Compare with MobileNetV2 under the similar Flops constraint.
DenseNet (Huang et al., (2077). IGCV2 (Xie
ct al. 2018 and IGCV3 (Sun et al. 2018)
Comparison of different network architectures over classification error and computation cost. The number in the brackets denotes the channel number controller (Sandler al. 2018).
Table 1:
Methods
MFLOPs
Top-1 err. (%)
Dh-shuffle (1.0)
92
29.6
Dy-mobile (1.0)
135
28.27
Dy-Re Dy-Re
567
31.01
Dy- Dy-ResNet50
1119
23.75
ShuffleNet v1 (1.0) (Zhang et al.
2018
140
32,60
MobileNet v2 (0.75) (andler 2018)
2018
145
32.10
MobileNet v2 (0.6) (Sandler et al.
141
33.30
MobileNet (0.5) Howard
2017
149
36.30
DenseNet (1.0) (Huang et all
201
142
45.20
Xception (1.0) (Chollet 2017
145
34.10
IGCV2 (0.5) (Xie et al.
2018
156
34.50
IGCV3-D (0.7) (Sun et al.
2018
210
31.50
ShuffleNet V2 (1.0) (Ma et al.
2018
146
30.60
MobileNerV2 (1.0) (Sandler al. ResNet18
2018
298
28.00
1730
30.41
ResNet50
3890
23.67

---

Under review as a conference paper at ICLR 2020
4.3 EXPERIMENT RESULTS AND ANALYSIS
Analysis of accuracy and computation cost We demonstrate the results in Table 1 where the number in the brackets indicates the channel number controller (Sandler et al. 2018). We partitioned the result table into three parts: (1) The proposed dynamic networks; (2) Compared state-of-thee networks under mobile settings; (3) The original networks corresponding to the implemented dynamic networks.
Table provides several valuable observations: (1) Compared with these well known models under mobile setting, the proposed Dy-mobile and Dy-shuffle achieves the best classification error with lowest computation cost. This demonstrates that the proposed dynamic convolution is a simple yet effective way to reduce computation cost. (2) Compared with the corresponding basic neural structures, the proposed Dy-shuffle (1.0), Dy-mobile (1.0). Dy-ResNet18 and Dy-ResNet500 reduce 37.0%, 54.7%, 67.2% and 71.3% computation cost respectively with little drop on Top-1 accuracy. This shows that even though the proposed network significantly reduces the convolution computation cost, the generated dynamic kernel can still capture sufficient information from image contents. The results also indicate that the proposed dynamic convolution is powerful plugin, which can be implemented on convolution layers to reduce computation cost while maintaining the accuracy.
Furthermore, we conduct detailed experiments on MobileNetV2. We replace the conventional convolution with the proposed dynamic one and get Dy-MohileNerV2. The accuracy of classification for models with different number of channels are shown in Figure 5 It is observed that Dy- MobileNetV2 consistently outperforms MobileNetV2 but the ascendancy is weaken with the increase of number of channels.
-0.3
-0.2
-0.1
0.1
0.2
0.3
0.5
0.6
0.7
0.8
0.9
Correlation value
MobileNetv2
Dy- Mo-MobileeNee22
Figure 6: The correlation distribution of fixed kernel and the generated dynamic kernel, S, M, W. denote strong, middle, weak and no correlation respectively. We can observe that compared with conventional fixed kernels, the generated dynamic kernels have small correlation values.
Analysis of the dynamic kernel Aside from the quantitative analysis, we also demonstrate the redundancy of the generated dynamic kernels compared with conventional fixed kernels in Figure 6 We calculate the correlation between each feature maps output by the second last stage for the original MobileNetV211.cm .0) and ByyMooileNeet22. (1.0). Note that Dy-MobileNetV2 (1.0) is different with Dy-mobile(1.00 .0. Dy-MobileNerv2(1.. keeps the channels of each layer the same as the original one, while replace the conventional convolution with dynamic convolution. As shown in Figure 6 we can observe that the correlation distribution of dynamic kernels have more values distributed between - and 0.2 compared with fixed convolution kernel, which indicates that the redundancy between dynamic convolution kernels much smaller than the fixed convolution kernels.
7

---

Under review as a conference paper at ICLR 2020
59 59.00%
250
57.00%
Latency Reduced Ratio
55.00%
Latency (ms))
200
53.00%
150
51.00%
49.00%
100
47.00%
50
45.00%
224
256
288
320
352
Input Size
MobileNerV2((10)
-Dy-mobile(1.0)
Latency Reduced Ratio
Figure 7: Latency for different input size. we denote the latency of MoiileNerV2(1....)yymmooii Liiz and Dymm then Latency Reduced Ratio is defined 100%
LDym
I,iis
Analysis of speed the hardware We also analysis the inference speed of DyNet. We carry experiments on the CPU platform (Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz) with Caffe (Jia jtt 2014). We set the size of input as 224 and report the average inference time of 50 iterations. It is reasonable to set mini-batch size as I, which is consistent with most inference scenarios. The results are shown in T Moreover, the latency fusing fixed kernels is independent with the input size, thus expect achieve bigger acceleration ratio when the input size of networks become larger. We conduct experiments to verify this assumption, the results are shown in Figure 7 We can observe that the ratio of reduced latency achieved by DyNet gets bigger the input size becomes larger. As shown in Tan & Le 2019), a larger input size can make networks perform significantly better, thus DyNet is more effective in this scenario.
We also analysis the training speed on the GPU platform. The model is trained with 32 NVIDIA Tesla V100 GPUs and the batch size is 2048. We report the average training time of one iteration in Table 2 It is observed that the training speed of DyNet slower, is reasonable because we fuse feature maps rather than kernels according Eq. in the training stage.
Table 2: Speed on the hardware.
Methods
Top-1 I err. (%)
Inference Time
Training Time
Dy-mobile(1.0)
28.27
58.3ms
250ms
MobileNerV2((.0)
28.00
109.1ms
173ms
Dy- Re
31.01
68.7ms
213ms
ResNet18
30.41
90.7ms
170ms
Dy-ResNethe ResNet50
23.75
135.1ms
510ms
23.67
199.6ms
308ms
4.4 EXPERIMENTS ON SEGMENTATION
To verify the scalability of DyNet on other tasks, we conduct experiments on segmentation. Compared to the method Dilated FCN with ResNet50 as basenet (Fu et al. 2018), Dilated FCN with Dy- ResNet50 reduces 69.3% FLOPs while maintaining the MloU on Cityscapes validation set. The result shown in Table 3
8

---

Under review as a conference paper at ICLR 2020
Table 3: Experiments of segmentation on Cityscapes val set.
Methods
BaseNet
GFLOPs
Mean IoU%
Dilated FCN al.
2018
ResNet50 Dy-ResNet50
310.8
70.03
Dilated FCN Fu et al.
2018
95.6
70.48
4.5 ABLATION STUDY
Comparison between dynamic convolution and static convolution We correspondingly design two networks without dynamic convolution. Specifically, we remove the correlation prediction module and use fixed convolution kernel for Dy-mobile (1.0) and Dy-shuffle (1.5), and keep the channel number the same as the dynamic convolution neural networks. We denote the baseline networks Fix-mobile(1.0) .0) and Fix-shuffle (1.5) respectively. The results shown in Table 4 compare with baseline networks Fix-mobile (1.0) and Fix-shuffle (1.5), the proposed Dy-mobile (1.0) and Dy-shuffle (1.5) achieve absolute classification improvements by 5.19% and 2.82% respectively, This shows that directly decreasing the channel number to reduce computation cost influences the classification performance lot. While the proposed dynamic kernel can retain the representation ability as mush as possible.
Table 4: Ablation experiments results of dynamic convolution and fixed convolution.
Methods
MParams
MFLOPs
Top- emr (%)
Dy-mobile (1.0)
7.36
135
28.27
Dy-shuffle (1.5)
11.0
180
27.48
Fix-mobile (1.0)
2.16
129
33.57
Fix-shuffle (1.5)
2.47
171
30.30
Table 5: Ablation experiments on 9t-
Methods
MParams
MFLOPs
Top- I err. (%)
Fix-mobile(1.00 .0)
2.16
129
33.57
Dy-mobile(1.. g: 2)
3.58
131
29.43
Dy-mobile(I.0, 00, 9: = 4)
5.47
133
28.69
Dy-mobile(1.0) 9: 6)
7.36
135
28.27
Effectiveness of gi for dynamic kernel The group size ge in Eq. does not change the computation cost of dynamic convolution, but affects the performance of network. Thus we provide ablative study on 9t. We set ge as 2.4.6 for dy-mobile(1.0) respectively and the results are shown Table The performance of dy-mobile(1.0)) 1.0) becomes better when gets larger. It is reasonable because larger 9r means the number of kernels cooperated for obtaining one noise-irrelevant feature becomes larger. When 9: = 1, the coefficient prediction module can be regarded as merely learning the attention for different channels, which can improve the performance of networks as well (Hu et al. 2018). Therefore provide ablative study for comparing Si 1 and 9e 6 on Dy-mobile(L.0) and Dy-ResNet18. The results are shown in Table From the table we can see that, setting Sr. = will reduce the Top-1 accuracy ImageNet for Dy-mobile(1.0) (1.0) and Dy-ResNer18 by 2.58% and 2.79% respectively. It proves that the improvement of our proposed dynamic networks does not only come from the attention mechanism.

---

Under review as a conference paper at ICLR 2020
Table 6: Comparison for Dt.
and 9t 6.
Methods
MParams
MFLOPs
Top-1 err. (%)
Dy-mobile (1.0, 9t
2.64
131
30.85
Do-mobile (1.0, St = 6)
7.36
135
28.27
Dy-ResNet18 (g:- = 1)
3,04
553
33.8
Dy-ResNet18 (g: 16.6
567
31.01
CONCLUSION
In this paper, we propose a DyNet method to adaptively generate convolution kernels based on image content, which reduces the redundant computation cost existed in conventional fixed convolution kernels. Based on the proposed DyNet, we design several dynamic convolution neural networks based on well known architectures. i... Dy-mobile, Dy-shuffle, Dy-ResNet18, Dy-ResNet50. The experiment results show that DyNet reduces 37.02 .7.0 54.7%, 67.2% and 71.3% FLOPs respectively, while maintaining the performance unchanged. As future work, we want to further explore the redundancy phenomenon existed in convolution kernels, and find other ways to reduce computation cost, such as dynamically aggregate different kernels for different images other than fixed groups used in this paper.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pp. 2654-2662, 2014.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference on Machine Learning, pp. 2285-2294 2015.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Jun Fu, Jing Liu, Haijie Tian, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. 2018.
Jingjing Gong. Xipeng Qiu, Xinchi Chen, Dong Liang, and Xuanjing Huang. Convolutional interac- tion network for natural language inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1576-1585, 2018
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 201 5a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Kaiming He and Jian Sun. Convolutional neural networks constrained time cost. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5353-5360, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.025.: 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 17040017,, 2017.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatle networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint 17Xiv:160..07360, 2016.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 2014.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. In Advances in Neural Information Processing Systems, pp. 667-675, 2016.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv: 1408.5093, 2014.
Benjamin Klein, Lior Wolf, and Yehuda Afek. A dynamic convolutional layer for short range weather prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4840 4848, 2015.
11

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint 141.6554 2014.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang- Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. arXiv preprint 10Xhv:1903.10258, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 1rXiv:1412.65500 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Dinghan Shen, Martin Renqiang Min, Yitong Li, and Lawrence Carin. Learning context-sensitive convolutional filters for text processing. In Proceedingy of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1839-1848, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1555 2014.
Ke Sun, Mingjie Li, Dong Liu, and Jingdong Wang. Igev3: Interleaved low-rank group convolutions for efficient deep neural networks. arXiv preprint 1806. 2018.
Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for object detection. In Advances in neural information processing systems, pp. 2553-2561, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint
2019.
Guotian Xie, Jingdong Wang, Ting Zhang, Jianhuang Lai, Richang Hong, and Guo-Jun Qi. Interleaved structured sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8847-8856, 2018
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Soft conditional computation. arXiv preprint 1904::9900044711, 2019.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6856, 2018
12

---

Under review as a conference paper at ICLR 2020
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng- Lin Liu. Practical block-wit wise neural network architecture generation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2423 2232, 2018a.
Zhao Zhong, Zichen Yang, Boyang Deng, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Block- qnn: Efficient block-wise neural network architecture generation. arXiv preprint 1808: 2018b.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DETAILED ANALYSIS OF OUR MOTIVATION
We illustrate our motivation from a convolution with output f(z), i..,
f(z) = x Ø
(4)
where @ denotes the convolutional operator, x E R a vectorized input and w E R means the filter. Specifically, the den element of the convolution output f()) is calculated as:
fi(x)) (x(()) w))
(5)
where ( provides an inner product and [()) is the circular shift of by elements. We define the index i started from O.
We denote the noises ((()) as Σj= Orylyy where aj E and {yo, y/, Da-1) are the base vectors of noise space V. Then the kernels in one convolutional layer can be represented as {wo, W1. We)- } is Ω. We prove if the kernels trained until v S then for each Wi & V we can get the noise-irrelevant fi(wehhtt)) white wx) by the cooperation of other kernels wig: W1.....
The space expanded by (Wo, Wiy..., w
Firstly Z(()) can be decomposed as:
I(A) ((i)) Bwk
Agyli:
(6)
where B R and R" vertical to wi and Vj- For concision we assume the norm of wg and yj Then,
d-1
f((x) = (1)) wa) = (T(() + Bwn +
Σ 1=0
0g wx) = B(wn, wx) +
Σ
ay wk)
(7)
When there is no noise, i.e. aj = 0 for j = 0, 1,.., .... d 1. the white output (iiite
becomes:
fxwwhett (I(()) (xiihitee, wa) (z + B.uk, wx) B(wk, wk) B.
It is proved in the Appendix A.2 that:
fhitht (aog + Σ Bears (()) (aco + (a))(wk a(a) + Σ (w),
t
where fo.... Br is determined by the input image.
Eq. is fulfilled by linearly combine convolution output (we, .r()) and (we, (()) ) for those 3g * 0 in the following layers. Thus if there are N coefficients in Eq. that not then need to carry N times convolution operation to get the noise-irrelevant output of kernel wa this causes redundant calculation.
In Eq. 9 we can observe that the computation cost can be reduced to one convolution operation by linearly fusing those kernels to dynamic one:
(ago BR) +
Σ
Baw,
(10)
fi(wwhite) 21)))
In Eq. 10, the coefficients 8o, 30,31,.. 31, determined by co. Oo. thus they should be generated based on the input of network. This is the motivation of our proposed dynamic convolution.
A.2 PROVING OF EQ.
We denote (1) as (I(1)) U.). O, I,.,, d 1. Then,
d-1
g:5(2) (x) = (E()) mi) = (Z() + Bwr +
aelts U1) = B(wn, us) +
Σ 1=0
au (ve, yi).
(11)
0=0
14

---

Under review as a conference paper at ICLR 2020
By summarize Eq. 7 and Eq.
11
get the following equation:
(wk: wi)
(yo, wx)
(y1, wx) ... (ya-1, wkk
fi(x)
(wkk yo? (WEs y1)
(yo, Mo!
(31 you ... (yd-1. yo!
ou
Ie g
(yo, yn)
(un, 31
1;
Da (z)
(12)
:
(wk, Ma-1) (Wo, yd-1) We simplify this equation as:
(ya-1, (d
Ld-11
(i(d-1)(x))
AF b.
(13)
Because w/ $ V. we can denote wi as:
d-1
Wi 71
W1
+
(14)
1=0
where w1 is vertical to yo...., Dd-1 and 7r + O. moreover because |rrg| ,ttus
d-1
hr12 2 +
Σ 5=0
1781 1.
(15)
can be easily proved that:
1
70
71
.. 7a
70
1
0
71
0
1
(16)
Ldd-1
thus,
1
70
71
Nd-1
70
1
0
|A|
71
0
1
:
Nd-1
0
70
71
7d-1
7O
0
71
7a-1
70
7
0
0
7o
0
71
0
1
(17)
7d-1
70
0
0
0
7d-1
77
0
70
71
0
...
1
Nd-1
7 # 0.
15

---

Under review as a conference paper at ICLR 2020
thus,
f= A 10.
(18)
If we denote the elements of the first row of A
as aoo. Ao1, .... aod. then
fi(xwhite) = = a00fi(2)) +
00(+1)91,(2)
==
d
= (wkk I())
+
aoi++1) (1i I(1))
(19)
j=0
= (ago
1) z(1)
1=0
Because V 22 there exists {Be Rlt
1,.., c) that
00(+1)33 Σ B,w.
(20)
Then,
whhtte) (ao0 Σ Bewr, (())) =(ao0 + (uk v(o) + Σ
(21)
tf
16


Reviews:
[{"review_id": "r1gRUl0TYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nMain contribution of the paper\n- The paper proposes a dynamic convolution selection method can be applied to arbitrary classification networks based on the global average pooled (GAP) feature map info.\n- The method obtained improvements over various networks (SuffleNet v2, MobileNet v2, ResNet 18) on ImageNet.\n\nMethods\n- Given the set of fixed convolutional filters, the method dynamically selects the (weighted sum) kernels by given a kind of channel attention.\n- The GAP of the features gives the channel attention on each stage, and the method applies the dynamic selection of the kernels.\n- The number of channels in skip-connection shluld be the same because it should be elementwise multiplied with the channel attention acquired from GAP.\n- The author slightly revises the baseline networks to set the networks integrated with the proposed method to have smaller Flops.\n\nQuestions\n- According to Figure 4, it seems that the proposed add-on requires many parameters because it would include a FC layer for each block. But we cannot find the number of parameters in this paper.\n- The parameter $g_t$ is defined as 6. The experiment shows the ablation to the case $g_t$ =1, but what if we set the parameters to other numbers?\n\nStrong points\n- The proposed model achieved improvement with fewer Flops on large scale image classification dataset.\n- The method shows effectiveness when it is attached to various classification networks.\n\nConcerns\n- The main concern of the reviewer is that the model shares the core contribution to the existing method; squeeze-and-excitation network (SEnet, Hu et.al.). The method also proposes the attention-based scaling of channels, where the attention comes from GAP, so the reviewer thinks that it is possible to explain this work as some variation of SEnet.\nThe author should clarify the difference and the strong points of the proposed block compared to SEnet.\n- Also, the reviewer cannot guarantee that the networks trained by the proposed method can transfer the knowledge to other tasks such as detection. \nThe reviewer thinks that it is a critical part because one of the primal reasons for training the network is to use them as the pre-trained backbone for the other tasks.\nRegarding this, the baseline methods (MobileNet V2, Shufflenet v2, ResNet)  are widely used as a pre-trained backbone for object detection, and the papers mention the CoCo object detection results using the pre-trained backbones from their method. The reviewer thinks that the experiment regarding this should be included.\n- The other thing is that the parameter increases. As in the question, the reviewer thinks that the number of parameters would be increased. The reviewer agrees that some recent works focus more on Flops, but the number of parameters is also discussed in general, when telling about the 'model size'.\n\nConclusion\n- The author proposed a dynamic kernel selection method (add-on), which can enhance the classification accuracy of the baseline network. \n- However, the reviewer cannot convince the novelty of the proposed approach and usefulness of the pre-trained backbone network from the proposed method when applying it to the other tasks (Object detection).\n\n\nInquiries\n- Clarifying the difference between SEnet.\n- Testing the ImageNet trained network of the proposed method into an object detection task (as the pre-trained backbone).\n- Discussing the number of the parameter as well.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxn_9yy9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed dynamic convolution (DyNet) to accelerating convolution networks. The new method is tested on the ImageNet dataset with three different backbones. It reduces the computation flops by a large margin while keeps similar classification accuracy. The additional segmentation experiment on the Cityscapes dataset also shows the new module can save computation a lot while maintaining similar segmentation accuracy.\n\nClarity:\nThe novelty of the paper is limited and the experimental results are weird for me.\n1. The proposed module named dynamic convolution is detailed in Sec 3.2. As far as I can see, it is very similar to the former SENet especially in Figure (3) and Equation (2). The only difference is the introduction of g_t where the output dimension is much larger than SENet.\n\n2. As shown in Equation (2), the proposed method contains the normal computation of fixed kernels. How can this method save computations compared to classical convolution? Is the computation flops calculated in the right way?\n\n3.  The results in Table 5 are strange to me. Larger g_t will increase the flops absolutely according to Equation (2).\n\n4. The author may need to show the comparisons of the number of parameters. In my opinion, the new module will increase the parameters a lot (the output dimension of the fully connected layer is as large as C_cout*g_t).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lC6hMCYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ===\nThe authors propose to use dynamic convolutional kernels as a means to reduce the computation cost in static CNNs while maintaining their performance. The dynamic kernels are obtained by a linear combination of static kernels where the weights of the linear combination are input-dependent (they are obtained similarly to the coefficients in squeeze-and-excite). The authors also include a theoretical and experimental study of the correlation.\nThe authors conduct extensive experiments on image classification and segmentation and show that dynamic convolutional kernels with reduced number of channels lead to significant reduction in FLOPS and increase in inference speed (for batch size 1) compared to their static counterparts with higher number of channels.\n\n=== Recommendation ===\nThe experimental setup is rigorous but the current draft lacks some metrics that should be reported (as training times, parameter counts, memory requirements at training/inference) since the focus is on making CNNs more efficient.\n\nThe presented experimental results are satisfactory but the studied networks are not quite SOTA: they are much more competitive alternatives to ResNet and MobileNetv2. The correlation study is interesting.\n\nMy main issue with the paper is the lack of novelty. The use of dynamic convolutions is by no means a novel idea and has been studied in multiple previous works in vision (mixture of experts, soft conditional computation, pay less attention with dynamic convolutions, ...) which the authors fail to cite/compare against.\nHowever, most previous work focuses on leveraging dynamic kernels to use more parameters so the focus on accelerating CNNs is novel.\n\nOverall, I am on the fence with this paper but slightly leaning towards rejecting it for the above reasons.\n\n=== Questions/Comments ===\n- Figure 5: how are the models constrained to have same FLOPS? Is it by changing the number of channels?\n- Consider adding training times for more transparency\n- Consider adding parameter counts in experiment tables\n- The related work subsection 2.3 is rather poor compared to existing work.\n- 'While model compression based methods' -> 'On the other hand, model compression based methods'\n- 'computing efficient' -> 'compute efficient'\n- 'values distribute' -> 'values distributed'\n- 'DETAIL ANALYSIS OF OUR MOTIVATION' -> 'Detailed analysis of our motivation'", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed the use of dynamic convolutional kernels as a way to reduce inference computation cost, which is a linear combination of static kernels and fused after training for inference to reduce computation cost. The authors evaluated the proposed methods on a variety models and shown good FLOPS reduction while maintaining accuracy. 

The main concern for this paper is the limited novelty. There have been many works use dynamic convolutions as pointed out by all the reviewers. The most similar ones are SENet and soft conditional computation. Although the authors claim that soft conditional computation "focus on using more parameters to make models to be more expressive while we focus on reducing redundant calculations", the methods are pretty the same and moreover in the abstract of soft conditional computation they have "CondConv improves the performance and inference cost trade-off".
Decision: Reject