Year: 2019

Abstract:
Convolution operator is the core of convolutional neural networks (CNNs) and occupies the most computation cost. To make CNNs more efficient, many methods have been proposed to either design lightweight networks or compress models. Although some efficient network structures have been proposed, such as MobileNet or ShuffleNet, we find that there still exists redundant information between convolution kernels. To address this issue, we propose a novel dynamic convolution method named DyNet in this paper, which can adaptively generate convolution kernels based on image contents. To demonstrate the effectiveness, we apply DyNet on multiple state-of-the-art CNNs. The experiment results show that DyNet can reduce the computation cost remarkably, while maintaining the performance nearly unchanged. Specifically, for ShuffleNetV2 (1.0), MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 37.0%, 54.7%, 67.2% and 71.3% FLOPs respectively while the Top-1 accuracy on ImageNet only changes by +1.0%, −0.27%, −0.6% and −0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87×, 1.32× and 1.48× on CPU platform respectively. To verify the scalability, we also apply DyNet on segmentation task, the results show that DyNet can reduces 69.3% FLOPs while maintaining the Mean IoU on segmentation task.

heading: 1 INTRODUCTION
text: Convolutional neural networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), and the neural architectures of CNNs are evolving over the years (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Hu et al., 2018; Zhong et al., 2018a;b). However, modern high-performance CNNs often require a lot of computation resources to execute large amount of convolution kernel operations. Aside from the accuracy, to make CNNs applicable on mobile devices, building lightweight and efficient deep models has attracting much more attention recently (Howard et al., 2017; Sandler et al., 2018; Zhang et al., 2018; Ma et al., 2018). These methods can be roughly categorized into two types: efficient network design and model compression. Representative methods for the former category are MobileNet (Howard et al., 2017; Sandler et al., 2018) and ShuffleNet (Ma et al., 2018; Zhang et al., 2018), which use depth-wise separable convolution and channel-level shuffle techniques to reduce computation cost. On the other hand, model compression based methods tend to obtain a smaller network by compressing a larger network via pruning, factorization or mimic (Chen et al., 2015; Han et al., 2015a; Jaderberg et al., 2014; Lebedev et al., 2014; Ba & Caruana, 2014).
Although some handcrafted efficient network structures have been designed, we observe that the significant correlations still exist among convolutional kernels, and introduce large amount of redundant calculations. Moreover, these small networks are hard to compress. For example, Liu et al. (2019) compress MobileNetV2 to 124M, but the accuracy drops by 5.4% on ImageNet. We theoretically analyze above observation, and find that this phenomenon is caused by the nature of static convolution, where correlated kernels are cooperated to extract noise-irrelevant features. Thus it is hard to compress the fixed convolution kernels without information loss. We also find that if we linearly fuse several convolution kernels to generate one dynamic kernel based on the input, we can obtain the noise-irrelevant features without the cooperation of multiple kernels, and further reduce the computation cost of convolution layer remarkably.
Based on above observations and analysis, in this paper, we propose a novel dynamic convolution method named DyNet. The overall framework of DyNet is shown in Figure 1, which consists of a coefficient prediction module and a dynamic generation module. The coefficient prediction module is trainable and designed to predict the coefficients of fixed convolution kernels. Then the dynamic generation module further generates a dynamic kernel based on the predicted coefficients.
Our proposed dynamic convolution method is simple to implement, and can be used as a drop-in plugin for any convolution layer to reduce computation cost. We evaluate the proposed DyNet on state-of-the-art networks such as MobileNetV2, ShuffleNetV2 and ResNets. Experiment results show that DyNet reduces 37.0% FLOPs of ShuffleNetV2 (1.0) while further improve the Top-1 accuracy on ImageNet by 1.0%. For MobileNetV2 (1.0), ResNet18 and ResNet50, DyNet reduces 54.7%, 67.2% and 71.3% FLOPs respectively, the Top-1 accuracy on ImageNet changes by −0.27%, −0.6% and −0.08%. Meanwhile, DyNet further accelerates the inference speed of MobileNetV2 (1.0), ResNet18 and ResNet50 by 1.87×,1.32×and 1.48× on CPU platform respectively.

heading: 2 RELATED WORK
text: We review related works from three aspects: efficient convolution neural network design, model compression and dynamic convolutional kernels.

heading: 2.1 EFFICIENT CONVOLUTION NEURAL NETWORK DESIGN
text: In many computer vision tasks (Krizhevsky et al., 2012; Szegedy et al., 2013), model design plays a key role. The increasing demands of high quality networks on mobile/embedding devices have driven the study on efficient network design (He & Sun, 2015). For example, GoogleNet (Szegedy et al., 2015) increases the depth of networks with lower complexity compared to simply stacking convolution layers; SqueezeNet (Iandola et al., 2016) deploys a bottleneck approach to design a very small network; Xception (Chollet, 2017), MobileNet (Howard et al., 2017) and MobileNetV2 (Sandler et al., 2018) use depth-wise separable convolution to reduce computation and model size. ShuffleNet (Zhang et al., 2018) and ShuffleNetV2 (Ma et al., 2018) shuffle channels to reduce computation of 1× 1 convolution kernel and improve accuracy. Despite the progress made by these efforts, we find that there still exists redundancy between convolution kernels and cause redundant computation.

heading: 2.2 MODEL COMPRESSION
text: Another trend to obtaining small network is model compression. Factorization based methods (Jaderberg et al., 2014; Lebedev et al., 2014) try to speed up convolution operation by using tensor decomposition to approximate original convolution operation. Knowledge distillation based methods (Ba & Caruana, 2014; Romero et al., 2014; Hinton et al., 2015) learn a small network to mimic a larger teacher network. Pruning based methods (Han et al., 2015a;b; Wen et al., 2016; Liu et al., 2019)
try to reduce computation by pruning the redundant connections or convolution channels. Compared with those methods, DyNet is more effective especially when the target network is already efficient enough. For example, in (Liu et al., 2019), they get a smaller model of 124M FLOPs by pruning the MobileNetV2, however it drops the accuracy by 5.4% on ImageNet compared with the model with 291M FLOPs. While in DyNet, we can reduce the FLOPs of MobileNetV2 (1.0) from 298M to 129M with the accuracy drops only 0.27%.

heading: 2.3 DYNAMIC CONVOLUTION KERNEL
text: Generating dynamic convolution kernels appears in both computer vision and natural language processing (NLP) tasks.
In computer vision domain, Klein et al. (Klein et al., 2015) and Brabandere et al. (Jia et al., 2016) directly generate convolution kernels via a linear layer based on the feature maps of previous layers. Because convolution kernels has a large amount of parameters, the linear layer will be inefficient on the hardware. Our proposed method solves this problem via merely predicting the coefficients for linearly combining static kernels and achieve real speed up for CNN on hardware. The idea of linearly combining static kernels using predicted coefficients has been proposed by Yang et al. (Yang et al., 2019), but they focus on using more parameters to make models more expressive while we focus on reducing redundant calculations in convolution. We make theoretical analysis and conduct correlation experiment to prove that correlations among convolutional kernels can be reduced via dynamically fusing several kernels.
In NLP domain, some works (Shen et al., 2018; Wu et al., 2019; Gong et al., 2018) incorporate context information to generate input-aware convolution filters which can be changed according to input sentences with various lengths. These methods also directly generate convolution kernels via a linear layer, etc. Because the size of CNN in NLP is smaller and the dimension of convolution kernel is one, the inefficiency issue for the linear layer is alleviated. Moreover, Wu et al. (Wu et al., 2019) alleviate this issue utilizing the depthwise convolution and the strategy of sharing weight across layers. These methods are designed to improve the adaptivity and flexibility of language modeling, while our method aims to cut down the redundant computation cost.

heading: 3 DYNET: DYNAMIC CONVOLUTION IN CNNS
text: In this section, we first describe the motivation of DyNet. Then we explain the proposed dynamic convolution in detail. Finally, we illustrate the DyNet based architectures of our proposed Dy-mobile, Dy-shuffle, Dy-ResNet18, Dy-ResNet50.

heading: 3.1 MOTIVATION
text: As illustrated in previous works (Han et al., 2015a;b; Wen et al., 2016; Liu et al., 2019), convolutional kernels are naturally correlated in deep models. For some of the well known networks, we plot the distribution of Pearson product-moment correlation coefficient between feature maps in Figure 2. Most existing works try to reduce correlations by compressing. However, efficient and small networks like MobileNets are harder to prune despite the correlation is still significant. We think these correlations are vital for maintaining the performance because they are cooperated to obtain noiseirrelevant features. We take face recognition as an example, where the pose or the illumination is not supposed to change the classification results. Therefore, the feature maps will gradually become noise-irrelevant when they go deeper. Based on the theoretical analysis in appendix A, we find that if we dynamically fuse several kernels, we can get noise-irrelevant feature without the cooperation of redundant kernels. In this paper, we propose dynamic convolution method, which learns the coefficients to fuse multiple kernels into a dynamic one based on image contents. We give more in depth analysis about our motivation in appendix A.

heading: 3.2 DYNAMIC CONVOLUTION
text: The goal of dynamic convolution is to learn a group of kernel coefficients, which fuse multiple fixed kernels to a dynamic one. We demonstrate the overall framework of dynamic convolution in Figure 1. We first utilize a trainable coefficient prediction module to predict coefficients. Then we further propose a dynamic generation module to fuse fixed kernels to a dynamic one. We will illustrate the coefficient prediction module and dynamic generation module in detail in the following of this section.
Coefficient prediction module Coefficient prediction module is proposed to predict coefficients based on image contents. As shown in Figure 3, the coefficient prediction module can be composed by a global average pooling layer and a fully connected layer with Sigmoid as activation function. Global average pooling layer aggregates the input feature maps into a 1 × 1 × Cin vector, which serves as a feature extraction layer. Then the fully connected layer further maps the feature into a 1× 1×C vector, which are the coefficients for fixed convolution kernels of several dynamic convolution layers.
Dynamic generation module For a dynamic convolution layer with weight [Cout × gt, Cin, k, k], it corresponds with Cout × gt fixed kernels and Cout dynamic kernels, the shape of each kernel is [Cin, k, k]. gt denotes the group size, it is a hyperparameter. We denote the fixed kernels as wit, the dynamic kernels as w̃t, the coefficients as η i t, where t = 0, ..., Cout, i = 0, ..., gt.
After the coefficients are obtained, we generate dynamic kernels as follows:
w̃t = gt∑ i=1 ηit · wit (1)
Training algorithm For the training of the proposed dynamic convolution, it is not suitable to use batch based training scheme. It is because the convolution kernel is different for different input images in the same mini-batch. Therefore, we fuse feature maps based on the coefficients rather than kernels during training. They are mathematically equivalent as shown in Eq. 2:
Õt = w̃t ⊗ x = gt∑ i=1 (ηit · wit)⊗ x = gt∑ i=1 (ηit · wit ⊗ x)
= gt∑ i=1 (ηit · (wit ⊗ x)) = gt∑ i=1 (ηit ·Oit), (2)
where x denotes the input, Õt denotes the output of dynamic kernel w̃t, Oit denotes the output of fixed kernel wit.

heading: 3.3 DYNAMIC CONVOLUTION NEURAL NETWORKS
text: We equip the MobileNetV2, ShuffleNetV2 and ResNets with our proposed dynamic convolution, and propose Dy-mobile, Dy-shuffle, Dy-ResNet18 and Dy-ResNet50 respectively. The building blocks of these 4 network are shown in Figure 4. Based on above dynamic convolution, each dynamic kernel can get noise-irrelevant feature without the cooperation of other kernels. Therefore we can reduce the channels for each layer of those base models and remain the performance. We set the hyper-parameter gt as 6 for all of them, and we give details of these dynamic CNNs below.
Dy-mobile In our proposed Dy-mobile, we replace the original MobileNetV2 block with our dymobile block, which is shown in Figure 4 (a). The input of coefficient prediction module is the input of block, it produces the coefficients for all three dynamic convolution layers. Moreover, we further make two adjustments:
• We do not expand the channels in the middle layer like MobileNetV2. If we denote the output channels of the block as Cout, then the channels of all the three convolution layers will be Cout.
• Since the depth-wise convolution is efficient, we set groups = Cout6 for the dynamic depthwise convolution. We will enlarge Cout to make it becomes the multiple of 6 if needed.
After the aforementioned adjustments, the first dynamic convolution layer reduce the FLOPs from 6C2HW to C2HW . The second dynamic convolution layer keep the FLOPs as 6CHW × 32 unchanged because we reduce the output channels by 6x while setting the groups of convolution 6x smaller, too. For the third dynamic convolution layer, we reduce the FLOPs from 6C2HW to C2HW as well. The ratio of FLOPs for the original block and our dy-mobile block is:
6C2HW + 6CHW × 32 + 6C2HW C2HW + 6CHW × 32 + C2HW = 6C + 27 C + 27 = 6− 135 C + 27 (3)
Dy-shuffle In the original ShuffleNet V2, channel split operation will split feature maps to rightbranch and left-branch, the right branch will go through one pointwise convolution, one depthwise convolution and one pointwise convolution sequentially. We replace conventional convolution with dynamic convolution in the right branch as shown in Figure 4 (b). We feed the input of right branch into coefficient prediction module to produce the coefficients. In our dy-shuffle block, we split channels into left-branch and right-branch with ratio 3 : 1, thus we reduce the 75% computation cost for two dynamic pointwise convolutuon. Similar with dy-mobile, we adjust the parameter ”groups” in dynamic depthwise convolution to keep the FLOPs unchanged.
Dy-ResNet18/50 In Dy-ResNet18 and DyResNet50, we simple reduce half of the output channels for dynamic convolution layers of each residual block. Because the input channels of each block is large compared with dy-mobile and dy-shuffle, we use two linear layer as shown in Figure 4 (c) and Figure 4 (d) to reduce the amount of parameters. If the input channel is Cin, the output channels of the first linear layer will be Cin4 for Dy-ResNet18/50.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS
text: For the training of the proposed dynamic neural networks. Each image has data augmentation of randomly cropping and flipping, and is optimized with SGD strategy with cosine learning rate decay. We set batch size, initial learning rate, weight decay and momentum as 2048, 0.8, 5e-5 and 0.9 respectively. We also use the label smoothing with rate 0.1. We evaluate the accuracy on the test images with center crop.
4.2 EXPERIMENT SETTINGS AND COMPARED METHODS
We evaluate DyNet on ImageNet (Russakovsky et al., 2015), which contains 1.28 million training images and 50K validation images collected from 1000 different classes. We train the proposed networks on the training set and report the top-1 error on the validation set. To demonstrate the effectiveness, we compare the proposed dynamic convolution with state-of-the-art networks under mobile setting, including MobileNetV1 (Howard et al., 2017), MobileNetV2 (Sandler et al., 2018), ShuffleNet (Zhang et al., 2018), ShuffleNet V2 (Ma et al., 2018), Xception (Chollet, 2017), DenseNet (Huang et al., 2017), IGCV2 (Xie et al., 2018) and IGCV3 (Sun et al., 2018).

heading: 4.3 EXPERIMENT RESULTS AND ANALYSIS
text: Analysis of accuracy and computation cost We demonstrate the results in Table 1, where the number in the brackets indicates the channel number controller (Sandler et al., 2018). We partitioned the result table into three parts: (1) The proposed dynamic networks; (2) Compared state-of-the-art networks under mobile settings; (3) The original networks corresponding to the implemented dynamic networks.
Table 1 provides several valuable observations: (1) Compared with these well known models under mobile setting, the proposed Dy-mobile and Dy-shuffle achieves the best classification error with lowest computation cost. This demonstrates that the proposed dynamic convolution is a simple yet effective way to reduce computation cost. (2) Compared with the corresponding basic neural structures, the proposed Dy-shuffle (1.0), Dy-mobile (1.0), Dy-ResNet18 and Dy-ResNet50 reduce 37.0%, 54.7%, 67.2% and 71.3% computation cost respectively with little drop on Top-1 accuracy. This shows that even though the proposed network significantly reduces the convolution computation cost, the generated dynamic kernel can still capture sufficient information from image contents. The results also indicate that the proposed dynamic convolution is a powerful plugin, which can be implemented on convolution layers to reduce computation cost while maintaining the accuracy.
Furthermore, we conduct detailed experiments on MobileNetV2. We replace the conventional convolution with the proposed dynamic one and get Dy-MobileNetV2. The accuracy of classification for models with different number of channels are shown in Figure 5. It is observed that DyMobileNetV2 consistently outperforms MobileNetV2 but the ascendancy is weaken with the increase of number of channels.
Correlation value -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
MobileNetV2 Dy-MobileNetV2
SW N W M
Figure 6: The correlation distribution of fixed kernel and the generated dynamic kernel, S, M, W, N denote strong, middle, weak and no correlation respectively. We can observe that compared with conventional fixed kernels, the generated dynamic kernels have small correlation values.
Analysis of the dynamic kernel Aside from the quantitative analysis, we also demonstrate the redundancy of the generated dynamic kernels compared with conventional fixed kernels in Figure 6. We calculate the correlation between each feature maps output by the second last stage for the original MobileNetV2(1.0) and Dy-MobileNetV2 (1.0). Note that Dy-MobileNetV2 (1.0) is different with Dy-mobile(1.0). Dy-MobileNetV2(1.0) keeps the channels of each layer the same as the original one, while replace the conventional convolution with dynamic convolution. As shown in Figure 6, we can observe that the correlation distribution of dynamic kernels have more values distributed between −0.1 and 0.2 compared with fixed convolution kernel, which indicates that the redundancy between dynamic convolution kernels are much smaller than the fixed convolution kernels.
Analysis of speed on the hardware We also analysis the inference speed of DyNet. We carry out experiments on the CPU platform (Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz) with Caffe (Jia et al., 2014). We set the size of input as 224 and report the average inference time of 50 iterations. It is reasonable to set mini-batch size as 1, which is consistent with most inference scenarios. The results are shown in Table 2. Moreover, the latency of fusing fixed kernels is independent with the input size, thus we expect to achieve bigger acceleration ratio when the input size of networks become larger. We conduct experiments to verify this assumption, the results are shown in Figure 7. We can observe that the ratio of reduced latency achieved by DyNet gets bigger as the input size becomes larger. As shown in (Tan & Le, 2019), a larger input size can make networks perform significantly better, thus DyNet is more effective in this scenario.
We also analysis the training speed on the GPU platform. The model is trained with 32 NVIDIA Tesla V100 GPUs and the batch size is 2048. We report the average training time of one iteration in Table 2. It is observed that the training speed of DyNet is slower, it is reasonable because we fuse feature maps rather than kernels according to Eq. 2 in the training stage.

heading: 4.4 EXPERIMENTS ON SEGMENTATION
text: To verify the scalability of DyNet on other tasks, we conduct experiments on segmentation. Compared to the method Dilated FCN with ResNet50 as basenet (Fu et al., 2018), Dilated FCN with DyResNet50 reduces 69.3% FLOPs while maintaining the MIoU on Cityscapes validation set. The result are shown in Table 3.

heading: 4.5 ABLATION STUDY
text: Comparison between dynamic convolution and static convolution We correspondingly design two networks without dynamic convolution. Specifically, we remove the correlation prediction module and use fixed convolution kernel for Dy-mobile (1.0) and Dy-shuffle (1.5), and we keep the channel number the same as the dynamic convolution neural networks. We denote the baseline networks as Fix-mobile(1.0) and Fix-shuffle (1.5) respectively. The results are shown in Table 4, compare with baseline networks Fix-mobile (1.0) and Fix-shuffle (1.5), the proposed Dy-mobile (1.0) and Dy-shuffle (1.5) achieve absolute classification improvements by 5.19% and 2.82% respectively. This shows that directly decreasing the channel number to reduce computation cost influences the classification performance a lot. While the proposed dynamic kernel can retain the representation ability as mush as possible.
Effectiveness of gt for dynamic kernel The group size gt in Eq. 1 does not change the computation cost of dynamic convolution, but affects the performance of network. Thus we provide ablative study on gt. We set gt as 2,4,6 for dy-mobile(1.0) respectively and the results are shown in Table 5. The performance of dy-mobile(1.0) becomes better when gt gets larger. It is reasonable because larger gt means the number of kernels cooperated for obtaining one noise-irrelevant feature becomes larger. When gt = 1, the coefficient prediction module can be regarded as merely learning the attention for different channels, which can improve the performance of networks as well (Hu et al., 2018). Therefore we provide ablative study for comparing gt = 1 and gt = 6 on Dy-mobile(1.0) and Dy-ResNet18. The results are shown in Table 6. From the table we can see that, setting gt = 1 will reduce the Top-1 accuracy on ImageNet for Dy-mobile(1.0) and Dy-ResNet18 by 2.58% and 2.79% respectively. It proves that the improvement of our proposed dynamic networks does not only come from the attention mechanism.

heading: 5 CONCLUSION
text: In this paper, we propose a DyNet method to adaptively generate convolution kernels based on image content, which reduces the redundant computation cost existed in conventional fixed convolution kernels. Based on the proposed DyNet, we design several dynamic convolution neural networks based on well known architectures, i.e., Dy-mobile, Dy-shuffle, Dy-ResNet18, Dy-ResNet50. The experiment results show that DyNet reduces 37.0%, 54.7%, 67.2% and 71.3% FLOPs respectively, while maintaining the performance unchanged. As future work, we want to further explore the redundancy phenomenon existed in convolution kernels, and find other ways to reduce computation cost, such as dynamically aggregate different kernels for different images other than fixed groups used in this paper.

heading: A APPENDIX
text: A.1 DETAILED ANALYSIS OF OUR MOTIVATION
We illustrate our motivation from a convolution with output f(x), i.e.,
f(x) = x⊗ w, (4)
where ⊗ denotes the convolutional operator, x ∈ Rn is a vectorized input and w ∈ Rn means the filter. Specifically, the ith element of the convolution output f(x) is calculated as:
fi(x) = 〈x(i), w〉, (5)
where 〈·, ·〉 provides an inner product and x(i) is the circular shift of x by i elements. We define the index i started from 0. We denote the noises in x(i) as ∑d−1 j=0 αjyj , where αj ∈ R and {y0, y1, ..., yd−1} are the base vectors of noise space Ψ. Then the kernels in one convolutional layer can be represented as {w0, w1, ..., wc}. The space expanded by {w0, w1, ..., wc} is Ω. We can prove if the kernels are trained until Ψ ⊂ Ω, then for each wk /∈ Ψ, we can get the noise-irrelevant fi(xwhite) = 〈xwhite(i) , wk〉 by the cooperation of other kernels w0, w1, .... Firstly x(i) can be decomposed as:
x(i) = x̄(i) + βwk + d−1∑ j=0 αjyj , (6)
where β ∈ R and x̄ ∈ Rn is vertical to wk and yj . For concision we assume the norm of wk and yj is 1. Then,
fi(x) = 〈x(i), wk〉 = 〈x̄(i) + βwk + d−1∑ j=0 αjyj , wk〉 = β〈wk, wk〉+ d−1∑ j=0 αj〈yj , wk〉 (7)
When there is no noise, i.e. αj = 0 for j = 0, 1, ..., d− 1, the white output fi(xwhite) becomes:
fi(x white) = 〈xwhite(i) , wk〉 = 〈x̄(i) + βwk, wk〉 = β〈wk, wk〉 = β. (8)
It is proved in the Appendix A.2 that:
fi(x white) = 〈a00wk + ∑ t βtwt, x(i)〉 = (a00 + βk)〈wk, x(i)〉+ ∑ t 6=k βt〈wt, x(i)〉, (9)
where β0, ..., βc is determined by the input image. Eq. 9 is fulfilled by linearly combine convolution output 〈wk, x(i)〉 and 〈wt, x(i)〉 for those βt 6= 0 in the following layers. Thus if there are N coefficients in Eq. 9 that are not 0, then we need to carry out N times convolution operation to get the noise-irrelevant output of kernel wt, this causes redundant calculation. In Eq. 9, we can observe that the computation cost can be reduced to one convolution operation by linearly fusing those kernels to a dynamic one:
w̃ = (a00 + βk)wk + ∑
t 6=k,βt 6=0
βtwt
fi(x white) = 〈w̃, x(i)〉.
(10)
In Eq. 10, the coefficients β0, β1, ... is determined by α0, α1, ..., thus they should be generated based on the input of network. This is the motivation of our proposed dynamic convolution.
A.2 PROVING OF EQ. 9
We denote gij(x) as 〈x(i), yj〉, j = 0, 1, ..., d− 1. Then,
gij(x) = 〈x(i), yj〉 = 〈x̄(i) + βwk + d−1∑ t=0 αtyt, yj〉 = β〈wk, yj〉+ d−1∑ t=0 αt〈yt, yj〉. (11)
By summarize Eq. 7 and Eq. 11, we get the following equation: 〈wk, wk〉 〈y0, wk〉 〈y1, wk〉... 〈yd−1, wk〉 〈wk, y0〉 〈y0, y0〉 〈y1, y0〉 ... 〈yd−1, y0〉 〈wk, y1〉 〈y0, y1〉 〈y1, y1〉 ... 〈yd−1, y1〉
... ...
... ... ...
〈wk, yd−1〉〈y0, yd−1〉 . . . ...〈yd−1, yd−1〉
  β α0 α1 ...
αd−1
 =  fi(x) gi0(x) gi1(x)
... gi(d−1)(x)  , (12) We simplify this equation as:
A~x = ~b. (13) Because wk /∈ Ψ, we can denote wk as:
wk = γ⊥w⊥ + d−1∑ j=0 γjyj , (14)
where w⊥ is vertical to y0, ..., yd−1 and γ⊥ 6= 0. moreover because |wk| = 1 ,thus
|γ⊥|2 + d−1∑ j=0 |γj |2 = 1. (15)
It can be easily proved that:
A =  1 γ0 γ1 ...γd−1 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1  . (16) thus,
|A| = ∣∣∣∣∣∣∣∣∣∣ 1 γ0 γ1 ... γd−1 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1 ∣∣∣∣∣∣∣∣∣∣ =
∣∣∣∣∣∣∣∣∣∣ 1− γ20 0 γ1 ... γd−1 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1 ∣∣∣∣∣∣∣∣∣∣ =
∣∣∣∣∣∣∣∣∣∣ 1− γ20 − γ21 0 0 ... γd−1 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1 ∣∣∣∣∣∣∣∣∣∣ =
∣∣∣∣∣∣∣∣∣∣ 1− γ20 − γ21 − ...− γ2d−1 0 0 ... 0 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1 ∣∣∣∣∣∣∣∣∣∣ =
∣∣∣∣∣∣∣∣∣∣ γ2⊥ 0 0 ... 0 γ0 1 0 ... 0 γ1 0 1 ... 0 ... ... ... ...
... γd−1 0 . . . ... 1 ∣∣∣∣∣∣∣∣∣∣ =γ2⊥ 6= 0.
(17)
thus, ~x = A−1~b. (18)
If we denote the elements of the first row of A−1 as a00, a01, ..., a0d, then
fi(x white) = β = a00fi(x) + d−1∑ j=0 a0(j+1)gi,j(x)
= a00〈wk, x(i)〉+ d−1∑ j=0 a0(j+1)〈yj , x(i)〉 = 〈a00wk + d−1∑ j=0 a0(j+1)yj , x(i)〉.
(19)
Because Ψ ⊂ Ω, there exists {βt ∈ R|t = 0, 1, ..., c} that d−1∑ j=0 a0(j+1)yj = ∑ t βtwt. (20)
Then,
fi(x white) = 〈a00wk + ∑ t βtwt, x(i)〉 = (a00 + βk)〈wk, x(i)〉+ ∑ t 6=k βt〈wt, x(i)〉, (21)

Review summaries:
The authors propose to use dynamic convolutional kernels as a means to reduce the computation cost in static CNNs while maintaining their performance. The dynamic kernels are obtained by a linear combination of static kernels where the weights of the linear combination are input-dependent (they are obtained similarly to the coefficients in squeeze-and-excite). The authors also include a theoretical and experimental study of the correlation.

This paper proposed dynamic convolution (DyNet) to accelerating convolution networks. The new method is tested on the ImageNet dataset with three different backbones. It reduces the computation flops by a large margin while keeps similar classification accuracy. The additional segmentation experiment on the Cityscapes dataset also shows the new module can save computation a lot while maintaining similar segmentation accuracy.