Year: 2021

Abstract:
Learning a single static convolutional kernel 1 in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning a linear combination of n convolutional kernels weighted adaptively with their inputdependent attentions can significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensional Dynamic Convolution (ODConv), a more powerful dynamic convolution design, to advance this line of research. ODConv leverages a multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many prevailing CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various CNN backbones including both light-weight (e.g., MobileNetV2 family) and large ones (e.g., ResNet family). Intriguingly, thanks to its improved feature extraction ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, introducing substantially fewer extra parameters to the final models. Code will be made publicly available.

heading: 1 INTRODUCTION
text: In the past decade, we have witnessed the tremendous success of deep Convolutional Neural Networks (CNNs) in many computer vision applications (Krizhevsky et al., 2012; Girshick et al., 2014; Long et al., 2015; He et al., 2017). The most common way of constructing a deep CNN is to stack a number of convolutional layers as well as other basic layers organized with the selected feature connection topology. Along with great advances in CNN architecture design by manual engineering (Krizhevsky et al., 2012; He et al., 2016; Howard et al., 2017) and automatic searching (Zoph & Le, 2017; Pham et al., 2018; Howard et al., 2019), lots of prevailing classification backbones have been presented. Recent works (Wang et al., 2017; Hu et al., 2018b; Park et al., 2018; Woo et al., 2018; Yang et al., 2019; Chen et al., 2020) show that incorporating attention mechanisms into convolutional blocks can further push the performance boundaries of modern CNNs, and thus it has attracted great research interest in the deep learning community.
The well-known SENet (Hu et al., 2018b) uses an attention module consisting of squeeze and excitation operations to adaptively recalibrate the output features of convolutional layers, strengthening the representation power of a CNN via encouraging informative feature channels while suppressing less important ones. Numerous attentive feature recalibration variants (Woo et al., 2018; Park et al., 2018; Hu et al., 2018a) have been proposed since then. In Lin et al. (2020) and Quader et al. (2020), two attention extensions to modulate the convolutional weights instead of the output features are also presented. Unlike the aforementioned methods in which the number of convolutional parameters of a target network is fixed, dynamic convolution, which applies the attention mechanism over n
1Here we follow the definitions in Yang et al. (2019) and Chen et al. (2020) where a convolutional kernel refers to the filter set of a convolutional layer.
additive convolutional kernels to increase the size and the capacity of a network while maintaining efficient inference, has recently become popular in optimizing efficient CNNs. This line of research is pioneered by Conditionally Parameterized Convolutions (CondConv) (Yang et al., 2019) and Dynamic Convolution (DyConv) (Chen et al., 2020) whose basic ideas are the same. Generally, unlike a regular convolutional layer which applies the same (i.e., static) convolutional kernel to all input samples, a dynamic convolutional layer learns a linear combination of n convolutional kernels weighted with their attentions conditioned on the input features. Despite significant accuracy improvements for light-weight CNNs, dynamic convolution designed in this way has two limitations. Firstly, the main limitation lies in the attention mechanism design. The dynamic property of CondConv and DyConv comes from computing convolutional kernels as a function of the input features. However, we observe that they endow the dynamic property to convolutional kernels through one dimension (regarding the convolutional kernel number) of the kernel space while the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. As a result, the weights of each convolutional kernel share the same attention scalar for a given input, limiting their abilities to capture rich contextual cues. That is, the potentials of dynamic convolutional property have not been fully explored by existing works, and thus they leave much room for improving the model performance. Secondly, at a convolutional layer, replacing regular convolution by dynamic convolution increases the number of convolutional parameters by n times. When applying dynamic convolution to a lot of convolutional layers, it will heavily increase the model size. To handle this limitation, Li et al. (2021) proposes a dynamic convolution decomposition method which can get more compact yet competitive models. Instead, in this paper we address both of the above limitations from a new perspective: inserting a more diverse and effective attention mechanism into the convolutional kernel space.
Our core contribution is an improved dynamic convolution design called Omni-dimensional Dynamic Convolution (ODConv). Unlike existing works discussed above, at any convolutional layer, ODConv leverages a multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner. We show that these four types of attentions learnt by our ODConv are complementary to each other, and progressively applying them to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN. Consequently, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, introducing substantially fewer extra parameters to the final models.
As a drop-in design, ODConv can be used to replace regular convolutions in many prevailing CNN architectures. It strikes a better tradeoff between model accuracy and efficiency compared to existing dynamic convolution designs, as validated by extensive experiments on the large-scale ImageNet classification dataset (Russakovsky et al., 2015) with various CNN backbones. ODConv also shows better recognition performance under similar model complexities when compared to other state-ofthe-art attention methods for output feature recalibration (Woo et al., 2018; Hu et al., 2018b; Wang et al., 2020; Lin et al., 2020) and for convolutional weights modification (Ma et al., 2020; Lin et al., 2020; Quader et al., 2020). Furthermore, the performance improvements by ODConv for the pretrained classification models can transfer well to downstream tasks such as object detection on the MS-COCO dataset (Lin et al., 2014), validating its promising generalization ability.

heading: 2 RELATED WORK
text: Deep CNN Architectures. AlexNet (Krizhevsky et al., 2012) ignited the surge of deep CNNs by winning the ImageNet classification challenge 2012. Since then, lots of well-known CNN architectures such as VGGNet (Simonyan & Zisserman, 2015), InceptionNet (Szegedy et al., 2015), ResNet (He et al., 2016), DenseNet (Huang et al., 2017) and ResNeXt (Xie et al., 2017) have been proposed, which are designed to be much deeper and have more sophisticated connection topologies compared with AlexNet. To ease the deployment of inference models on resource-limited platforms, MobileNets (Howard et al., 2017; Sandler et al., 2018) and ShuffleNet (Zhang et al., 2018b; Ma et al., 2018) are presented, which particularly make a tradeoff between model accuracy and efficiency. All the aforementioned CNNs are manually designed. Recently, researchers have also made great efforts (Zoph & Le, 2017; Pham et al., 2018; Howard et al., 2019) to automate the network design process. Our ODConv could be potentially used to boost the performance of most of these prevailing CNN architectures constructed with regular convolutional layers.
Attentive Feature Recalibration. Designing attentive feature recalibration modules to improve the performance of a CNN has been widely studied in recent years. Wang et al. (2017) proposes a specialized attention module consisting of a trunk branch and a mask branch, and inserts it into the intermediate stages of deep residual networks. SENet (Hu et al., 2018b) uses a seminal channel attention module termed Squeeze-and-Excitation (SE) to exploit the interdependencies between the channels of convolutional features. Many subsequent works improve SE from different aspects, following its two-stage design (i.e., feature aggregation and feature recalibration). BAM (Park et al., 2018) and CBAM (Woo et al., 2018) combine the channel attention module with the spatial attention module. Misra et al. (2021) presents an attention module having three branches conditioned on the features rotated along three different dimensions. GE (Hu et al., 2018a) introduces a gather operator to extract better global context from a large spatial extent. To enhance the feature aggregation capability, SRM (Lee et al., 2019) replaces the global average by the channel-wise mean and standard deviation. SKNets (Li et al., 2019) add an attention design over two branches with different sized convolutions to fuse multi-scale feature outputs. ECA (Wang et al., 2020) provides a more efficient channel attention design using cheaper 1D convolutions to replace the first fully connected layer of SE. Instead of recalibrating the output convolutional features by attention modules, dynamic convolution methods apply attention mechanisms to a linear combination of n convolutional kernels.
Dynamic Weight Networks. Making the weights of a neural network to be sample-adaptive via dynamic mechanisms has shown great potentials for boosting model capacity and generalization. Hypernetworks (Ha et al., 2017) use a small network called hypernetwork to generate the weights for a larger recurrent network called main network. MetaNet (Munkhdalai & Yu, 2017) adopts a meta learning model to parameterize the task-adaptive network for rapid generalization across a sequence of tasks. Jaderberg et al. (2015) proposes a Spatial Transformer module conditioned on the learnt features to predict the parametric transformation, and applies it to align the distorted input image. Dynamic Filter Network (Jia et al., 2016) uses a filter generation network to produce filters conditioned on an input, and processes another input with the generated filters. DynamoNet (Diba et al., 2019) uses dynamically generated motion filters to handle the action recognition problem. Kernel Prediction Networks (Bako et al., 2017; Mildenhall et al., 2018) leverage a CNN architecture to predict spatially varying kernels used for video denoising. WeightNet (Ma et al., 2020) appends a grouped fully connected layer to the attention feature vector of an SE block, generating the weights of a CNN used for image recognition. Lin et al. (2020) modifies the weights of convolutional layers with a gated module under the guidance of global context, while Quader et al. (2020) directly uses either an SE block or a simple activation function conditioned on the magnitudes of convolutional weights to modify the weights themselves. Our ODConv aims to address the limitations of recently proposed dynamic convolution (Yang et al., 2019; Chen et al., 2020) which differs from these methods both in focus and formulation, see the Introduction and Method sections for details.

heading: 3 METHOD
text: In this section, we first make a review of dynamic convolution via a general formulation. Then, we describe the formulation of our ODConv, clarify its properties and detail its implementation.

heading: 3.1 REVIEW OF DYNAMIC CONVOLUTION
text: Basic concept. A regular convolutional layer has a single static convolutional kernel which is applied to all input samples. For a dynamic convolutional layer, it uses a linear combination of n convolutional kernels weighted dynamically with an attention mechanism, making convolution operations be input-dependent. Mathematically, dynamic convolution operations can be defined as
y = (αw1W1 + ...+ αwnWn) ∗ x, (1)
where x ∈ Rh×w×cin and y ∈ Rh×w×cout denote the input features and the output features (having cin/cout channels with the height h and the width w), respectively; Wi denotes the ith convolutional kernel consisting of cout filters Wmi ∈ Rk×k×cin ,m = 1, ..., cout; αwi ∈ R is the attention scalar for weighting Wi, which is computed by an attention function πwi(x) conditioned on the input features; ∗ denotes the convolution operation. For conciseness, here we omit the bias term. CondConv vs. DyConv. Although the concept of dynamic convolution defined in Eq. 1 is proposed separately in CondConv (Yang et al., 2019) and DyConv (Chen et al., 2020), their implementations
are different, mainly in the structure of πwi(x) to compute αwi, the model training strategy, and the layer locations to apply dynamic convolutions. Specifically, both methods choose the modified SE structure for πwi(x), and CondConv uses a Sigmoid function while DyConv uses a Softmax function as the activation function to compute αwi. DyConv adopts a temperature annealing strategy in the training process to suppress the near one-hot output of the Softmax function. For all their tested CNN architectures, CondConv replaces the convolutional layers in the final several blocks (e.g., 6 for the MobileNetV2 backbones and 3 for the ResNet backbones) and the last fully connected layer, while DyConv replaces all convolutional layers except the first layer. These implementation differences lead to different results in model accuracy, size and efficiency for CondConv and DyConv.
Limitation Discussions. According to Eq. 1, dynamic convolution has two basic components: the convolutional kernels {W1, ...Wn}, and the attention function πwi(x) to compute their attention scalars {αw1, ...αwn}. Given n convolutional kernels, the corresponding kernel space has four dimensions regarding the spatial kernel size k×k, the input channel number cin and the output channel number cout for each convolutional kernel, and the convolutional kernel number n. However, for CondConv and DyConv, we can observe that πwi(x) allocates a single attention scalar αwi to the convolutional kernel Wi, meaning that all its cout filters Wmi ∈ Rk×k×cin ,m = 1, ..., cout have the same attention value for the input x. In other words, the spatial dimension, the input channel dimension and the output channel dimension for the convolutional kernel Wi are ignored by CondConv and DyConv. This leads to a coarse exploitation of the kernel space when they design their attention mechanisms for endowing n convolutional kernels with the dynamic property. This may also be one of the reasons why CondConv and DyConv show much lower performance gains to relatively large CNNs compared to efficient ones. Besides, compared to a regular convolutional layer, a dynamic convolutional layer increases the number of convolutional parameters by n times (but the increase of Multiply-Adds (MAdds) is marginal due to the additive property of n convolutional kernels). Typically, CondConv sets n = 8 and DyConv sets n = 4. Therefore, it will heavily increase the model size when applying dynamic convolution to a lot of convolutional layers. However, we empirically find that removing the attention mechanism from CondConv|DyConv (i.e., setting αwi = 1) almost diminishes the accuracy boosts for prevailing CNN backbones on the ImageNet dataset close to zero. For instance, on ResNet18, the top-1 gain averaged over 3 runs decreases from 1.78%|2.51% to 0.08%|0.14% when removing the attention mechanism from CondConv|DyConv, respectively. These observations indicate that the attention mechanism design plays the key role in dynamic convolution, and a more effective design may make a good balance between model accuracy and size.

heading: 3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION
text: In light of the above discussions, our ODConv introduces a multi-dimensional attention mechanism with a parallel strategy to learn diverse attentions for convolutional kernels along all four dimensions of the kernel space. Fig. 1 provides a schematic comparison of CondConv, DyConv and ODConv.
Formulation of ODConv. Following the notations in Eq. 1, ODConv can be defined as y = (αw1 αf1 αc1 αs1 W1 + ...+ αwn αfn αcn αsn Wn) ∗ x, (2)
where αwi ∈ R denotes the attention scalar for the convolutional kernel Wi, which is the same to that in Eq. 1; αsi ∈ Rk×k, αci ∈ Rcin and αfi ∈ Rcout denote three newly introduced atten-
tions, which are computed along the spatial dimension, the input channel dimension and the output channel dimension of the kernel space for the convolutional kernel Wi, respectively; denotes the multiplication operations along different dimensions of the kernel space. Here, αsi, αci, αfi and αwi are computed with a multi-head attention module πi(x) which will be clarified later.
A Deep Understanding of ODConv. In ODConv, for the convolutional kernel Wi: (1) αsi assigns different attention scalars to convolutional parameters (per filter) at k × k spatial locations; (2) αci assigns different attention scalars to cin channels of each convolutional filter Wmi ; (3) αfi assigns different attention scalars to cout convolutional filters; (4) αwi assigns an attention scalar to the whole convolutional kernel. Fig. 2 illustrates the process of multiplying these four types of attentions to n convolutional kernels. In principle, these four types of attentions are complementary to each other, and progressively multiplying them to the convolutional kernel Wi in the location-wise, channel-wise, filter-wise and kernel-wise orders makes convolution operations be different w.r.t. all spatial locations, all input channels, all filters and all kernels for the input x, providing a performance guarantee to capture rich context cues. Therefore, ODConv can significantly strengthen the feature extraction ability of basic convolution operations of a CNN. Moreover, ODConv with much fewer convolutional kernels can compete with or outperform CondConv and DyConv, introducing substantially fewer extra parameters to the final models. Extensive experiments are provided to validate these advantages. By comparing Eq. 1 and Eq. 2, we can clearly see that ODConv is a more generalized dynamic convolution. Moreover, when setting n = 1 and all components of αs1, αc1 and αw1 to 1, ODConv with only filter-wise attention αf1 will be reduced into: applying an SE variant conditioned on the input features to the convolutional filters, then followed by convolution operations (note the original SE (Hu et al., 2018b) is conditioned on the output features, and is used to recalibrate the output features themselves). Such an SE variant is a special case of ODConv.
Implementation. For ODConv, a critical question is how to compute four types of attentions αsi, αci, αfi and αwi for the convolutional kernel Wi. Following CondConv and DyConv, we also use a SE-typed attention module (Hu et al., 2018b) but with multiple heads as πi(x) to compute them, whose structure is shown in Fig. 1. Specifically, the input x is squeezed into a feature vector with the length of cin by channel-wise Global Average Pooling (GAP) operations first. Subsequently, there is a Fully Connected (FC) layer and four head branches. A Rectified Linear Unit (ReLU) (Krizhevsky et al., 2012) comes after the FC layer. The FC layer maps the squeezed feature vector to a lower dimensional space with the reduction ratio r (according to the ablative experiments, we set r = 1/16 in all main experiments, avoiding high model complexity). For four head branches, each has an FC layer with the output size of k × k, cin × 1, cout × 1 and n× 1, and a Softmax or Sigmoid function to generate the normalized attentions αsi, αci, αfi and αwi, respectively. We adopt the temperature annealing strategy proposed in DyConv to facilitate the training process. For easy implementation, we follow DyConv and apply ODConv to all convolutional layers except the first layer of each CNN architecture tested in our main experiments, and share αsi, αci and αfi to all convolutional kernels. In the Experiments section, a comparison of the inference speed for different dynamic convolution methods is provided. In the Appendix, we further provide a computational cost analysis of ODConv and an ablation study of applying ODConv to different layer locations.

heading: 4 EXPERIMENTS
text: In this section, we provide comprehensive experiments on two large-scale image recognition datasets with different CNN architectures to validate the effectiveness of ODConv, compare its performance with many attention based methods, and study the design of ODConv from different aspects.

heading: 4.1 IMAGE CLASSIFICATION ON IMAGENET
text: Our main experiments are performed on the ImageNet dataset (Russakovsky et al., 2015) which is one of the most challenging image classification datasets. It has over 1.2 million images for training and 50,000 images for validation, including 1,000 object classes.
CNN Backbones. We use MobileNetV2 (Sandler et al., 2018) and ResNet (He et al., 2016) families for experiments, covering both light-weight CNN architectures and larger ones. Specifically, we choose ResNet18, ResNet50, ResNet101, and MobileNetV2 (1.0×, 0.75×, 0.5×) as the backbones. Experimental Setup. In the experiments, we consider existing dynamic convolution methods including CondConv (Yang et al., 2019), DyConv (Chen et al., 2020) and DCD (Li et al., 2021) as the key reference methods for performance comparisons on all CNN backbones. On the ResNet backbones, we also compare our ODConv with many state-of-the-art methods using attention modules: (1) for output feature recalibration including SE (Hu et al., 2018b), CBAM (Woo et al., 2018) and ECA (Wang et al., 2020), and (2) for convolutional weights modification including CGC (Lin et al., 2020), WE (Quader et al., 2020) and WeightNet (Ma et al., 2020). For fair comparisons, we use public codes of these methods and adopt the popular training and test settings used in the community for implementing the experiments. The models trained by all methods use the same settings including the batch size, the number of training epochs, the learning rate schedule, the weight decay, the momentum and the data processing pipeline. Moreover, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons. Experimental details are described in the Appendix.
Results Comparison on MobileNets. Table 1 shows the results comparison on the MobileNetV2 (1.0×, 0.75×, 0.5×) backbones. As CondConv and DyConv are primarily proposed to improve the performance of efficient CNNs, they all bring promising top-1 gains to the light-weight MobileNetV2 (1.0×, 0.75×, 0.5×) backbones. Comparatively, our ODConv (1×) with one single convolutional kernel performs better than CondConv (8×) with 8 convolutional kernels, and its performance is also on par with DyConv (4×) with 4 convolutional kernels. Note that these competitive results of ODConv (1×) are obtained with significantly lower numbers of extra parameters, validating that our ODConv can strike a better tradeoff between model accuracy and size. Besides, ODConv (1×) also performs better than DCD in most cases. ODConv (4×) always achieves the best results on all backbones. The results comparison with an increase number of training epochs (from 150 to 300) can be found in the Appendix, from which we can observe similar performance trends.
Results Comparison on ResNets. Table 2 shows the results comparison on the ResNet18 and ResNet50 backbones which are much larger than the MobileNetV2 backbones. We can obtain the following observations: (1) On the ResNet18 backbone, dynamic convolution methods (CondConv, DyConv, DCD and our ODConv) and convolutional weights modification methods (CGC, WE and WeightNet) mostly show better performance than output feature recalibration methods (SE, CBAM and ECA), although they all use attention mechanisms. Comparatively, our ODConv (1×) with one single convolutional kernel outperforms the other methods both in model accuracy and size, bringing 2.85% top-1 gain to the baseline model. ODConv (4×) gets the best results, yielding a top-1 gain of 3.72%; (2) However, on the larger ResNet50 backbone, CondConv, DyConv and DCD show worse performance than most of the other methods even though they have significantly increase numbers of parameters. Due to the increased parameter redundancy, adding more parameters to larger networks is usually less effective in improving model accuracy compared to small networks. Thanks to the proposed multi-dimensional attention mechanism, ODConv can address this problem better, achieving superior performance both in model accuracy and size for larger backbones besides lightweight ones. To further validate the performance of ODConv on very deep and large networks, we also apply ODConv to the ResNet101 backbone, and the results comparison is provided in Table 3. Again, ODConv shows very promising results, yielding 1.57% top-1 gain with ODConv (1×).

heading: 4.2 OBJECT DETECTION ON MS-COCO
text: Given a backbone model pre-trained on the ImageNet classification dataset, a critical question is whether the performance improvement by ODConv can be transferred to downstream tasks or not. To explore it, we next perform comparative experiments on the object detection track of MS-COCO dataset (Lin et al., 2014). The 2017 version of MS-COCO dataset contains 118,000 training images and 5,000 validation images with 80 object classes.
Experimental Setup. We use the popular MMDetection toolbox (Chen et al., 2019) for experiments with the pre-trained ResNet50 and MobileNetV2 (1.0×) models as the backbones for the detector. We select the mainstream Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) detectors with Feature Pyramid Networks (FPNs) (Lin et al., 2017) as the necks to build the basic object detection systems. For a neat comparison, the convolution layers in the FPN necks still use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively. Experimental details are described in the Appendix.
Results Comparison. From the results shown in Table 4, we can observe similar performance improvement trends as on the ImageNet dataset. For Faster R-CNN|Mask R-CNN with the pretrained ResNet50 backbone models, CondConv (8×) and DyConv (4×) show an AP improvement of 0.9%|0.8% and 1.1%|1.2% to the baseline model, respectively, while our method performs much better, e.g., ODConv (1×) with one single convolutional kernel shows an AP improvement of 2.0%|2.1%, respectively. With the pre-trained MobileNetV2 (1.0×) backbone models, our ODConv (1×) performs obviously better than CondConv (8×), and its performance is on par with that of DyConv (4×) as their AP gap is only 0.2% for both two detectors, striking a better model accuracy and efficiency tradeoff. Similar boosts to AP scores for small, medium and large objects are also obtained by three methods on both detectors. ODConv (4×) always achieves the best AP scores.

heading: 4.3 ABLATION STUDIES
text: Finally, we conduct a lot of ablative experiments on the ImageNet dataset, in order to have a better understanding of our ODConv.
Reduction Ratio Selection. Our first set of ablative experiments is for the selection of the reduction ratio r used in the attention structure πi(x). From the results shown in Table 5, we can find that under r = 1/4, r = 1/8 and r = 1/16, ODConv consistently obtains large performance improvements to the baseline ResNet18 model (2.85-3.16% top-1 gain), and the extra MAdds are negligible. Comparatively, ODConv with r = 1/16 strikes the best tradeoff between model accuracy and efficiency, and thus we choose it as our default setting.
Convolutional Kernel Number. Accordingly to the main experiments discussed beforehand, ODConv can strike a better tradeoff between model accuracy and size, compared to existing dynamic convolution methods. For a better understanding of this advantage, we next perform the second set of ablative experiments on the ImageNet dataset, training the ResNet18 backbone based on ODConv with different settings of the convolutional kernel number n. Table 6 shows the results. It can be seen that ODConv with one single convolutional kernel brings near 3.0% top-1 gain to the ResNet18 baseline, and the gain tends to be saturated when the number of convolutional kernels is set to 8.
Four Types of Attentions. Note that our ODConv has four types of convolutional kernel attentions αsi, αci, αfi and αwi computed along all four dimensions of the kernel space, respectively. Then, we perform another set of ablative experiments on the ResNet18 backbone with different combinations of them to investigate their dependencies. Results are summarized in Table 7, from which the strong complementarity of four types of attentions αsi, αci, αfi and αwi can be clearly observed.
Inference Speed. Besides the size and the MAdds for a CNN model, the inference speed at runtime is very important in the practical model deployment. Table 8 provides a comparison of the inference speed for different dynamic convolution methods (both on GPU and CPU). It can be seen that the models trained with our ODConv (1×) are faster than the counterparts trained with DyConv and DCD. Comparatively, the models trained with CondConv show the fastest run-time speed, this is because CondConv is merely used to the final several blocks and the last fully connected layer as discussed in the Method section. When adding ODConv (1×) to the same layer locations as for CondConv, we can obtain more accurate and efficient models.
More Ablations. In the Appendix, we provide more ablative experiments to study: (1) the effects of applying ODConv to different layer locations; (2) the importance of the temperature annealing strategy; (3) the choices of the activation functions (Sigmoid or Softmax); (4) the influence of the attention sharing strategy; (5) illustrative train+val curves; (5) new experiments for the rebuttal.

heading: 5 CONCLUSION
text: In this paper, we present a new dynamic convolution design called Omni-dimensional Dynamic Convolution (ODConv) to promote the representation power of deep CNNs. ODConv leverages a multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner, and progressively applying these attentions to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN. Experimental results on the ImageNet and MS-COCO datasets with several prevailing CNN architectures validate its superior performance.

heading: A APPENDIX
text: In this section, we describe the supplementary materials including: (1) a computational cost analysis of ODConv; (2) implementation details for the experiments on the ImageNet and MS-COCO datasets; (3) more ablative experiments on the ImageNet dataset; (4) illustrative training and validation curves to compare the stability of the model training process with different dynamic convolution methods.
A.1 COMPUTATIONAL COST OF ODCONV
For a convolutional layer with ODConv, its computational cost can be easily calculated according to the design and the multiplication process of four types of attentions shown in Fig. 1 and Fig. 2, respectively. Specifically, following the notations defined in the Method section of the main paper, the extra MAdds of ODConv (1×) over regular convolution (which has hwk2cincout MAdds, without consideration of the bias term) can be calculated as
MAdds = hwcin + cin(2cin + cout + k
2)
r + k2cin(1 + 2cout). (3)
For ODConv (n×), the extra MAdds over regular convolution can be calculated as
MAdds = hwcin + cin(2cin + cout + k
2 + n)
r + k2cin(1 + cout + 2ncout). (4)
Compared to hwk2cincout MAdds for regular convolution, the extra MAdds by ODConv are small.
A.2 EXPERIMENTAL DETAILS
Experimental Details on ImageNet. Recall that we use MobileNetV2 (Sandler et al., 2018) and ResNet (He et al., 2016) families for experiments on the ImageNet dataset, covering both lightweight CNN architectures and larger ones. For fair comparisons, we adopt the popular training settings used in the community to train the respective backbone models with all methods. Specifically, for ResNet18, ResNet50 and ResNet101, all models are trained with SGD for 100 epochs. We set the batch size as 256, the weight decay as 0.0001 and the momentum as 0.9. The learning rate starts at 0.1, and is divided by 10 every 30 epochs. Following DyConv (Chen et al., 2020), for our ODConv, we also use dropout rate of 0.1 for ResNet18. We use dropout rate of 0.2 for ResNet50 and ResNet101 for our ODConv. For MobileNetV2 (1.0×, 0.75×, 0.5×), all models are trained with SGD for 150 epochs (we also have the experiments for training all models with 300 epochs in this Appendix). We set the batch size as 256, the weight decay as 0.00004 and the momentum as 0.9. The learning rate starts at 0.05, and is scheduled to arrive at zero within a single cosine cycle. Following DyConv, for our ODConv, we also use dropout rate of 0.2 for MobileNetV2 (1.0×) and dropout rate of 0.1 for MobileNetV2 (0.75×, 0.5×). Regarding the temperature annealing strategy used for DyConv and ODConv, the temperature reduces from 30 to 1 linearly in the first 10 epochs for all models. All experiments are performed on the servers having 8 GPUs. We follow the standard protocols to train and evaluate each CNN backbone. For training, images are resized to 256 × 256 first, and then 224 × 224 crops are randomly sampled from the resized images or their horizontal flips normalized with the per-channel mean and standard deviation values. For evaluation, we report top-1 and top-5 recognition rates using the center image crops.
Experimental Details on MS-COCO. Recall that we use the popular MMDetection toolbox (Chen et al., 2019) for experiments on the MS-COCO dataset with the pre-trained ResNet50 and MobileNetV2 (1.0×) models as the backbones for the detector. We select the mainstream Faster RCNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) detectors with Feature Pyramid Networks (FPNs) (Lin et al., 2017) as the necks to build the basic object detection systems. For a neat and fair comparison, the convolution layers in the FPN necks still use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively. All experiments are performed on a server having 8 GPUs with a mini-batch size of 2 images per GPU. We finetune these detectors on the MS-COCO training set following the 1× learning rate schedule, which indicates a total of 12 epochs with the learning rate divided by 10 at the 8th epoch and the 11th epoch, respectively.
Following DyConv, the temperature annealing strategy is not used in the downstream experiments on the MS-COCO dataset, as it is prone to worse results. In the validation, we report the standard Average Precision (AP) under IOU thresholds ranging from 0.5 to 0.95 with an increment of 0.05. We also keep AP scores for small, medium and large objects.
A.3 MORE ABLATIVE EXPERIMENTS ON IMAGENET
Besides the ablative experiments described in the main paper, here we provide more ablative experiments, for a better understanding of our ODConv.
Effects of Applying ODConv to Different Layer Locations. In the Experiments section of the main paper, we show that applying dynamic convolution to less layers leads to a faster runtime speed compared to applying it to more layers. For easy implementation, we follow DyConv and apply ODConv to all convolutional layers except the first layer of each CNN architecture tested in our main experiments. However, it is necessary to study the effects of applying ODConv to different layer locations. A results comparison of adding ODConv to different layers of the MobileNetV2 (0.5×) backbone is given in Table 9, showing that our default setting (i.e., adding ODConv to all convolutional layers except the first layer) has the largest gain. This indicates that adding ODConv to more convolutional layers tends to have a model with higher recognition accuracy.
Importance of Temperature Annealing Strategy. Following DyConv, we also use the temperature annealing strategy to facilitate the model training process of our ODConv. To study its effect, we perform a set of ablative experiments on the ImageNet dataset, training the ResNet18 backbone based on ODConv with or without using the temperature annealing strategy. From the results shown in Table 10, we can find that the temperature annealing strategy is really important, which brings 1.06%|1.04% top-1 improvement to the ResNet18 model based on ODConv (1×)|ODConv (4×).
Choices of the Activation Functions. Recall that in the structure of πi(x) for our ODConv, we use two different activation functions (either a Sigmoid function or a Softmax function) to compute four types of attentions αsi, αci, αfi and αwi. It is also critical to compare the performance of different activation functions for ODConv. Since the activation function choices for the channel dimension and the convolutional kernel dimension have been thoroughly discussed in the papers of SE and DyConv, respectively, we adopt their suggestions for three related attentions defined in our ODConv. Here we perform another set of ablative experiments, focusing on the choices of the activation function for the spatial dimension. With the ImageNet dataset, we use ResNet18 as the test backbone network to explore the effects of different activation functions for computing the attention of ODConv along the spatial dimension of the kernel space. Table 11 summarizes the results, from which we can see that the Sigmoid function performs better than the Softmax function, and thus we use the Sigmoid function to compute the attention scalars along the spatial dimension for ODConv.
Attention Sharing Strategy. As we discussed in the Method section, we share three attentions αsi, αci and αfi to all convolutional kernels for the easy implementation as well as for more efficient training. However, training separate αsi, αci and αfi for each additive convolutional kernel will bring slightly improved model accuracy, as can be seen from the ablative results in Table 12. That is, when preferring to get more accurate models, the attention sharing strategy would be removed.
Training MobileNetV2 Backbones for 300 Epochs. Recall that in the Experiments section of the main paper, we compare our ODConv with CondConv, DyConv and DCD on MobileNetV2 (1.0×, 0.75×, 0.5×), following the popular training settings used in the community, where all models are trained for 150 epochs. To validate the effectiveness of our ODConv further, we also conduct another set of experiments using an increased number of training epochs (300 instead of 150) as adopted in DyConv (Chen et al., 2020). All the other training settings remain the same to those for the experiments described in the main paper. For a clean comparison of all methods, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016). Results are summarized in Table 13 where performance gains show a similar trend as those in Table 1 for the model training with 150 epochs. Results in Fig. 3 further show that our ODConv gets a better trade-off between model accuracy and size even on the light-weight MobileNetV2 backbones compared to CondConv and DyConv.
A.4 ILLUSTRATION OF MODEL TRAINING AND VALIDATION CURVES
Fig. 4 illustrates the training and validation accuracy curves of the ResNet18 models trained on the ImageNet dataset with CondConv, DyConv, ODConv (1×) and ODConv (4×), respectively. We can see that our ODConv shows consistent high top-1 gains throughout the training process compared to three dynamic convolution counterparts.
A.5 MORE EXPERIMENTS FOR THE REBUTTAL
In this section, we provide a lot of extra experiments conducted for the rebuttal.
Performance Comparison on CNN Backbones Added with the SE Module. Recall that SE (Hu et al., 2018b) performs attention on the output features of a convolutional layer, while our ODConv
performs attention on the convolutional kernels, and thus they are likely to be complimentary. To validate this, we perform two sets of experiments on the ImageNet dataset with the MobileNetV3Small and ResNet18 backbones. Note that the MobileNetV3-Small backbone (Howard et al., 2019) is actually constructed with the SE module. For the ResNet18 backbone, we use its SE variant reported in Table 2. In the first set of experiments, we train the MobileNetV3-Small backbone with CondConv, DyConv, DCD and our ODConv separately, adopting the same training settings used for the MobileNetV2 backbones. In the second set of experiments, we use the ResNet18+SE to mainly test the potential of our ODConv. Detailed results are summarized in Table 14 and Table 15. It can be seen that our ODConv brings promising improvements to these two backbones already incorporating the SE module. Even for MobileNetV3-Small, a much stronger light-weight backbone network, our ODConv (4×) obviously performs better than all counterparts, and our ODConv (1×) with the smallest model size also outperforms 3 of 4 counterparts.
Performance Boost Using Heavy Data Augmentations and a Longer Training Schedule. Recall that in the main paper, we do not use advanced training tricks such as mixup (Zhang et al., 2018a) and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons. To better explore the potential of our ODConv, it would be necessary to have a study about whether ODConv can also work well when using heavy data augmentations and a longer training schedule. To this end, we perform a set of experiments on the ImageNet dataset with the ResNet18 backbone trained with ODConv (4×). In the experiments, we set r=1/16 for our method, first train ResNet18 with label smoothing, mixup and (label smoothing+mixup) separately for 100 epochs (the other training settings are the same to those used for Table 2), and then train ResNet18 with (label smoothing+mixup) for 120 epochs instead of 100 epochs. Detailed results are summarized in Table 16. It can be seen that our ODConv works well when using aggressive data augmentations and a longer training schedule, showing further improved performance.
Feature Pooling Strategy. Note that when computing four types of attentions in our ODConv, the inputs features are always globally averaged, reducing their spatial dimension to one pixel. According to the first three results of Table 7 in the main paper, the spatial convolutional kernel attention αsi brings the largest gain to the baseline model compared to the other two attentions (αci and αfi computed along the dimension of the input channels and the filters, respectively). Intuitively, at the first glance, it may easily lead to a weird feeling as the input features used to predict αsi (as well
as the other three attentions) do not include spatial information after the global average pooling. However, it should be noted that our ODConv uses the attentions generated from the input features to modify the convolutional kernels, which is quite different from popular self-calibration attention mechanisms (e.g., SE (Hu et al., 2018b) and CBAM (Woo et al., 2018) use the channel attentions generated from the output features of a convolutional layer to recalibrate the output features themselves). As our ODConv is not a self-calibration attention module, reducing the spatial dimension of the input features to one pixel (but the length of the reduced vector is large enough) will not affect its promising performance. To validate this, we conduct another experiment on the ImageNet dataset with the ResNet18 backbone. In the experiment, for αsi, we pool the input features to have a size of 3×3 instead of 1×1, and show the results in Table 17. It can be seen that preserving more spatial information only brings 0.06%|0.02% gain to top-1|top-5 accuracy, which well proves the effectiveness of our current design.
Comparison of Training Cost. Recall that in the main paper, we provide a comparison of the inference speed for different dynamic convolution methods. Here, we further perform a set of experiments to compare the training cost of our ODConv, CondConv, DyConv and DCD. Specifically, experiments are performed on the ImageNet dataset with the ResNet50 and MobileNetV2 backbones, using the same training settings as in the main paper. Regarding the training cost, we report results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of hours for the whole training) in Table 18. Regarding different dynamic convolution methods, it can be seen that the training cost trend is somewhat similar to that for the runtime inference speed reported in Table 8. Generally, the training cost for dynamic convolution methods is obviously heavier (about 2× to 3×) than that for the baseline model, which is mainly due to the time-intensive back-propagation process for additive convolutional kernels weighted with input-dependent attentions.
A Deep Understanding of Learnt Attention Values. Note that our ODConv leverages a multidimensional attention mechanism with a parallel strategy to learn four types of attentions αsi, αci, αfi, αwi for convolutional kernels along all dimensions of the kernel space at any convolutional layer. In principle, progressively multiplying these four types of attentions to the convolutional kernel in the location-wise, channel-wise, filter-wise and kernel-wise orders makes convolution operations be different w.r.t. all spatial locations, all input channels, all filters and all kernels for each input sample, providing a performance guarantee to capture rich context cues. To have a better understanding of this powerful attention mechanism, it is necessary to study the learnt attention values. To this end, we use the well-trained ResNet18 models reported in Table 7 and all 50,000 images in the ImageNet validation set, and conduct a set of experiments to analyze the learnt attention values for αsi, αci, αfi, αwi and their combination by providing lots of visualization examples (obtained with the Grad-CAM++ method (Chattopadhyay et al., 2018)) and the statistical distributions across different layers. Detailed computational process and the results are shown in Fig. 5 and Fig. 6. We can get the following observations: (1) Four types of attentions αsi, αci, αfi and αwi are complementary to each other by visualization examples, which echoes the conclusion from Table 7. The
most salient attention will combat the failure cases of the other three attentions, helping the model to produce accurate predications. (2) All four types of attentions demonstrate varying attention value distribution trends across different layers of the trained model, showing their capability to capture rich context cues. (3) Each attention has its own value distribution trend, which is more diverse for αsi and αwi than for αci and αfi, showing αsi and αwi maybe more important to some degree. Note that the dimension of attentions αsi, αci, αfi and αwi is quite different, e.g., 9 for αsi with 3 × 3 convolutional kernels, 4 for αwi with n = 4, and typically several hundred for αci and αfi. This is another perspective why they are different and complementary in their design nature regarding all four dimensions of the convolutional kernel space for any convolutional layer.
Limitations of ODConv. On the one side, according the results shown in Table 1, Table 2 and Table 8, although our ODConv shows obviously better performance under the similar model size (e.g., ODConv (4×) vs. DyConv (4×)), it leads to slightly increased FLOPs and introduces small extra latency to the runtime inference speed, as the proposed four types of convolutional kernel attentions αsi, αci, αfi and αwi introduce a bit more learnable parameters. Under the similar model size,
the training cost of our ODConv is also somewhat heavier than reference methods (e.g., ODConv (4×) vs. DyConv (4×)), according to the result shown in Table 18. On the other side, although we provide a wide range of ablative studies to analyze the effect of different hyperparameters of our ODConv using the ResNet18 backbone with the ImageNet dataset, and apply the resulting combination of the hyperparameters to all backbone networks, it is not the optimal setting to different backbone networks. This suggests the way to reduce the computational cost of our method, finding a proper combination to constrain the final model to the target speed and accuracy for a particular circumstance. Besides, the potential of applying ODConv to more deep and large backbones beyond ResNet101 has not been explored due to the constraint of our available computational resource.

Review summaries:
This work mainly focuses on designing a new dynamic network for large-scale image recognition problems. Specifically, the author discussed the weakness of the existing dynamic convolution operations, and based on the analysis the author proposed a novel framework with the name ODConv. Extensive experiments confirm the superiority of the proposed new framework.

The authors present ODConv, a type of dynamic convolutional operation. ODConv combines two prior ideas, i.e. (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, and also generalizes to all remaining dimensions of convolutional filters. The authors propose to use ODConv as drop-in replacements for regular convolutions in standard CNNs. Experiments and analysis over several tasks demonstrate that ODConv has noticeable advantages over alternatives and offers a good tradeoff between performance and compute.