Omni-Dimensional Dynamic Convolution
Authors: Chao Li
Aojun Zhou
Anbang Yao
Venue: ICLR 2022 Spotlight (2022)
Keywords: Convolutional Neural Networks
Dynamic Convolution
Attention
Image Classification

Abstract: Learning a single static convolutional kernel in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning a linear combination of n convolutional kernels weighted with their input-dependent attentions can significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research. ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77%~5.71%|1.86%~3.72% absolute top-1 improvements to MobivleNetV2|ResNet family on the ImageNet dataset. Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters. Furthermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights. Code and models will be available at https://github.com/OSVAI/ODConv.

Published as a conference paper at ICLR 2022
OMNI-DIMENSIONAL DYNAMIC CONVOLUTION
Chao L Aojun Zhou², Anbang Yao IIntel Labs China, (CUHK-SenseTing Joint Lab, The Chinese University of Hong Kong chao. 113e intel. com, aojun. zhoue com, anbang. yao i intelll...mm com
ABSTRACT
Learning single static convolutional kernel in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning linear com- bination of n convolutional kernels weighted with their input-dependent attentions significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convo- lutional kernel number) of the kernel space, but the other three dimensions (re- garding the spatial size, the input channel number and the output channel num- ber for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensionnll Dynamic Convolution (ODConv), more generalized yet ele- gant dynamic convolution design, to advance this line of research. ODConv lever- ages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimen- sions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that OD- Conv brings solid accuracy boosts for various prevailing CNN backbones includ- ing both light-weight and large ones, e. 3.77%~55.71 (5.7% 1.86%~~3.72%% absolute top-1 improvements to MobivleNetV2 Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters. Fur- thermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights. Code and models will be available at /tttss:://ttttbb.. http:///occooom///////////ooo
family on the ImageNet dataset.
INTRODUCTION
In the past decade, we have witnessed the tremendous success of deep Convolutional Neural Net- works (CNNs) in many computer vision applications (Krizhessky et al. 2012 Girshick et al.) 2014 Long al. 2015 He et al., 2017). The most common way of constructing deep CNN is stack a number of convolutional layers as well as other basic layers organized with the predefined feature connection topology. Along with great advances in CNN architecture design by manual engineer- ing (Krizhevsky et al.] [2012] He et al.] [2166; Howard et al. 2017) and automatic searching (Zoph) [ [219)) , lots of prevailing classification backbones have been presented. Recent works (Wang et al. 2017 Hu et al., 2018b) Park et al., 2018: Wooo et al. 2018 Yang et al., 2019 Chen al. (020)) show that incorporating attention mechanisms into convolutional blocks can further push the performance boundaries of modern CNNs, and thus it has attracted great research interest in the deep learning community.
& Le [017 Pham al.) 2018 Howard et al.
The well-known SENet (Hu et al. 2018b) uses attention module consisting of squeeze and exci- tation operations to adaptively recalibrate the output features of convolutional layers, strengthening
"This work was done when Chao Li was intern Intel Labs China, supervised by Anhang Yao who proposed the original idea and led the writing of the paper. Corresponding author.
Here, we follow the definitions in (Yang al. 2019 Chen et al. 2020) where convolutional kernel refers the filter of convolutional layer.

---

Published as a conference paper at ICLR 2022
the representation power of a CNN via encouraging informative feature channels while suppressing less important ones. Numerous attentive feature recalibration variants (Woo et al. 2018 Park et al, 2018 [Hu et al. 2018a) have been proposed since then. In Lin et al. (2020) and Quader et al. (2020).
two attention extensions to modulate the convolutional weights instead of the output features are al- so presented. Unlike the aforementioned methods in which the number of convolutional parameters of a target network is fixed, dynamic convolution, which applies the attention mechanism over n additive convolutional kernels increase the size and the capacity of a network while maintaining efficient inference, has recently become popular in optimizing efficient CNNs. This line of research pioneered by Conditionally Parameterized Convolutions (CondConv) (Yang et al., (019) and Dy- namic Convolution (DyConv) (Chen et al. 2020) whose basic ideas are the same. Generally, unlike regular convolutional layer which applies the same (i.e., static) convolutional kernel to all input sam- ples, a dynamic convolutional layer learns a linear combination of n convolutional kernels weighted with their attentions conditioned on the input features. Despite significant accuracy improvements for light-weigh CNNs, dynamic convolution designed in this way has two limitations. Firstly, the main limitation lies in the attention mechanism design. The dynamic property of CondConv and DyCony comes from computing convolutional kernels function of the input features. However, we observe that they endow the dynamic property to convolutional kernels through one dimension (regarding the convolutional kernel number) of the kernel space while the other three dimension- s (regarding the spatial size, the input channel number and the ouiput channel number for each convolutional kernel) are overlooked. As result, the weights of each convolutional kernel share the same attention scalar for given input, limiting their abilities to capture rich contextual cues. That is, the potentials of dynamic convolutional property have not been fully explored by existing works, and thus they leave considerable room for improving the model performance. Secondly, at convolutional layer, replacing regular convolution by dynamic convolution increases the number of convolutional parameters by times. When applying dynamic convolution to lot of convolutional layers, it will heavily increase the model size. To handle this limitation, Li et al. (021)) proposes a dynamic convolution decomposition method which can get more compact yet competitive models. Instead, in this paper we address both of the above limitations in a new perspective: formulating a more diverse and effective attention mechanism and inserting it into the convolutional kernel space.
Our core contribution is a more generalized yet elegant dynamic convolution design called Omni- dimensional Dynamic Convolution (ODConv). Unlike existing works discussed above, at any con- volutional layer, ODConv leverages a novel multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in par- allel manner. We show that these four types of attentions learnt by our ODConv are complementary to each other, and progressively applying them to the corresponding convolutional kernels can sub- stantially strengthen the feature extraction ability of basic convolution operations of CNN. Con- sequently, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters.
As a drop-in design, ODConv can be used to replace regular convolutions in many CNN archi- tectures. It strikes better tradeoff between model accuracy and efficiency compared to existing dynamic convolution designs, as validated by extensive experiments on the large-scale ImageNet classification dataset (Russakovsky et al., 2015) with various prevailing CNN backbones. ODCon- v also shows better recognition performance under similar model complexities when compared to other state-of-the-aa attention methods for output feature recalibration (Woo al., 2018 Hu al. 2018b] Wang et (002) [in et al. (2220) or for convolutional weight modification (Ma et al.) 2020 Lin et al./ 2020; Quader et al. (2020). Furthermore, the performance improvements by ODConv for the pre-trained classification models can transfer well to downstream tasks such as object detection
on the MS-COCO dataset (Lin et al. 2014), validating its promising generalization ability.
2 RELATED WORK
Deep CNN Architectures. AlexNet (Krizhevsky et al.] 2012) ignited the surge of deep CNNs
by winning the ImageNet classification challenge 20 Since then, lots of well-known CNN ar- chitectures such as VGGNet (Simonyan & Zisserman 2015). InceptionNet (Szegedy et al. 2015).
ResNet (He et al. 2016), DenseNet (Huang et al.) (017) and ResNeXt (Xie et al.. (017)) > have been proposed, which are designed to be much deeper and have more sophisticated connection topologies compared with Alex To ease the deployment of inference models on resource-limited platforms,
2

---

Published as a conference paper at ICLR 2022
MobileNets (Howard al. 2017 Sandler al. 2018 and ShuffleNet (Zhang et al. 2018b Ma et al., 2018) are presented. All the aforementioned CNNs are manually designed. Recently, researchers have also made great efforts (Zoph & Le 2017 Pham et al.) 2018 Howard et al. 2019) to automate the network design process. Our ODConv could be potentially ly used to boost their performance.
Attentive Feature Recalibration. Designing attentive feature recalibration modules improve the performance of a CNN has been widely studied in recent years. Wang et al. (2017) proposes a specialized attention module consisting of trunk branch and mask branch, and inserts into the intermediate stages of deep residual networks. SENet (Hu et al., 2018b) uses a seminal channel attention module termed Squeeze-ande suueeze-and-Excittlle channels of convolutional features. Many subsequent works improve SE from different aspects, fol- lowing its two-stage design (i.e., feature aggregation and feature recalibration). BAM (Park et al.) 2018) and CBAM (Woo et al. 2018) combine the channel attention module with the spatial attention module. Misra et al. (2021) presents an attention module having three branches conditioned on the features rotated along three different dimensions. GE (Hu et al. 200 8a) introduces gather opera- tor to extract better global context from a large spatial extent. To enhance the feature aggregation capability, SRM (Lee et al., 2019) replaces the global average by the thannel-wist mean and stan- dard deviation. SKNets (Li et al. 2019) add an attention design over two branches with different sized convolutions to fuse multi-scale feature outputs. ECA (Wang al. 2020) provides a more efficient channel attention design using cheaper 1D convolutions to replace the first fully connected layer of SE. Instead of recalibrating the output convolutional features by attention modules, dynamic convolution methods apply attention mechanisms to a linear combination of n convolutional kernels.
(SE) to exploit the inteddepndenncees between the
Dynamic Weight Networks. Making the weights of neural network to be sample-adaptive via dynamic mechanisms has shown great potentials for boosting model capacity and generalization. Hypernetorks (Ha et al. 2017) use a small network called hypernetwork to generate the weights for a larger recurrent network called main network. MetaNet (Munkhdalai & Yu, (017) adopts a meta learning model to parameterize the task task-adaptive network for rapid generalization across a se- quence of tasks. Jaderberg et al. (2015) proposes Spatial Transformer module conditioned on the learnt features to predict the parametric transformation, and applics it to align the distorted input image. Dynamic Filter Network (Jia et al. 2016) uses a filter generation network to produce filters conditioned on an input, and processes another input with the generated filters. DynamoNet (Dibal eet al., 2019) uses dynamically generated motion filters to handle the action recognition problem. Kernel Prediction Networks (Bako et al. 2017 Mildenhall et al. 2018)) leverage a CNN architecture to predict spatially varying kernels used for video denoising. WeightNet (Ma et al., (020)) appends grouped fully connected layer to the attention feature vector of an SE block, generating the weights of a CNN used for image recognition. [in al. (2020) modifies the weights of convolutional layers with a gated module under the guidance of global context, while Quader et al. (2020) directly uses either an SE block or a simple activation function conditioned on the magnitudes of convolutional weights to modify the weights themselves. Our ODCony aims to address the limitations of recent- ly proposed dynamic convolution (Yang et al.) 2019: Chen et al., (020)) which differs from these methods both in focus and formulation, see the Introduction and Method sections for details.
3
METHOD
In this section, we first make review of dynamic convolution via general formulation. Then, we describe the formulation of our ODConv, clarify its properties and detail its implementation.
3.1 REVIEW OF DYNAMIC CONVOLUTION
Basic concept. A regular convolutional layer has single static convolutional kernel which is ap- plied to all input samples. For dynamic convolutional layer, it uses a linear combination of n convolutional kernels weighted dynamically with attention mechanism, making convolution op- erations be input-dependent. Mathematically. dynamic convolution operations can be defined as
y == (a W1 + ... Qwn W., * x,
where x E R and y R bxx denote the input features and the output features (having Cin/Cout channels with the height h and the width w). respectively; W, denotes the 7th convolutional kernel consisting of Cout filters Wine k m 1, ,,, Cout's Ohei IR is the attention scalar
3

---

Published as a conference paper at ICLR 2022
W1
Axt
W2
W2
4x2
MeLU
KeLU
Wn:
Wn
Seltross
Sigroed
Sproid
Sigood
FC
PC
PC
(b)
A schematic comparison of (a) DyConv (CondConv uses GAP+FC+Sigmoid) and (b) ODConv. Unlike CondConv and DyConv which compute a single attention scalar Qui for the con- volutional kernel W;, ODConv leverages novel multi-dimensiona. attention mechanism compute four types of attentions Otsi, Oci, fifi and Owni for W; along all four dimensions of the kernel space in parallel manner. Their formulations and implementations are clarified in the Method section.
Figure 1:
for weighting W, which is computed by an attention function Twis (z) conditioned on the input features; denotes the convolution operation. For conciseness, here we omit the bias term.
CondConv vs. DyConv. Although the concept of dynamic convolution defined in Eq. is proposed separately in CondConv (Yang et al. 2019) and DyConv (Chen et al. 2020), their implementations are different, mainly in the structure of (I) to compute Qwss the model training strategy, and the layer locations apply dynamic convolutions. Specifically, both methods choose the modified SE structure for Twi (x). and CondConv uses a Sigmoid function while DyConv uses a Softmax function as the activation function to compute Ouri- DyConv adopts temperature annealing strategy in the training process to suppress the near one-hot output of the Softmax function. For all their tested CNN architectures, CondConv replaces the convolutional layers the final several blocks (e.g., 6 for the MobileNetV2 backbones and 3 for the ResNet backbones) and the last fully connected layer, while DyCony replaces all convolutional layers except the first layer. These implementation differences lead to different results in model accuracy, size and efficiency for CondConv and DyConv.
Limitation Discussions. According to Eq. [ dynamic convolution has two basic components: the convolutional kernels (W1, }. and the attention function (w) to compute their attention scalars (n..n }. Given n convolutional kernels, the corresponding kernel space has four di- mensions regarding the spatial kernel size x the input channel number Cin and the output channel number Cour for each convolutional kernel, and the convolutional kernel number n. However, for CondConv and DyConv, we can observe that (x) allocates single attention scalar Oses to the convolutional kernel W,, meaning that all its Cous filters W E R* m L...., Cout have the same attention value for the input x. In other words, the spatial dimension, the input channel dimen- sion and the output channel dimension for the convolutional kernel W; are ignored by CondConv and DyConv. This leads to a coarse exploitation of the kernel space when they design their attention mechanisms for endowing π. convolutional kernels with the dynamic property. This may also be one the reasons why CondConv and DyConv show much lower performance gains to relatively larger CNNs compared to efficient ones. Besides, compared to regular convolutional layer, dy- namic convolutional layer increases the number of convolutional parameters by n times (although the increase of Multiply-Adds (MAdds) is marginal due to the additive property of n convolutional kernels). Typically, CondConv sets n = 8 and DyCony sets n = 4. Therefore, it will heavily in- crease the model size when applying dynamic convolution to lot of convolutional layers. However, we empirically find that removing the attention mechanism from CondConv CondCony/DyCom (i.c., setting Owi 1) almost diminishes the accuracy boosts for prevailing CNN backbones on the ImageNet dataset close to zero. For instance, on ResNet18, 8, the top-1 gain averaged over 3 runs decreases from 1.74%|2.50 51.1% % to 0.08%)0..44 144 when removing the attention mechanism from CondConv DDyConv. DonCCo. These observations indicate that the attention mechanism design plays the key role in dynamic con- volution, and a more effective design may strike good balance between model accuracy and size.
3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION
In light of the above discussions, ODConv introduces multi-dimensiona. attention mechanism with a parallel strategy to learn diverse attentions for convolutional kernels along all four dimensions of the kernel space. Fig. F provides schematic comparison of CondConv, DyCony and ODConv.
4

---

Published as a conference paper at ICLR 2022
(a)
(b)
W7"
aei
ae W7
(c)
yt
a W
(d)
[W
W
aw
W
W
Figure 2: Illustration of multiplying four types of attentions in ODConv to convolutional kernels progressively. (a) Location-wist multiplication operations along the spatial dimension, (b) channel- wise multiplication operations along the imput channel dimension, (c) filter-wise multiplication op- erations along the output channel dimension, and (d) kernel-wise multiplication operations along the kernel dimension of the convolutional kernel space. Notations are clarified in the Method section.
Formulation of ODConv. Following the notations in Eq. | ODConv can be defined as
(2) where Owi E I denotes the attention scalar for the convolutional kernel W,, which the same to that in Eq. O:si E R Occi E R and as E Roou denote three newly introduced atten- tions, which are computed along the spatial dimension, the input channel dimension and the output channel dimension of the kernel space for the convolutional kernel W. respectively; O denotes the multiplication operations along different dimensions of the kernel space. Here, Oksi Orai O'fi and awi computed with multi-head attention module (2) which will be clarified later.
y = (Owrl ② O'11 @ Oel ④ 0421 @ W1 + + Own @ O' ② Geen ② Otan ④ Wn) +
A Deep Understanding of ODConv. In ODConv, for the convolutional kernel Wi: (1) agi assigns different attention scalars to convolutional parameters (per filter) at & x k spatial locations; (2) Ocii assigns different attention scalars to Gin channels of each convolutional filter W (3) a assign- s different attention scalars to Cout convolutional filters; (4) Quri assigns attention scalar to the whole convolutional kernel. Fig. Dillustrates the process of multiplying these four types of atten- tions to n convolutional kernels. In principle, these four types of attentions are complementary to each other, and progressively multiplying them to the convolutional kernel We in the location-wise, channel-wise, witeee filter-wise and kernel-wise orders makes convolution operations be different w.r.t. all spatial locations, all input channels, all filters and all kernels for the input æ, providing perfor- mance guarantee to capture rich context cues. Therefore, ODConv can significantly strengthen the feature extraction ability of basic convolution operations of a CNN. Moreover, ODConv with one single convolutional kernel can compete with or outperform standard CondConv and DyConv, intro- ducing substantially fewer extra parameters to the final models. Extensive experiments are provided to validate these advantages. By comparing Eq. D and Eq. 2 we can clearly see that ODConv is a more generalized dynamic convolution. Morcover, when setting ae1 and aw1 to 1, ODConv with only filter-wise attention 0 will be reduced into: applying an SE variant conditioned on the input features the convolutional filters, then followed by convolution operations (note the original SE (Hu et al. (0188) is conditioned on the output features, and is used recalibrate the output features themselves). Such an SE variant a special case of ODConv.
and all components of a,1,
Implementation. For ODConv, a critical question is how to compute four types of attentions Osis Oeis Offz and Owi for the convolutional kernel W.. Following CondConv and DyConv, we also use a SE-typed attention module (Hu et al. 2018b) but with multiple heads as (,) to compute them, whose structure is shown in Fig. I Specifically, the input x squeezed into feature vector with the length of Cin by channel-wise Global Average Pooling (GAP) operations first. Subsequently, there is a Fully Connected (FC) layer and four head branches. A Rectified Linear Unit (ReL.U) (Krizhevsky et al. 2012) comes after the FC layer. The FC layer maps the squeezed feature vector to lower dimensional space with the reduction ratio r (according to the ablative experiments, we set r = 1/16 in all main experiments, avoiding high model complexity). For four head branches, each has an FC layer with the output size of k x k. Cin x 1, Cout X 1 and n x 1, and a Softmax or Sigmoid function to generate the normalized attentions asis Oici afj and awi, respectively. We adopt the temperature annealing strategy proposed in DyConv to facilitate the training process. For easy implementation, we apply ODConv all convolutional layers except the first layer of each CNN architecture tested
5

---

Published as a conference paper at ICLR 2022
in our main experiments just like DyConv. and share aggi aes and a all convolutional kernels. In the Experiments section, a comparison of the inference speed for different dynamic convolution methods is provided. In the Appendix, further provide computational cost analysis of ODConv and an ablation study of applying ODConv to different layer locations.
4 EXPERIMENTS
In this section, we provide comprehensive experiments on two large-scale image recognition datasets with different CNN architectures to validate the effectiveness of ODConv, compare its performance with many attention based methods, and study the design of ODConv from different aspects.
4.1 IMAGE CLASSIFICATION ON IMAGENET
Our main experiments performed on the ImageNet dataset (Russakovsky al. 2015). It has over 1.2 million images for training and 50,000 images for validation, including 1,000 object classes.
CNN Backbones. We use MobileNetV2 (Sandler et al. 2018) and ResNet (He et al. 2016) families for experiments, covering both light- weight CNN architectures and larger ones. Specifically, we choose ResNet18, ResNet50, ResNet101, and Mo (1.0x, 0.75 0.5x) as the backbones.
Experimental Setup. In the experiments, we consider existing dynamic convolution methods in- cluding CondConv (Yang et al.) 2019), DyConv (Chen al. 2020) and DCD (Li et al. 2021) as the key reference methods for comparisons on all CNN backbones. On the ResNet backbones, we also compare our ODConv with many sfate-of-the-ar methods using attention modules: (1) for out-
put feature recalibration including SE (Hu et al. (0018)), CBAM (Woo et al. 2018) and ECA (Wang) al. 2020), and (2) for convolutional weight modification including CGC (Lin et al. 2020). Weight-
Net (Ma et al. 2020) and WE (Quader et al., 2020). For fair comparisons, we use public codes of these methods unless otherwise stated, and adopt the popular training and test settings used in the
community for implementing the experiments. The models trained by all methods use the same set- tings including the batch size, the number of training epochs, the learning rate schedule, the weight decay, the momentum and the data processing pipeline. Moreover, do not use advanced training tricks such as mixup (Zhang et al. (2018a) and label smoothing (Szegedy et al.) (2016), aiming to have clean performance comparisons. Experimental details are described in the Appendix.
Table 1: Results comparison on the ImageNet validation set with the MobileNetV2 (1.0x, 0.75x, 0.5x) backbones trained for 150 epochs. For our ODConv, we set 1/16. Best results are bolded.
Models
Params
MAdds
Top-1 Acc (%)
Top-3 (%3)
MobileNetV2 (LOx)
3.30M
300.M
71.65
90.22
+ CondConv (8x)
22.88M
318.1M
74.13 (12.48)
91.67 (11.45)
DyConv (4x)
1240M
317.1M
74.94 (13.29)
91.83 (11.6))
DCD
5.72M
318.4M
18 (12.53)
91.72 (71.50)
ODCom (1x)
4.94M
311 RM
74.84 (13.19)
92.13 (11.91)
ODConv (4x)
11.52M
327. IM
75.42 (13.77)
(21.9)
MobileNetV2 (0.75x)
2.64M
209. LM
69.18
88.82
+ CondConv (8x)
17.51M
223.9M 9M
71.79 (12.61)
90.17
DyCoav (1x)
7.95M
220.1M IM
72.75 (13.57)
90.93 (12.
DCD
4.08M
222.9M
71.92 (12.74)
90.20 (8))
ODCoav (1x)
3.51M
217.1M
72.43 (13.25)
(12.00)
ODConv (4x)
7.50M
226.3M
73.81 (14.63)
91.33 (*2.51)
MobileNetV2 (0.5x)
2.00M
97.IM
64,30
85.21
CondConn (8x)
13.61M
110.0M
67.24 (12.94)
87.51 (?2.30)
DyCoav (1x)
4.57M
69.05 (14.75)
88.37 (13.16)
DCD
3.06M
105.6M 6M
69.32 (15.02)
88.44 (?3.23)
ODCoav (1x)
2.43M
101 8M
68. 26 (13.96)
87. (12.77)
+ ODCony (4x)
4.44M
106.4M
70.01 (15.71)
89.01 (?3.80)
Results Comparison on MobileNets. Table shows the results comparison on the MobileNetV2 (1.0x, 0.75x, 0.5x) backbones. As CondConv and DyConv are primarily proposed to improve the performance of efficient CNNs, they all bring promising top-1 gains to the light-weight Mo- bileNetV2 (1.0x, 0.75x, 0.5x) backbones. Comparatively, our ODConv (1x) with one single convolutional kernel performs better than CondConv (8x) with 8 convolutional kernels, and its per- formance is also on par with DyConv (4x) with 4 convolutional kernels. Note that these competitive results of ODConv (1x) obtained with significantly lower numbers of extra parameters, validat- ing that our ODConv can strike a better tradeoff between model accuracy and size. Besides, ODConv (1 x) also performs better than DCD in most cases. ODConv (4x) always achieves the best results

---

Published as a conference paper at ICLR 2022
on all backbones. The results comparison with an increase number of training epochs (from 150 to 300) can be found in the Appendix, from which we can observe similar performance trends.
Results Comparison ResNets. Table 2 shows the results comparison on the ResNet18 and ResNet50 backbones which are much larger than the MobileNetV2 backbones. We can obtain the following observations: (1) On the ResNet18 backbone, dynamic convolution methods (CondConv, DyCony, DCD and our ODConv) and convolutional weight modification methods (CGC, WeightNet and WE) mostly show better performance than output feature recalibration methods (SE, CBAM and ECA), although they all use attention mechanisms. Comparatively, our ODConv (1x) with one single convolutional kernel outperforms the other methods both in model accuracy and size, bringing 2.85% top-1 gain to the baseline model. ODConv (4x) gets the best results, yielding top-1 gain of 3.72%: (2) However, on the larger ResNet50 backbone, CondConv, DyConv and DCD show worse results than most of the other methods even though they have significantly increased numbers of parameters. Due to the increased parameter redundancy, adding more parameters to larger networks tends to be less effective in improving model accuracy, compared to small networks. Thanks to the proposed multi-dimensional attention mechanism, ODConv can address this problem better, achieving superior performance both in model accuracy and size for larger backbones well as fight-weight ones. To further validate the performance of ODConv on very deep and large networks, apply ODConv to the ResNer101 backbone, and the results comparison is provided in Table Again, ODCony shows very promising results, yielding 1.57% top-1 gain with ODConv (1x).
Table 2: Results comparison the ImageNet validation set with the ResNet18 and ResNet50 back- bones trained for 100 epochs. For our ODConv, we set r = 1/16. denotes the results are from the paper of WE Quader et al. 2020) its code is not publicly available. Best results are bolded.
Notwork
ResNet18
ResNer50
Models
Params
MAdds
Top-1 (30)
Top
Params
MAdds
Top-1 Ace (8)
Top
Baseline
11.99
L814G
10.25
89.38
25.36M
38888
76.21
93.01
CondConv (8x)
81.35M
L84G
71.99
(11.74
90.27 (10.89)
129 3GM
3.978G
(50.2 (60.7)
DyConn (1x)
45.47M
1861G
72.76
(12.51)
90.79 (f1.41)
100.88M
3,965G
76.82
(70.59)
93.16 (10.15)
DCCD
14.70M
1.84G
72.33 (12.08)
90.65 (f1))7)
84M
3,944G
76.92
(10.69)
93.46 (10.45)
-ODCorv (1x)
11,94M
L883G
73.10 (12.85)
91.10 (11.72)
28.64M
3,916G
77.96 (11.73)
93.84 (10.83)
-ODConv (4x)
44,90M
19166
73.97 (13.72)
91.35 (11.97)
90.67M
(12.29)
94.01 (11.00
SE
1L78M
L816G
70.98 (10.73)
90.03 (10.63)
28.07M
3.872G
77.31
93.63 (70.62)
CRAM
11.78M
1.818
71.01 (10.76)
85 (10.47)
28.07M
3,886G
77.46 (11.23)
93.59 (10. 58)
ECA
11.69M
1886G
70.60 (10.35)
89.68 (10.30)
25.56M
3,870G
77.34
93.64 (10.63)
CGC
11.69M
L.27G
71.60 (F1.35)
90.35 (10.97)
25.39M
3877G
(50.50)
93.37 (10.36)
WeightNee
11.93M
18866
71.56 (11.31)
90.38 (11.00)
30.38M
3,885G
77.51 (11 28)
93. 69 (10.68)
WE (*)
11.90M
1.820G
71.00 (10.75)
90.00 (10.62)
10M
3.860G
770.0
93.50 (10.49)
Table 3: Results comparison on the ImageNet validation set with the ResNet101 backbone trained for 100 epochs. For our ODConv, set
1/16. Best results are bolded.
Models
Params
MAdds
Top-1 Aec (%)
Top-5 Acc (5)
ResNet101
44.35M
7.370G
77.41
93.67
+ SE
49.29M
7.593G
78.42 (11.01)
94.15 (10.48)
CBAM
49.30M
7.617G
78.50 (1109)
94.20 (10.53)
ECA
44.55M
7.590G
78.60 (11.19)
94.34 (10.67)
+ ODCony (Ix)
50.82M
7.675G
78.98 (11.57)
94.38 (10.71)
+ ODConv (2x)
90.44M
7.802G
79.27 (11.6)
94.47 (10.80)
4.2 OBJECT DETECTION ON MS-COCO
Given a backbone model pre-trained on the ImageNet classification dataset, a critical question is whether the performance improvement by ODConv can be transferred to downstream tasks or not. To explore it, we next perform comparative experiments on the object detection track of MS-COCO dataset (Lin et al. 2014). The 2017 version of MS-COCO dataset contains 1 118,000 training images and 5,000 validation images with 80 object classes.
Experimental Setup. We use the popular MMDetection toolbox (Chen et al., 2019) for experiments with the pre-trained ResNet50 and MobileNetV2 (1.0x) ) models as the backbones for the detector. We select the mainstream Faster R-CNN (Ren et al., 2015) and Mask R-CNN (He et al., 2017) detectors with Feature Pyramid Networks (FPNs) (Lin al. 2017) the necks to build the basic object detection systems. For a neat comparison, the convolutional layers in the FPN necks still use regular convolutions, and maintain the same data preparation pipeline and hyperparameter
7

---

Published as a conference paper at ICLR 2022
Table 4: Results comparison on the MS-COCO 2017 validation set. Regarding Params or MAdds, the number in the bracket is for the pre-trained backbone models excluding the last fully connected layer, which is almost the same to that shown in Table [ and Table while the other number is for the whole object detector. Best results are bolded.
Rackhane Mixaos
Dotertors
AP)25
APheX]]
APSUNT
AP
APEUN
Params
MAGS
ResNet50
372
57.8
214
21.3
2015
43.0
23 (TM)
X0TAFRI (T6)
CondConn (8x)
38.1
38.9
41.5
22.4
42.1
48.7
133.75M (113.46M)
207,080 (76. S1C)
DyCan (4x)
416
119
DCD
49.5
+ ODDCow (1x]
39.0
00.5
42.3
50.5
46 (20.590)
(76.61G)
+ OLXCovv H4X)
Fister R-CNN
39.2
60.7
42.6
23.1
42.6
51.0
108.9M (SS 6263
207.42G (76.85))
Mohile (L00)
S1.1
51.1
55.1
17.4
15.5
41.2
21.TM (2 (2,225))
122, 122,590 45C)
+ CondClony (8X)
33.7
54.9
35.6
19.3
36.4
43.7
31 5.M (12.63))
122 59G (24, (24466)
DDyCom (4x)
34.5
55.6
36.5
19.8
37.3
44.7
30 50.02M (11. (11.2M)
125,010 (24 88C)
DCD
33.3
53.0
35.1
43.2
23 (4.44M)
125,01G (24
ODDCow
44.5
(24
ODCCow (4X)
35.1
56.7
37.0
45.2
22 NHL (10
125/125 (24 89G)
ResNec50
38.0
38.6
41.5
21.6
41.5
49.2
20 20-- 225 1MM
2601 (20,30))
+ CondClow (8x)
38.8
59.1
42.3
22.5
42.5
50.1
18 SANE (13 460N)
7640 1551 (76, SIC)
+ DyCon (4x)
39.2
60.3
42.5
23.0
42.9
51.4
121 77M 19848883300 (363)
260 300 (76, 66C)
DCD
38.8
39,8
42.2
23.1
42.7
49.8
90 73M (27 P9M1
260 250 (76 63C)
OLCCoe
39.9
61.2
43.5
52.3
29 SOM (26-5980)
260.25G (76,61))
M (100)
ODC
Mask R-CNN
52.1
18A4
52.3
2X 2IN0
(OM))
175.6K (2445G)
+ CondConv (8x)
34.4
53.4
36.6
19.8
36.9
44.6
34 19M (12.63M)
175.67G (24 460)
+ DyCan (4x)
35.2
56.2
37.5
20.7
38.0
45.5
32 ASM (11 120
126.0%G (24 88G)
+ DCD
34.3
54.9
36.6
20.6
37.1
44.8
26 00M (1.4MD)
176.09 (24 88C)
4 OIDCoe (3X)
35.0
56.1
37.3
19.9
37.7
46.2
25 22M (3.66M)
176,0863 (24 87C)
ODCon (4x)
35.8
$7.0
38.1
20.5
38.5
45.9
31 3OM (10.24M)
176, 100 (24))
settings for all pre-trained models built with CondConv, DyConv and our ODConv, respectively. Experimental details are described in the Appendix.
Results Comparison. From the results shown in Table 4 we can observe similar performance improvement trends as on the ImageNet dataset. For Faster R-CNN R-CNN with the pre- trained ResNet50 backbone models, CondConv (8x) and DyConv (4x) show an AP improvement of 00.9]...%% and 1.1%|1.22 to the baseline model respectively, while our method performs much better, e.g., ODConv (1x) with one single convolutional kernel even shows AP improvement of 1.8%, respectively. With the pre-trained MobileNetV2 (1.0x) backbone models, our ODConv (1x) performs obviously better than CondConv (8x), and its performance is on par with that of DyConv (4x) as their AP gap is only 0.2% for both two detectors, striking better model accuracy and efficiency tradeoff. Similar boosts AP scores for small, medium and large objects are also obtained by three methods on both detectors. ODConv (4x) always achieves the best AP scores.
4.3 ABLATION STUDIES
Finally, we conduct lot of ablative experiments on the ImageNet dataset, in order to have better analysis of our ODConv.
Table 5: Results comparison of the ResNet18 models based on ODConv with different settings of the reduction ratio All models are trained on the ImageNet dataset. Best results are bolded.
Midels ResNet18
Params
MAdas
Top-1 Acc (%)
Top-5 Acc (%)
11.09M
1814G
70.25
89.38
1/4
12.58M
1839G
73.41
91.29
ODCony
12.15M
1838G
73.11
91.10
1/16
11,94M
1838G
73.10
91.10
Table 6: Results comparison of the ResNet18 models based on ODConv with different numbers of convolutional kernels n. All models are trained on the ImageNet datasct. Best results are bolded.
Models
Pararas
MAdds
Acc (%)
Top-5 Ace (%)
ResNet18
11.69M
1814G
70.25
89.38
1x
11.44M
L838G
73.10
91.10
2x
22.93M
1.822
ODCom 1/10)
3x
33,92M
1894G
73.77
91.35
4x
44.90M
1.966G
73.97
91.35
88.84M
2,0063
74.08
91.44
Reduction Ratio Selection. Our first set of ablative experiments is for the selection the reduction ratio used in the attention structure (z)). From the results shown in Table we can find that under r r= 1/4, r = 1/8 and r = 1/16, ODConv consistently obtains large performance improvements to the baseline ResNet18 model (2.85~33.16) 16% top-1 gain), and the extra MAdds negligible. Com- paratively, ODConv with r = 1/16 strikes the best tradeoff between model accuracy and efficiency, and thus we choose it as default setting.
8

---

Published as a conference paper at ICLR 2022
Table 7: Investigating the complementarity of four types of attentions proposed in ODConv. In the experiments, when (kwi is used, we set v 4, and otherwise n. For the optimal comparison, we use the best r setting reported in Table S Best results are bolded.
All ResNet18 8 models are trained on the ImageNet dataset.
Models
Oec
Mut
Params
MAdds
Top-1 Acc (%)
Top-5 Ace (%)
ResNer18
11.6M
L,844
70.25
89.38
11.88
1.827G
72.42
90.76
V
12.26M
1,827G
72.07
90.59
1.827G
90.43
4
12.27M
1.827G
73.13
91.14
+ ODConv (r- 1/4)
12.29M
1,827G
72.80
90.99
12.57M
1,829G
72.20
90.67
12.58M
1.8399
73.41
91.29
1.933G
91.53
Table 8: Comparison of the inference speed (frames per second) for different dynamic convolution methods. All pre-trained models are tested on an NVIDIA TITAN X GPU (with batch size 200) and a single core of Intel E5-2683 v3 CPU (with batch size 1) separately, and the input image size is 224 x 224 pixels.
ResNet50
Speed GPU
Speed CPU
MobileNetV
Spead on GPU
Speed CPU
Baseline model
652.2
6.4
Baseline model
1452.3
17.3
CondConv
425.1
4.0
CondConv
1076.9
14.8
DyConv
326.6
3.8
DyConv
918.2
11.9
DCD
400.0
3,6
DCD
875.7
9.5
ODConn (1x)
293.9
3.9
ODDCon (1x)
1029.2
12.8
ODCouv (4x)
152.7
2.5
ODCoav (4x)
608.3
11.2
Convolutional Kernel Number. Accordingly to the main experiments discussed beforehand, OD- Conv strike better tradeoff between model accuracy and size, compared to existing dynamic convolution methods. For better understanding of this advantage, we next perform the second set of ablative experiments the ImageNet dataset, training the ResNet18 backbone based ODConv with different settings of the convolutional kernel number n. Table 6 shows the results. It can be seen that ODConv with one single convolutional kernel brings near 3.0% top-1 gain to the ResNet18 baseline, and the gain tends to be saturated when the number of convolutional kernels is set to 8.
Four Types of Attentions. Note that our ODConv has four types of convolutional kernel attentions Qsi. Oees f; and Oud computed along all four dimensions of the kernel space, respectively. Then, we perform another set of ablative experiments on the ResNet18 backbone with different combinations of them to investigate their dependencies. Results are summarized in Table 7 from which the strong complementarity of four types of attentions asss Oceis fft and Oses can clearly observed.
Inference Speed. Besides the size and the MAdds for CNN model, the inference speed at runtime very important in the practical model deployment. Table provides a comparison of the inference speed for different dynamic convolution methods (both on GPU and CPU). It can be seen that the models trained with our ODConv (1x) are faster than the counterparts trained with DyConv and DCD CPU. Comparatively, the models trained with CondConv show the fastest run-time speed both on a GPU and a CPU, this is because CondConv is merely added to the final several blocks and the last fully connected layer discussed in the Method section. When adding ODConv (1 x)) the same layer locations as for CondCony, we can obtain more accurate and efficient models.
More Ablations. In the Appendix. we provide more ablative experiments to study: (1) the effects of applying ODConv to different layer locations; (2) the importance of the temperature annealing strategy; (3) the choices of the activation functions for m=( (4) the influence of the attention sharing strategy; (5) the stability of the model training process; (6) other potentials of ODConv.
5 CONCLUSION
In this paper, we present new dynamic convolution design called Omni-dimensional Dynamic Convolution (ODConv) to promote the representation power of deep CNNs. ODConV leverages a multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions the kernel space in a parallel manner, and progressively applying these attentions to the corresponding convolutional kernels can substantially strengthen the feature extrac- tion ability of basic convolution operations of CNN. Experimental results on the ImageNet and MS-COCO datasets with various prevailing CNN architectures validate its superior performance.

---

Published as a conference paper at ICLR 2022
REFERENCES
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan NovDollák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. Kernel-predictie convolutional networks for denois- ing monte carlo renderings. In Siggraph, 2017.
Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++ Improved visual explanations for deep convolutional networks. In WACV, 2018.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: /906v.:00000000 0tt///900..7755 07150, 2019.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In CVPR, 2020.
Ali Diba, Vivek Sharma, Lue Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In ICCV, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014.
David Ha, Andrew M. Dai, and Quoc V. Le. Hyperneteworks. In ICLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chinh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In ICCV, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint wwxiiv1704..4240 2017.
Jie Hu, Li Shen, Samuel Albanic, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In NeurIPS, 2018a.
Jie Hu, Li Shen, and Gang Sun. ggueezz-and-exciing
networks. In CVPR, 2018b.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NIPS, 2015.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue Van Gool. Dynamic filter networks. In NIPS, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
HyunJae Lec, Hyo-Eun Kim, and Hyeonscob Nam. Srm: A style-based recalibration module for convolutional neural networks. In ICCV, 2019.
Xiang Li. Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, 2019.
Yunsheng Li. Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Ye Yu, Yuan Lu, Zicheng Liu, Mei Chen, and Nuno Vasconcelos. Revisiting dynamic convolution via matrix de- composition. In ICLR, 2021.
10

---

Published as a conference paper at ICLR 2022
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017.
Xudong Lin, Lin Ma, Wei Liu, and Shih-Fu Chang. Context-gated convolution. In ECCV, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Ningning Ma, Xiangyu Zhang, Hai-Tar Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018.
Ningning Ma, Xiangyu Zhang, Jiawei Huang, and Jian Sun. Weightnet: Revisiting the design space of weight networks. In ECCV, 2020.
Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll. Burst denoising with kernel prediction networks. In CVPR, 2018.
Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou. Rotate to attend: Con- volutional triplet attention module. In WACV, 2021.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, 2017.
Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So Kweon. Bam: Bottleneck attention module. BMVC, 2018
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Niamul Quader, Md Mafijul Islam Bhuiyan. Juwei Lu, Peng Dai, and Wei Li. Weight excitation: Built-in attention mechanisms convolutional neural networks. In ECCV, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Christian Szegedy. Vincent Vanhoucke. Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. CVPR, 2017.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. In CVPR, 2020.
Sanghyun Woo, Jongchan Park, Loon-Youny Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.

---

Published as a conference paper at ICLR 2022
Brandon Yang, Gabriel Bender. Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. NewrIPS, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018a.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018b.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
In this section, we describe the supplementary materials including: (1) a computational cost anal- ysis of ODConv; (2) implementation details for the experiments the ImageNet and MS-COCO datasets; (3) more ablative experiments on the ImageNet dataset; (4) illustrative training and valida- tion curves compare the stability of the model training process with different dynamic convolution methods; (5) more experiments to study other potentials of ODConv.
A.1 COMPUTATIONAL COST OF ODCONV
For convolutional layer with ODConv, its computational cost can be easily calculated according to the design and the multiplication process of four types of attentions shown in Fig. and Fig. 2 re- spectively. Specifically, following the notations defined in the Method section of the main paper, the extra MAdds of ODConv (1x) over regular convolution (which has hwk? Cout MAdds, without consideration of the bias term) can be calculated as
Cin (2Cin + Cout + k2) + k22 k2cnn(11 (1 2Cout).
M Adds hwcc,
(3)
For ODConv (nx), the extra MAdds over regular convolution can be calculated as
M Addss huwcin Cin (2cin Cout + k2 + n) k- (1 Cout + 2raCout).
(4)
r
Compared to hwnk? h Cout MAdds for regular convolution, the extra MAdds by ODConv are small.
A.2 EXPERIMENTAL DETAILS
Experimental Details ImageNet. Recall that we use MobileNetV2 (Sandler et al., 2018) and ResNet (He et al. 2016) families for experiments on the ImageNet dataset, covering both light- weight CNN architectures and larger ones. For fair comparisons, we adopt the popular training settings used in the community to train the respective backbone models with all methods. Specifi- cally, for R8 ResNet50 and ResNet101, all models trained with SGD for 100 epochs. We set the batch size as 256, the weight decay as 0.0001 and the momentum as 0.9. The learning rate starts at 0.1, and is divided by 10 every 30 epochs. Following DyConv (Chen al. 2020), for our ODConv, we also use dropout rate of 0.1 for ResNet18. We use dropout rate of 0.2 for ResNet50 and ResNet101 for our ODConv. For MobileNetV2 (1.0x, 0. 0.75 0.5x), all models are trained with SGD for 150 epochs (we also have the experiments for training all models with 300 epochs in this Appendix). We set the batch size as 256, the weight decay as 0.00004 and the momentum as 0.9. The learning rate starts at 0.05, and is scheduled to arrive at zero within single cosine cycle. Following DyConv, for our ODConv, we also use dropout rate 0.2 for MobileNetV2 (1.0x) and dropout rate of (0.1, 0) for MobileNetV2 (0.75x, 0.5x). Regarding the temperature annealing strategy used for DyConv and ODConv, the temperature reduces from 30 to linearly in the first 10 epochs for all models. All experiments are performed the servers having GPUs. We follow the standard protocols to train and evaluate each CNN backbone. For training, images are resized to 256 x 256 first, and then 224 224 crops are randomly sampled from the resized images or their horizontal flips normalized with the per-channel mean and standard deviation values. For evaluation, we report 1op-1 and top-5 recognition rates using the center image crops.
Experimental Details on MS-COCO. Recall that we use the popular MMDetection toolbox (Chen al. 2019) for experiments on the MS-COCO dataset with the pre-trained ResNet50 and Mo- bileNetV2 (1.0x) models as the backbones for the detector. We select the mainstream Faster R- CNN (Ren et al. 2015) and Mask R-CNN (He et al.. 2017) detectors with Feature Pyramid Net- works (FPNs) (Lin et al., (017) as the necks to build the basic object detection systems. For neat and fair comparison, the convolutional layers in the FPN necks still use regular convolutions, and we maintain the same data preparation pipeline and hyperparameter settings for all pre-trained models built with CondConv. DyConv and our ODConv, respectively. All experiments are performed on server having 8 GPUs with mini-batch size of 2 images per GPU. We finetune these detectors on the MS-COCO training set following the 1x learning rate schedule, which indicates a total of 12 epochs with the learning rate divided by 10 at the 8t epoch and the 1hth epoch, respectively.
13

---

Published as a conference paper at ICLR 2022
Following DyConv, the temperature annealing strategy is not used in the downstream experiments on the MS-COCO dataset, as it is prone to worse results. In the validation, we report the standard Average Precision (AP) under IOÚ thresholds ranging from 0.5 to 0.95 with an increment of 0.05. We also keep AP scores for small, medium and large objects.
A.3 MORE ABLATIVE EXPERIMENTS ON IMAGENET
Besides the ablative experiments described in the main paper, here we provide more ablative exper- iments, for a better understanding of our ODConv.
Effects of Applying ODConv to Different Layer Locations. In the Experiments section of the main paper, we show that applying dynamic convolution to less layers leads to a faster runtime speed compared applying it more layers. For easy implementation, we follow DyConv and apply ODConv to all convolutional layers except the first layer of each CNN architecture tested in our main experiments. However, it is necessary to study the effects of applying ODConv to different layer locations. The performance comparison of adding ODConv to different layers of the MobileNetV2 (0.5x) backbone is given in Table 9 showing that our default setting (i.e.. adding ODConv to all convolutional layers except the first layer) has the largest gain. This indicates that adding ODConv to more convolutional layers tends to have a model with higher recognition accuracy.
Table 9: Results comparison of adding ODConv to different layers of the MobileNetV2 (0.5x) backbone. All models are trained on the ImageNet dataset for 150 epochs, and we set 1/16 and n = 1. For the inference speed, all pre-trained models are tested on an NVIDIA TITAN X GPU (with batch size 200) and single core of Intel E5-268 v3 CPU (with batch size 1) separately, and the input image size is 224 x 224 pixels. Best results are bolded.
Models
Params
MAdds
Top-I Acc (%)
Top-5 Ace
Speed GPU
Speed CPU
MobileNetV2 (0.3K)
2.00M
97.1M
64.30
85.21
2810.9
30.1
OLXCor (to all comv layers the lirst layer)
2.43M
101 8M
68,26
87.98
18374
18.3
ODCoav
conv layers)
2M
2083.3
27.1
1 ODCon (to all 3x33 conv layers)
2.17M
99. IM
67.54
87.55
2064.9
27.4
OLXOn last 6 residual blocks)
2.31M
99. 1M
67.29
87,39
1879,5
19,6
Importance of Temperature Annealing Strategy. Following DyConv, we also use the temperature annealing strategy to facilitate the model training process of our ODConv. To study its effect, we perform of ablative experiments on the ImageNet dataset, training the ResNet18 backbone based on ODConv with or without using the temperature annealing strategy. From the results shown in Table [0] we can find that the temperature annealing strategy really important, which brings 1.06%| 1 .04% top-1 improvement to the ResNet18 model based on ODConv ODCon (4x).
Table 10: Analyzing the effect of the temperature annealing when training R8 with ODConv. All models are trained on the ImageNet dataset, and we set r 1/4. Best results are bolded.
Models
Temperature Annealing
Top-1 Acc (25
Top-5 Acc (7:)
ReaNet18
70.25
89.38
73.41
91.29
ODConn (1x)
72.35
90.65
+ ODConv (1x)
74.33
91.53
73,29
90.95
Choices of the Activation Functions. Recall that in the structure of (T( for our ODConv, we use two different activation functions (either a Sigmoid function or a Softmax function) to compute four types of attentions Ogss C'eis O'ys and aws- It is also critical compare the performance of different activation functions for ODConv. Since the activation function choices for the channel dimension and the convolutional kernel dimension have been thoroughly discussed in the papers of SE and DyConv respectively, we adopt their suggestions for three related attentions defined in our ODConv. Here we perform another set of ablative experiments, focusing on the choices of the activation function for the spatial dimension. With the ImageNet dataset, use R 8 the test backbone network to explore the effects of different activation functions for computing the attention of ODConv along the spatial dimension of the kernel space. Table 11 summarizes the results, from which we can see that the Sigmoid function performs better than the Softmax function, and thus we use the Sigmoid function to compute the attention scalars along the spatial dimension for ODConv.
A Attention Sharing Strategy, As we discussed in the Method section, we share three attentions agis Oai and oft to all convolutional kernels for the easy implementation as well as for more efficient
14

---

Published as a conference paper at ICLR 2022
Table 11: Results comparison of ODConv with different activation functions. All models are trained on the ImageNet dataset, and we set 1/4. Best results are bolded.
Models
Activation Function
Top- Acc
Top-5 Acc (%)
ResNet18
70.25
89.38
Sigmoid
73.41
91.29
ODConn (1x)
Softmax
73.23
Sigmoid
74.33
91.53
ODDoony (4x)
Softmax
73.97
91.50
Table 12: Results comparison of ODConv with or without using the attention sharing strategy. All models trained the ImageNet dataset. Best results bolded.
Models
Params
MAdds
Top-1 1 Ace (%)
Top-5 Acc (%)
ResNet18
11.69M
1.814G
70.25
89.38
+ ODConv w/ attention sharing (r 1/16, 4x)
44.90M
1.916G
73.97
91.35
+ ODConv w/o attention sharing (r = 1/16, 4x)
45.43M
1.950G
74.16
91.47
training. Actually, training separate O'si. Otei and O'ys for each additive convolutional kernel will bring further improved model accuracy, as can be seen from the ablative results in Table 12] That is, when preferring to get more accurate models, the attention sharing strategy would be removed.
Training MobileNetV2 Backbones for 300 Epochs. Recall that in the Experiments section of the main paper, we compare our ODConv with CondConv, DyConv and DCD on MobileNetV2 (1.0x, 0.75x, 0.5x), following the popular training settings used in the community, where all models are trained for 150 epochs. To validate the effectiveness of our ODConv further, we also conduct another set of experiments using an increased number of training epochs (300 instead of 150) as adopted in DyConv (Chen et al., 2020). All the other training settings remain the same to those for the experiments described in the main paper. For a clean comparison of all methods, we do not use advanced training tricks such as mixup (Zhang et al., 2018a 8a) and label smoothing (Szegedy et al. 2016). Results are summarized in Table 13 where performance gains show a similar trend as those in [ for the model training with 150 epochs. Results in Fig. B further show that our ODConv gets a better trade-off between model accuracy and size even on the light-weight MobileNetV2 backbones compared CondConv and DyConv.
Results comparison on the ImageNet validation set with MobileNetV2 (1.0x, 0.75x, 0.5x) x) as the backbones. All models are trained for 300 epochs. Best results are bolded.
Table 13:
Models
Params
MAdds
Top-1 Acc (%)
MobileNetV2 (1.0x)
3.50M
300.8M
72,07
+ CondConv (8x)
22.88M
318.1M
74.31(+2.24)
DyConv (4x)
12.40M
317.1M
75.08(13.01)
DCD
5.72M
318.4M
74.48(12.41)
ODConv (1x)
4.94M
311.8M
75.13(13.06)
+ ODConv (4x)
11.52M
327.1M
33.61)
MohileNerV2 (0.75x)
2.64M
209.
69.76
+ CondConv (8x)
17.51M
223 9M
72. 72.18(12.42)
+ DyCony (1x)
7.95M
220.1M
73 38(13.2))
DCD
4.08M
222.9M
(22.3))2..63)
ODCon (1x)
3.51M
M17.IM
73.03(13.27)
ODConv (4x)
7.50M
226.3M
74.45(14.69)
MobileNeiV2 (0.5x)
2.00M
97.1M
65.10
CondConv (8x)
13.61M
110.0M
68. (8.45(13.35)
+ DyConv (4x)
4.57M
103.2M
69 19.83((4.77)
+ DCD
3.06M
105.6M
70.08(14.998
+ ODCony (1x)
2.43M
101.8M
69. 16(64.06)
+ ODConv (4x)
4.44M
106.4M
(5.77)
A.4 IILUSTRATION OF MODEL TRAINING AND VALIDATION CURVES
Fig. illustrates the training and validation accuracy curves of the ResNet18 models trained on the ImageNet dataset with CondConv, DyConv, ODConv (1x) and ODConv (4x), respectively. We can see that our ODConv shows consistent high top-1 gains throughout the training process compared to the other three dynamic convolution counterparts.
15

---

Published as a conference paper at ICLR 2022
76
74
X1.0
Tap-1 Accuracy (%)
72
x0:75
70
68
x0.5
66
MobileNetV2
MobileNetV2 ODConv (4x)
MobileNetV2 CondConv (8x)
64
MobileNetV2 DyConv (4x)
0
5
10
15
20
25
Parameters (Millions)
Figure 3: Comparison of model accuracy and size for the pre-trained MobileNetV2 models based on different dynamic convolution methods. All models are trained for 150 epochs on the ImageNet dataset. It can be seen that our ODConv (1 x) makes better accuracy and size tradeoff for the light- weight MobileNetV2 backbones compared to CondConv and DyConv. On the larger ResNet18 8 and ResNet50 backhones,
better results are obtained by ODConv, can be seen from the results shown in Table of the main paper.
80
75
Top-1 Acturacy (96
60
ResNet18
55
ResNel18 val
CondConv (8x) train
50
CondConw (Bx) val
DyConv (4x) train
DyConv (4x) val
ODConw (1x)
35
ODConv (4x) train
ODConv (4x) val
30
0
20
40
60
80
100
Epoch
Figure 4: Curves of top-1 training accuracy (dashed line) and validation accuracy (solid line) of the ResNet18 models trained on the ImageNet dataset with CondConv (8x), DyConv (4x), our ODConv (1x) and ODConv (4x), respectively. Comparatively, both CondConv and DyConv yet has only 14.68% 26.26% parameters of the model trained with CondCony Dono.. Our ODConv (4x) converges with the best validation accuracy, which outper- forms CondCony DyConv by 1.98% 1.21% top-1 gain with fewer parameters.
ODConv (1x) outperforms
A.5 MORE EXPERIMENTS FOR STUDYING OTHER POTENTIALS OF ODCONV
In this section, we provide a lot of extra experiments conducted for studying other potentials of ODConv.
16

---

Published as a conference paper at ICLR 2022
Performance Comparison A CNN Backbone Added with the SE Module. Recall that SE (Hu ct al. 2018b) performs attention on the output features of a convolutional layer, while our ODConv performs attention on the convolutional kernels, and thus they are likely to be complimentary. To validate this, we perform another set of experiments on the ImageNet dataset with ResNet18 back- bone. In the experiments, we use the ResNet 8 variant reported in Table as the backhone, and train this backbone with CondConv, DyCony, DCD and our ODConv separately, adopting the same training settings used for the 8 backbone. Results are shown in Table [4] It can be seen that our ODConv brings significantly large performance improvement to the ResNet18 backbone already incorporating the SE module, showing its great combination potential.
Table 14: Results comparison on the ImageNet validation set with the BesNetl8+SS backbone trained for 100 epochs. For our ODConv, we set r 1/16. Best results are bolded.
Models
Params
MAdds
Top-1 Acc (%)
Top-3 Ace (%)
ResNet18
11.69M
18814
70.25
89,38
SE
11.78G
18866
70.98 (10.73)
90.03 (10.65)
SE & ODConv(4x)
44.99M
19988G
74.04 (13.79)
91.38 (T2.00)
Performance Boost Using Heavy Data Augmentations and a Longer Training Schedule. Recall that in the main paper, we do not use advanced training tricks such as mixup (Zhang et al. 2018a) and label smoothing (Szegedy et al., 2016), aiming to have clean performance comparisons. To better explore the potential of our ODConv, it would be necessary to have a study about whether ODConv can also work well when using heavy data augmentations and longer training schedule. To this end, we perform a set of experiments on the ImageNet dataset with the ResNet1 8 back- bone trained with ODConv (4x). In the experiments, we set = 1/16 for our method, first train ResNet18 with label smoothing, mixup and (label +mixx)) separately for 100 epochs (the other training settings are the same to those used for Table 2 and then train ResNet18 with (label smoothing+mixun for 120 epochs instead of 100 epochs. Detailed results are summarized in Ta- [I] It can be seen that our ODConv works well when using aggressive data augmentations and a longer training schedule, showing further improved performance.
Table 15: Results comparison of the ResNet18 models based on ODConv with aggressive data augmentations and a longer training schedule. All models are trained on the ImageNet dataset. Best results are bolded.
Models
Top-1 Ace (%)
Top-5 Acc (%)
ResNet18 ODConv (r 1/16, 4x)
73.97
91.35
+ Label Smoothing
74.15 (f0.18)
91.42 (f0.07)
+ Mixup
74.05 (10.08)
91.63 (10.28)
+ Label Smoothing & Mixup
74.18 (10.21)
91.68 (10.33)
+ Label Smoothing & Mixup & Longer Training Schedule
74.59 (10.62)
91.74 (10.39)
Feature Pooling Strategy. Note that when computing four types of attentions in our ODConv, the inputs features are always globally averaged, reducing their spatial dimension to one pixel. Accord- ing to the first three results of Table 7 in the main paper, the spatial convolutional kernel attention as brings the largest gain to the baseline model compared to the other two attentions (Orgi and a computed along the dimension of the input channels and the output channels, respectively). Intu- itively, at the first glance, it may easily lead to a counterintuitive feeling as the input features used to predict ass (as well as the other three attentions) do not include spatial information after the global average pooling. However, it should be noted that our ODConv uses the attentions generated from the input features modify the convolutional kernels. which is quite different from popular self- calibration attention mechanisms (c.g., SE (Hu et al. (2018b) and CBAM (Woo et al.] 2018) use the channel attentions generated from the output features of convolutional layer to recalibrate the output features themselves). As our ODConv is not a self-calibration attention module, reducing the spatial dimension of the input features to one pixel (but the length of the reduced vector is large enough) will not affect its promising performance. To validate this, we conduct another experiment on the ImageNet dataset with the ResNet18 backbone. In the experiment, for Oigis we pool the input features to have size of 3x3 instead of 1x 1, and show the results in Table 16 It can be seen that preserving more spatial information only brings 0.06%|0.00 0..0%% gain to top- top-5 accuracy, which proves the effectiveness of our current design large degree.
17

---

Published as a conference paper at ICLR 2022
Table 16: Comparison of the feature pooling strategy in ODConv using different spatial sizes for the reduced features. All R models are trained on the ImageNet dataset. Best results are bolded.
Models
Params
MAdds
Top-1 Ace (%)
Top-3 Acc (%)
ResNer18
T1.69M
L814G
70.25
89.38
ODConv 1/4, 1x, pooling l)
12.58M
1.839G
73.41 (13.16)
91.29 (119)
ODConv (r= 1/4, 1x, pooling 3x3)
14.85M
1.842G
73.47 (73.22)
91.31 (11.93)
Comparison of Training Cost. Recall that in the main paper, we provide comparison of the inference speed for different dynamic convolution methods. Here, we further perform set of ex- periments to compare the training cost of our ODConv, CondConv, DyConv and DCD. Specifically, experiments are performed on the ImageNet dataset with the ResNet50 and MobileNetV2 back- bones, using the same training settings as in the main paper. Regarding the training cost, we report results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of hours for the whole training) in Table 17 Regarding different dynamic convolution methods, it can be seen that the training cost trend is somewhat similar to that for the runtime inference speed reported in Ta- ble 8 Generally, the training cost for dynamic convolution methods is obviously heavier (about 2 x to 4x) than that for the baseline model, which is mainly due to the inte-intessive back proppggationn process for additive convolutional kernels weighted with input-dependent attentions.
Table 17: Comparison of the training cost for different dynamic convolution methods. All models trained on the ImageNet dataset using the server with 8 NVIDIA TITAN X GPUs. We report results in terms of three metrics (seconds per batch, minutes per epoch, and the total number of hours for the whole training).
Network
ResNer5O Epoch Cost (minute)
MobileNetV2 (L.0x)
Batch Cost (sccond)
Total Cost (hour)
Batch Cost (second)
Epoch Cost (minute)
Total Cost
Models
(hour)
Baseline
0.216
18.4
31.7
0.113
9.7
25.3
+ CondConv (8x)
0.429
36.4
62.8
0.175
15.1
39.3
+ DyConv (4x)
0.569
48.0
82.3
0.310
26.4
67.9
+ DCD
0.414
35.1
60.4
0.282
24.0
61.9
+ ODConv (1x)
0.513
43.4
73.5
0.301
25.6
65.7
+ ODConv (4x)
0.896
75.5
129.3
0.402
34.0
86.8
A Deep Understanding of Learnt Attention Values. Note that our ODConv leverages a multi- dimensional attention mechanism with a parallel strategy learn four types of attentions asi acii att aws for convolutional kernels along all four dimensions of the kernel space at any convolution- al layer. In principle, progressively multiplying these four types of attentions to the convolutional kernel in the location- -iise, channel-wise filter-wise and kernel-wise orders makes convolution op- erations be different w.r.t. all spatial locations, all input channels, all filters and all kernels for each input sample, providing a performance guarantee to capture rich context cues. To have better un- derstanding of this powerful attention mechanism, it is necessary to study the learnt attention values. ResNet18 models reported in Table 7 and all 50,000 images in
To this end, use the the ImageNet validation set, and conduct a set of experiments to analyze the learnt attention val- ues for a'si O'ai Offi. Onei and their full combination by providing lots of visualization examples (obtained with the Grad-CAM++ method (Chattopadhyay et al., 2018)) and the statistical distribu- tions across different layers. The detailed computational process and results are shown in Fig 5 and Fig. 6 We can get the following observations: (1) Four types of attentions Ceei Ocii Ofj and awi are complementary each other by visualization examples, which echoes the conclusion from Table 7 The most salient attention will combat the failure cases of the other three attentions, helping the model to produce accurate predications: (2) All four types of attentions demonstrate varying atten- tion value distribution trends across different layers of the trained model, showing their capability to capture rich context cues; (3) Each attention has its own value distribution trend, which is more diverse for Ossi and Qwi than for aes and atss, showing Cesi and Qui maybe more important to some degree. Note that the dimension of attentions atii Ciei a'j and aui is quite different, e.g., 9 for a with 3 x 3 convolutional kernels, 4 for Oui with = and typically several hundred for aei and fi- This is another perspective why they are different and complementary in their design nature regarding all four dimensions of the convolutional kernel space for any convolutional layer.
Limitations of ODConv. On the one side, according to the results shown in Table [ Table 2 Table 8 although our ODCony shows obviously better performance under the similar model size
18

---

Published as a conference paper at ICLR 2022
Image
alf
GT
Image
ansmono
bow tie
grile
patio
Jean
(b)
Imsge
Paat
Feit
GT
Imige
band
CD player
guillotine
score board
plate rack
(6)
GF
Image
ell
mobile home
dnumst.ck
gryse
1e]
Figure 5: Comparison of illustrative visualization results with Grad-CAM++ (Chattopadhyay et al, 2018). Results are obtained from the pre-trained ResNet18 models (reported in Table with asis aes. att, Owi, and all of them (i.e., ODConv (4x)), separately. (a) The model with asi and the model with all four attentions make the right predications, while the other three models with one single attention fail. (b) The model with azi and the model with all four attentions make the right predications, while the other three models with one single attention fail. (c) The model with a and the model with all four attentions make the right predications, while the other three models with one single attention fail. (d) The model with ahe and the model with all four attentions make the right predications, while the other three models with one single attention fail. (e) Only the model with all four attentions makes the right predications, while the other four models with one single attention fail. These visualization results further backup the conclusion observed from Table 7 Best viewed with zoom-in.
(e.g., ODConv (4x) vs. DyConv (4x)), it leads to slightly increased FLOPs and introduces extra latency to the runtime inference speed, as the proposed four types of convolutional kernel attentions asis Ocis a'fz and awi introduce bit more learnable parameters. Under the similar model size, the training cost of our ODConv also heavier than reference methods (e.g., ODConv (4x) vs. DyConv the results shown in Table 17 On the other side, although we provide wide range of ablative studies to analyze the effect of different hyperparameters ResNet18 backbone with the ImageNet dataset, and apply the resulting combination of the hyperpa- rameters to all backbone networks, it is not the optimal setting to different backbone networks. This suggests the way reduce the computational cost of our method, finding a proper combination to constrain the final model to the target speed and accuracy for particular circumstance. Besides, the potential of applying ODConv to more deep and large backbones beyond ResNet101 has been explored due to the constraint of our available computational resource.
(4x)), according
our ODConv using the
19

---

Published as a conference paper at ICLR 2022
14 15
Attention
2
5
Layer Index
10
12
14
Attention Index
633
000
Layer Index
(b)
ad
aft
Layer Index
Layer Indes
(c)
(d)
Figure 6: Comparison of the statistical distributions of the learnt attention values regarding Osis Oci Offi and Osei across different layers of the pre-trained ResNer18 model with ODConv (4x). We run the model on all 50,000 images from the ImageNet validation dataset first collect learnt attention values for Quis Olois ayt and awi separately, and then compute the layer-wise statistical distribution for each of them over all image samples. (a) For asi we show the mean attention value for each weight of 3 convolutional kernels. (b) For Owss we show the mean attention value for each of 4 convolutional kernels. (c) For aui, we show the statistical distribution of the mean attention value over all input channels corresponding to 4 convolutional kernels. (d) For agi we show the statistical distribution of the mean attention value over all convolutional filters corresponding convolutional kernels. Best viewed with zoom-in.
20


Reviews:
[{"review_id": "z4cylznLraa", "reviewer": "Reviewer_kYmD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros)\n+ The paper is written very well and easy to follow.\n+ The idea is simple, and therefore the message is clear. The results look promising.\n+ Experiments are well performed with popular competitors.\n\nCons)\n- Involving orthogonal attentions seems complementary to each other, but any intuitions why decomposing like those work well compared with the naive ones (in dyconv and cordconv) are not clearly stated.\n- Too many italics harm the readability \n\n\nComments)\n1. Did the authors run all the experiments including the competitive methods in Table 1 and Table 2 by themselves? \n2. Why ODConv is faster than DyConv in Table 8 (right)?\n3. Please specify the training budget including speed compared with condconv and dyconv.\n4. It would be better to report the model speed in Table 9 to confirm the speed improvements from diverse configurations.\n5. Please specify the # parameters and FLOPs in Table 4. If possible, it would be nice to report the model accuracy as well.\n6. In the COCO detection experiments, did the authors use ImageNet-pretrained models without dynamic convolutions or the pretrained models with dynamic convolutions reported in Table 1 and Table 2? \n7. How does the model accuracy of CondConv used at every layer like the proposed method can be changed?\n8. Can the authors visualize a_si, a_ci, a_fi, and a_wi to provide some insights? I am just curious about a trend of the learned attention vectors and which one is dominant.\n9. The combination with the proposed method (additionally including dyconv and condconv) and SE-Net variants performing attention on a feature is likely to be complimentary. Can the authors show further improvement upon architectures incorporating a SE-like module?", "summary_of_the_paper": "This work presents a dynamic convolution method based on Dynamic Conv (dyconv) and Coordinate Conv (cordconv) by incorporating more attention weights to the multiple convolutional kernels to mix kernels more effectively resulting in better accuracies on the ImageNet classification and COCO object detection tasks.  ", "summary_of_the_review": "This paper is well written, and the results about the accuracy improvements look promising. A major drawback is the actual inference speed when compared with the competitors, but it would not be a matter because one can constrain the model to the target speed in a particular circumstance. I recommend the authors study more the way of speedup the model in practice for practitioners. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hOP7MPDfJCc", "reviewer": "Reviewer_fTd9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I think the topic of the work is an interesting and valuable one. The two ideas that this work borrows from prior methods, (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, are both well-received and are known to bring meaningful improvements to base architectures. Consequently, an effort to combine and generalize these ideas is indeed intriguing and may bring further improvements and insights.\n\nRegarding the technical details of the approach, it's heavily based on prior work, particularly the design of the attention computation. The changes and additions are justified well. The method is overall quite efficient without too much overhead, due to the separate attention weights for different dimensions. \n\nThe experiments are quite comprehensive, covering multiple backbones, reference methods, and tasks. It is particularly welcome that they help provide a complete picture of the tradeoffs by also comparing the model sizes and MAdds. Overall, the results indicate that the lighter model (ODConv 1x) offers comparable or slightly better accuracies with less compute, while the larger model (ODConv 4x) provides consistent accuracy improvements. The authors also provide a wide range of ablation studies, which help a better understanding of various aspects necessary when deploying the technique to other applications. \n\nA slight shortcoming is that it appears there might be better combinations of the hyperparameters, at least for certain scenarios, that are missed. This is understandably due to the many factors that are involved. For example, it was concluded that \"r=1/16 strikes the best tradeoff between model accuracy and efficiency\" (page 8). However, increasing r to 1/4 almost closes the gap to ODConv 4x while adding a much smaller overhead. Attention sharing, investigated in Tab. 12, is another such opportunity for better accuracy with little overhead compared to using more kernels.\n\nThe author does not discuss any potential limitations or disadvantages compared to reference methods. Is it the conclusion that ODConv offers improvements over competing methods, e.g., CondConv, DyConv, SENet, without any drawbacks? Some insights in this regard can help guide applications.\n\nAnother discussion or analysis I think can be valuable is the comparison with dynamic-filter/kernel prediction networks. Since all dimensions are now being recalibrated, the approach is closer to those types of methods. For example, ODConv can be seen as a separable version of predicting full kernels.", "summary_of_the_paper": "The authors present ODConv, a type of dynamic convolutional operation. ODConv combines two prior ideas, i.e. (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, and also generalizes to all remaining dimensions of convolutional filters. The authors propose to use ODConv as drop-in replacements for regular convolutions in standard CNNs. Experiments and analysis over several tasks demonstrate that ODConv has noticeable advantages over alternatives and offers a good tradeoff between performance and compute. ", "summary_of_the_review": "The idea and technical approach are well motivated and justified. It connects and generalizes two important prior ideas and offers an elegant solution. The results are convincing as the experiments show consistent advantages over a range of tasks and base architectures. The work also has good potential in practical applications due to its convenience as a drop-in replacement for standard convolutions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "G23e4r5NO-8", "reviewer": "Reviewer_CKv4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Highlights:\n1. The authors provide a detailed analysis of dynamic convolution operations and reveal the limitation of previous works.\n2. Based on the analysis, the authors propose omni-dimensional dynamic convolution (ODConv), where multiple attention layers are employed to generate attention weights of different convolution dimensions.\n3. The proposed method can be plugged into most existing CNN architectures and has good performance on various public datasets, such as ImageNet and MS-COCO.\n4. The technical part is easy to follow, and the experimental part is comprehensive.\n\nDrawbacks:\nActually not too much. The following are just some minor questions:\n1. As shown in Table {1,2,3}, the improvement brought by ODConv decreases as the model size increases. So it seems that the improvement of strong baselines (#param > 100M) would be limited.\n2. Although advanced training tricks are not necessary for clean performance comparisons, it would be better to have a study about whether ODConv can also work well when using heavy data augmentations and a longer training schedule.\n", "summary_of_the_paper": "This work proposes a dynamic convolution that is equipped with attention layers in all dimensions. Extensive experiments show that the proposed dynamic convolution can yield a more powerful representation than counterparts, especially in light-weight backbones.", "summary_of_the_review": "In summary, this work presents a new dynamic convolution that can provide a powerful representation for image classification and object detection. The paper is well written and properly structured, and the extensive experiments prove the effectiveness of the proposed method.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "98YWaeS9Dno", "reviewer": "Reviewer_PSFr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper starts by analyzing the problem of traditional dynamic convolution frameworks, e.g. CondCond and DyConv, and find the previous networks are not fully adapted with the kernel-wise dynamic coefficient. In order to overcome this issue, the author proposed to dynamically weight all four dimensions for the convolutional kernel. This idea can be viewed as a more general form of dynamic convolution and SE, which I think is quite novel. Another strength of this paper is its comprehensive experiments. Both the results in the main paper and the supplementary further justify the benefits brought by the newly designed ODConv. \n\nMy main concerns about this submission are as follows.\n\n1. In Table 7, the ablations for the performance of dynamic operation added on different dimensions are shown. From the first three results, I observe the gain brought by the dynamics added on the spatial dimension (70.25% vs 72.42%) is the largest compared to the other two, which is very weird to me. Based on the design of the branch that generates dynamic coefficients, I find the features are always globally averaged, which removes the spatial information. I can hardly understand why the spatially averaged features can still generate the dynamic coefficients to adjust the kernels from the spatial view. Furthermore, I doubt the effectiveness of adding dynamics to the input channels, since the input features have been dynamically adapted by the previous layer. Although non-linearity might exist between two convolutional layers, the activation functions do not fuse channels and will not thoroughly break the dynamic properties. Thus I doubt some offsets might happen and weaken the efficiency of the proposed model.\n\n2. The experimental results are just built on ResNet and MobileNet V2. MobileNet V3, which is more efficient, is not compared with. I wonder whether the author can further compare ODConv to CondConv and DCD with MobileNet V3 as backbone. Moreover, in the object detection part, I do not find the results for DCD. I'm curious about how DCD performs on object detection and whether the proposed ODConv is still better than DCD.\n\n3. In Table 8, the inference time is tested, which is good. But I do not find the latency for the static model. Besides, for the inference time tested on CPU, I'm not sure whether multiple cores are used or not. Please clarify. \n\n4. In the paper the author claim \"ODConv with only filter-wise attention αf1 will be reduced into performing SE\" is not accurate. For SE, the expression is y=f(x) \\dot x, while ODConv with only filter-wise attention is y = f(x) \\dot wx. The key difference is whether the dynamic coefficient f(x) is generated based on the input or output of the convolutional layers.", "summary_of_the_paper": "This work mainly focuses on designing a new dynamic network for large-scale image recognition problems. Specifically, the author discussed the weakness of the existing dynamic convolution operations, and based on the analysis the author proposed a novel framework with the name ODConv. Extensive experiments confirm the superiority of the proposed new framework.", "summary_of_the_review": "Overall speaking, the paper has novelty and very comprehensive experiments to support it. I am prone to accept this submission. But the weakness of this paper is too experimental-oriented without thorough analysis on the results. My first concern is critical, since I want to understand what happens when different dimension is dynamically adapted. If possible, I hope the author to add some visualization or other analysis on the dynamic coefficient from the statistical view to help readers have a better understand of the proposed dynamic mechanism. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents ODConv, a convolution pattern which uses attention in the convolutions across all dimensions of the weight tensor.
The paper is well motivated and well explained, easy to follow.
This work is built on top of previous work, but reviewers all agree that the contributions of this paper are significant.
The experimental section is comprehensive, with several benchmarks, and show clear improvements.
The reviewers suggested a few additional remarks, and discussions to add to the paper, which the authors have addressed in the rebuttal. Reviewers seem in general happy with the authors answers to their concerns.
This seems like a sound and meaningful paper. I am fully in favour of acceptance, and I recommend this paper to be presented as a spotlight.
Decision: Accept (Spotlight)