Omni-Dimensional Dynamic Convolution

1 INTRODUCTION . In the past decade , we have witnessed the tremendous success of deep Convolutional Neural Networks ( CNNs ) in many computer vision applications ( Krizhevsky et al. , 2012 ; Girshick et al. , 2014 ; Long et al. , 2015 ; He et al. , 2017 ) . The most common way of constructing a deep CNN is to stack a number of convolutional layers as well as other basic layers organized with the selected feature connection topology . Along with great advances in CNN architecture design by manual engineering ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Howard et al. , 2017 ) and automatic searching ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Howard et al. , 2019 ) , lots of prevailing classification backbones have been presented . Recent works ( Wang et al. , 2017 ; Hu et al. , 2018b ; Park et al. , 2018 ; Woo et al. , 2018 ; Yang et al. , 2019 ; Chen et al. , 2020 ) show that incorporating attention mechanisms into convolutional blocks can further push the performance boundaries of modern CNNs , and thus it has attracted great research interest in the deep learning community . The well-known SENet ( Hu et al. , 2018b ) uses an attention module consisting of squeeze and excitation operations to adaptively recalibrate the output features of convolutional layers , strengthening the representation power of a CNN via encouraging informative feature channels while suppressing less important ones . Numerous attentive feature recalibration variants ( Woo et al. , 2018 ; Park et al. , 2018 ; Hu et al. , 2018a ) have been proposed since then . In Lin et al . ( 2020 ) and Quader et al . ( 2020 ) , two attention extensions to modulate the convolutional weights instead of the output features are also presented . Unlike the aforementioned methods in which the number of convolutional parameters of a target network is fixed , dynamic convolution , which applies the attention mechanism over n 1Here we follow the definitions in Yang et al . ( 2019 ) and Chen et al . ( 2020 ) where a convolutional kernel refers to the filter set of a convolutional layer . additive convolutional kernels to increase the size and the capacity of a network while maintaining efficient inference , has recently become popular in optimizing efficient CNNs . This line of research is pioneered by Conditionally Parameterized Convolutions ( CondConv ) ( Yang et al. , 2019 ) and Dynamic Convolution ( DyConv ) ( Chen et al. , 2020 ) whose basic ideas are the same . Generally , unlike a regular convolutional layer which applies the same ( i.e. , static ) convolutional kernel to all input samples , a dynamic convolutional layer learns a linear combination of n convolutional kernels weighted with their attentions conditioned on the input features . Despite significant accuracy improvements for light-weight CNNs , dynamic convolution designed in this way has two limitations . Firstly , the main limitation lies in the attention mechanism design . The dynamic property of CondConv and DyConv comes from computing convolutional kernels as a function of the input features . However , we observe that they endow the dynamic property to convolutional kernels through one dimension ( regarding the convolutional kernel number ) of the kernel space while the other three dimensions ( regarding the spatial size , the input channel number and the output channel number for each convolutional kernel ) are overlooked . As a result , the weights of each convolutional kernel share the same attention scalar for a given input , limiting their abilities to capture rich contextual cues . That is , the potentials of dynamic convolutional property have not been fully explored by existing works , and thus they leave much room for improving the model performance . Secondly , at a convolutional layer , replacing regular convolution by dynamic convolution increases the number of convolutional parameters by n times . When applying dynamic convolution to a lot of convolutional layers , it will heavily increase the model size . To handle this limitation , Li et al . ( 2021 ) proposes a dynamic convolution decomposition method which can get more compact yet competitive models . Instead , in this paper we address both of the above limitations from a new perspective : inserting a more diverse and effective attention mechanism into the convolutional kernel space . Our core contribution is an improved dynamic convolution design called Omni-dimensional Dynamic Convolution ( ODConv ) . Unlike existing works discussed above , at any convolutional layer , ODConv leverages a multi-dimensional attention mechanism to learn four types of attentions for convolutional kernels along all four dimensions of the kernel space in a parallel manner . We show that these four types of attentions learnt by our ODConv are complementary to each other , and progressively applying them to the corresponding convolutional kernels can substantially strengthen the feature extraction ability of basic convolution operations of a CNN . Consequently , ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels , introducing substantially fewer extra parameters to the final models . As a drop-in design , ODConv can be used to replace regular convolutions in many prevailing CNN architectures . It strikes a better tradeoff between model accuracy and efficiency compared to existing dynamic convolution designs , as validated by extensive experiments on the large-scale ImageNet classification dataset ( Russakovsky et al. , 2015 ) with various CNN backbones . ODConv also shows better recognition performance under similar model complexities when compared to other state-ofthe-art attention methods for output feature recalibration ( Woo et al. , 2018 ; Hu et al. , 2018b ; Wang et al. , 2020 ; Lin et al. , 2020 ) and for convolutional weights modification ( Ma et al. , 2020 ; Lin et al. , 2020 ; Quader et al. , 2020 ) . Furthermore , the performance improvements by ODConv for the pretrained classification models can transfer well to downstream tasks such as object detection on the MS-COCO dataset ( Lin et al. , 2014 ) , validating its promising generalization ability . 2 RELATED WORK . Deep CNN Architectures . AlexNet ( Krizhevsky et al. , 2012 ) ignited the surge of deep CNNs by winning the ImageNet classification challenge 2012 . Since then , lots of well-known CNN architectures such as VGGNet ( Simonyan & Zisserman , 2015 ) , InceptionNet ( Szegedy et al. , 2015 ) , ResNet ( He et al. , 2016 ) , DenseNet ( Huang et al. , 2017 ) and ResNeXt ( Xie et al. , 2017 ) have been proposed , which are designed to be much deeper and have more sophisticated connection topologies compared with AlexNet . To ease the deployment of inference models on resource-limited platforms , MobileNets ( Howard et al. , 2017 ; Sandler et al. , 2018 ) and ShuffleNet ( Zhang et al. , 2018b ; Ma et al. , 2018 ) are presented , which particularly make a tradeoff between model accuracy and efficiency . All the aforementioned CNNs are manually designed . Recently , researchers have also made great efforts ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Howard et al. , 2019 ) to automate the network design process . Our ODConv could be potentially used to boost the performance of most of these prevailing CNN architectures constructed with regular convolutional layers . Attentive Feature Recalibration . Designing attentive feature recalibration modules to improve the performance of a CNN has been widely studied in recent years . Wang et al . ( 2017 ) proposes a specialized attention module consisting of a trunk branch and a mask branch , and inserts it into the intermediate stages of deep residual networks . SENet ( Hu et al. , 2018b ) uses a seminal channel attention module termed Squeeze-and-Excitation ( SE ) to exploit the interdependencies between the channels of convolutional features . Many subsequent works improve SE from different aspects , following its two-stage design ( i.e. , feature aggregation and feature recalibration ) . BAM ( Park et al. , 2018 ) and CBAM ( Woo et al. , 2018 ) combine the channel attention module with the spatial attention module . Misra et al . ( 2021 ) presents an attention module having three branches conditioned on the features rotated along three different dimensions . GE ( Hu et al. , 2018a ) introduces a gather operator to extract better global context from a large spatial extent . To enhance the feature aggregation capability , SRM ( Lee et al. , 2019 ) replaces the global average by the channel-wise mean and standard deviation . SKNets ( Li et al. , 2019 ) add an attention design over two branches with different sized convolutions to fuse multi-scale feature outputs . ECA ( Wang et al. , 2020 ) provides a more efficient channel attention design using cheaper 1D convolutions to replace the first fully connected layer of SE . Instead of recalibrating the output convolutional features by attention modules , dynamic convolution methods apply attention mechanisms to a linear combination of n convolutional kernels . Dynamic Weight Networks . Making the weights of a neural network to be sample-adaptive via dynamic mechanisms has shown great potentials for boosting model capacity and generalization . Hypernetworks ( Ha et al. , 2017 ) use a small network called hypernetwork to generate the weights for a larger recurrent network called main network . MetaNet ( Munkhdalai & Yu , 2017 ) adopts a meta learning model to parameterize the task-adaptive network for rapid generalization across a sequence of tasks . Jaderberg et al . ( 2015 ) proposes a Spatial Transformer module conditioned on the learnt features to predict the parametric transformation , and applies it to align the distorted input image . Dynamic Filter Network ( Jia et al. , 2016 ) uses a filter generation network to produce filters conditioned on an input , and processes another input with the generated filters . DynamoNet ( Diba et al. , 2019 ) uses dynamically generated motion filters to handle the action recognition problem . Kernel Prediction Networks ( Bako et al. , 2017 ; Mildenhall et al. , 2018 ) leverage a CNN architecture to predict spatially varying kernels used for video denoising . WeightNet ( Ma et al. , 2020 ) appends a grouped fully connected layer to the attention feature vector of an SE block , generating the weights of a CNN used for image recognition . Lin et al . ( 2020 ) modifies the weights of convolutional layers with a gated module under the guidance of global context , while Quader et al . ( 2020 ) directly uses either an SE block or a simple activation function conditioned on the magnitudes of convolutional weights to modify the weights themselves . Our ODConv aims to address the limitations of recently proposed dynamic convolution ( Yang et al. , 2019 ; Chen et al. , 2020 ) which differs from these methods both in focus and formulation , see the Introduction and Method sections for details . 3 METHOD . In this section , we first make a review of dynamic convolution via a general formulation . Then , we describe the formulation of our ODConv , clarify its properties and detail its implementation . 3.1 REVIEW OF DYNAMIC CONVOLUTION . Basic concept . A regular convolutional layer has a single static convolutional kernel which is applied to all input samples . For a dynamic convolutional layer , it uses a linear combination of n convolutional kernels weighted dynamically with an attention mechanism , making convolution operations be input-dependent . Mathematically , dynamic convolution operations can be defined as y = ( αw1W1 + ... + αwnWn ) ∗ x , ( 1 ) where x ∈ Rh×w×cin and y ∈ Rh×w×cout denote the input features and the output features ( having cin/cout channels with the height h and the width w ) , respectively ; Wi denotes the ith convolutional kernel consisting of cout filters Wmi ∈ Rk×k×cin , m = 1 , ... , cout ; αwi ∈ R is the attention scalar for weighting Wi , which is computed by an attention function πwi ( x ) conditioned on the input features ; ∗ denotes the convolution operation . For conciseness , here we omit the bias term . CondConv vs. DyConv . Although the concept of dynamic convolution defined in Eq . 1 is proposed separately in CondConv ( Yang et al. , 2019 ) and DyConv ( Chen et al. , 2020 ) , their implementations are different , mainly in the structure of πwi ( x ) to compute αwi , the model training strategy , and the layer locations to apply dynamic convolutions . Specifically , both methods choose the modified SE structure for πwi ( x ) , and CondConv uses a Sigmoid function while DyConv uses a Softmax function as the activation function to compute αwi . DyConv adopts a temperature annealing strategy in the training process to suppress the near one-hot output of the Softmax function . For all their tested CNN architectures , CondConv replaces the convolutional layers in the final several blocks ( e.g. , 6 for the MobileNetV2 backbones and 3 for the ResNet backbones ) and the last fully connected layer , while DyConv replaces all convolutional layers except the first layer . These implementation differences lead to different results in model accuracy , size and efficiency for CondConv and DyConv . Limitation Discussions . According to Eq . 1 , dynamic convolution has two basic components : the convolutional kernels { W1 , ... Wn } , and the attention function πwi ( x ) to compute their attention scalars { αw1 , ... αwn } . Given n convolutional kernels , the corresponding kernel space has four dimensions regarding the spatial kernel size k×k , the input channel number cin and the output channel number cout for each convolutional kernel , and the convolutional kernel number n. However , for CondConv and DyConv , we can observe that πwi ( x ) allocates a single attention scalar αwi to the convolutional kernel Wi , meaning that all its cout filters Wmi ∈ Rk×k×cin , m = 1 , ... , cout have the same attention value for the input x . In other words , the spatial dimension , the input channel dimension and the output channel dimension for the convolutional kernel Wi are ignored by CondConv and DyConv . This leads to a coarse exploitation of the kernel space when they design their attention mechanisms for endowing n convolutional kernels with the dynamic property . This may also be one of the reasons why CondConv and DyConv show much lower performance gains to relatively large CNNs compared to efficient ones . Besides , compared to a regular convolutional layer , a dynamic convolutional layer increases the number of convolutional parameters by n times ( but the increase of Multiply-Adds ( MAdds ) is marginal due to the additive property of n convolutional kernels ) . Typically , CondConv sets n = 8 and DyConv sets n = 4 . Therefore , it will heavily increase the model size when applying dynamic convolution to a lot of convolutional layers . However , we empirically find that removing the attention mechanism from CondConv|DyConv ( i.e. , setting αwi = 1 ) almost diminishes the accuracy boosts for prevailing CNN backbones on the ImageNet dataset close to zero . For instance , on ResNet18 , the top-1 gain averaged over 3 runs decreases from 1.78 % |2.51 % to 0.08 % |0.14 % when removing the attention mechanism from CondConv|DyConv , respectively . These observations indicate that the attention mechanism design plays the key role in dynamic convolution , and a more effective design may make a good balance between model accuracy and size .

Summary: The authors present ODConv, a type of dynamic convolutional operation. ODConv combines two prior ideas, i.e. (1) filter recalibration with attention in SENet and (2) additive kernels in CondConv/DyConv, and also generalizes to all remaining dimensions of convolutional filters. The authors propose to use ODConv as drop-in replacements for regular convolutions in standard CNNs. Experiments and analysis over several tasks demonstrate that ODConv has noticeable advantages over alternatives and offers a good tradeoff between performance and compute.