DMSANET: DUAL MULTI SCALE ATTENTION NETWORK

1 INTRODUCTION . The local receptive field of the human eye has led to the construction of convolutional neural networks which has powered much of the recent advances in computer vision . Multi scale architecture used in the famous InceptionNet ( Szegedy et al. , 2016 ) aggregates multi-scale information from different size convolutional kernels . Attention Networks has attracted a lot of attention recently as it allows the network to focus on only then essential aspects while ignoring the ones which are not useful ( Li et al. , 2019 ) , ( Cao et al. , 2019 ) and ( Li et al. , 2019 ) . A lot of problems have been successfully tackled using attention mechanism in computer vision like image classification , image segmentation , object detection and image generation . Most of the attention mechanisms can be broadly classified into two types channel attention and spatial attention , both of which strengthens the original features by aggregating the same feature from all the positions with different aggregation strategies , transformations , and strengthening functions ( Zhang et al. , 2021 ) . Some of the work combined both these mechanism together and achieved better results ( Cao et al. , 2019 ) and ( Woo et al. , 2018 ) . The computational burden was reduced by ( Wang et al. , 2020 ) using efficient channel attention and 1 × 1 convolution . The most popular attention mechanism is the Squeeze-and Excitation module ( Hu et al. , 2018b ) , which can significantly improve the performance with a considerably low cost . The “ channel shuffle ” operator is used ( Zhang and Yang , 2021 ) to enable information communication between the two branches . It uses a grouping strategy , which divides the input feature map into groups along the channel dimension . 2 RELATED WORK . There are two main problems which hinders the progress in this field : 1 ) Both spatial and channel attention as well as network using combination of two uses only local information while ignoring long range channel dependency , 2 ) The previous architectures fail to capture spatial information at different scales to be more robust and handle more complex problems . These two challenges were tackled by ( Duta et al. , 2020 ) and ( Li et al. , 2019 ) respectivly . The problem with these architectures is that the number of parameters increased considerably . Pyramid Split Attention ( PSA ) ( Zhang et al. , 2021 ) has the ability to process the input tensor at multiple scales . A multi-scale pyramid convolution structure is used to integrate information at different scales on each channel-wise feature map . The channel-wise attention weight of the multi-scale feature maps are extracted hence long range channel dependency is done . Non-Local block ( Wang et al. , 2018 ) is proposed to build a dense spatial feature map and capture the long-range dependency using non-local operations . ( Li et al. , 2019 ) used a dynamic selection attention mechanism that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input feature map . ( Fu et al. , 2019 ) proposed a network to integrate local features with their global dependencies by summing these two attention modules from different branches . Multi scale architectures have been used sucessfully for a lot of vision problems ( ? ) , ( Hu et al. , 2018b ) and ( Sagar and Soundrapandiyan , 2020 ) . ( Fu et al. , 2019 ) adaptively integrated local features with their global dependencies by summing the two attention modules from different branches . ( Hu et al. , 2018a ) used spatial extension using a depth-wise convolution to aggregate individual features . Our network borrows ideas from ( Gao et al. , 2018 ) which used a network to capture local cross-channel interactions . The performance ( in terms of accuracy ) vs computational complexity ( in terms of number of parameters ) of the state of art attention modules is shown in Figure 1 : Our main contributions can be summarized as follows : • A new attention module is proposed which aggregates feature information at various scales . Our network is scalable and can be easily plugged into various computer vision problems . • Our network captures more contextual information using both spatial and channel attention at various scales . • Our experiments demonstrate that our network outperforms previous state of the art with lesser computational cost . 3 METHOD . 3.1 FEATURE GROUPING . Shuffle Attention module divides the input feature map into groups and uses Shuffle Unit to integrate the channel attention and spatial attention into one block for each group . The sub-features are aggregated and a “ channel shuffle ” operator is used for communicating the information between different sub-features . For a given feature map X ∈ RC×H×W , where C , H , W indicate the channel number , spatial height , and width , respectively , shuffle attention module divides X into G groups along the channel dimension , i.e. , X = [ X1 , XG ] , Xk ∈ RC/G×H×W . An attention module is used to weight the importance of each feature . The input of Xk is split into two networks along the channel dimension Xk1 , Xk2 ∈ RC/2G×H×W . The first branch is used to produce a channel attention map by using the relationship of channels , while the second branch is used to generate a spatial attention map by using the spatial relationship of different features . 3.2 CHANNEL ATTENTION MODULE . The channel attention module is used to selectively weight the importance of each channel and thus produces best output features . This helps in reducing the number of parameters of the network . Let X ∈ RC×H×W denotes the input feature map , where the quantityH , W , C represent its height , width and number of input channels respectively . A SE block consists of two parts : squeeze and excitation , which are respectively designed for encoding the global information and adaptively recalibrating the channel-wise relationship . The Global Average Pooling ( GAP ) operation can be calculated by the as shown in Equation 1 : GAPc = 1 H ×W H∑ i=1 W∑ j=1 xc ( i , j ) ( 1 ) The attention weight of the cth channel in the SE block can be written as denoted in Equation 2 : wc = σ ( W1ReLU ( W0 ( GAPc ) ) ) ( 2 ) where W0 ∈ RC×Cr and W1 ∈ RCr×C represent the fully-connected ( FC ) layers . The symbol σ represents the excitation function where Sigmoid function is usually used . We calculate the channel attention map X ∈ RC×C from the original features A ∈ RC×H×W . We reshape A to RC×N , and then perform a matrix multiplication between A and the transpose of A . We then apply a softmax layer to obtain the channel attention map X ∈ RC×C as shown in Equation 3 : xji = exp ( Ai ·Aj ) ∑C i=1 exp ( Ai ·Aj ) ( 3 ) where xji measures the ith channel ’ s impact on the jth channel . We perform a matrix multiplication between the transpose of X and A and reshape their result to RC×H×W . We also multiply the result by a scale parameter β and perform an element-wise sum operation with A to obtain the final output E ∈ RC×H×W as shown in Equation 4 : E1j = β C∑ i=1 ( xjiAi ) +Aj ( 4 ) 3.3 SPATIAL ATTENTION MODULE . We use Instance Normalization ( IN ) over Xk2 to obtain spatial-wise statistics . A Fc ( ) operation is used to enhance the representation of Xk2 . The final output of spatial attention is obtained by where W2 and b2 are parameters with shape RC/2G×1×1 . After that the two branches are concatenated to make the number of channels equal to the number of input . A local feature denoted by A ∈ RC×H×W is fed into a convolution layer to generate two new feature maps B and C , respectively where B , C ∈ RC×H×W . We reshape them to RC×N , where N = H ×W is the number of pixels . Next a matrix multiplication is done between the transpose of C and B , and apply a softmax layer to calculate the spatial attention map S ∈ RN×N . This operation is shown in Equation 1 : sji = exp ( Bi · Cj ) ∑N i=1 exp ( Bi · Cj ) ( 5 ) where sji measures the ith position ’ s impact on jth position . Next we feed feature A into a convolution layer to generate a new feature map D ∈ RC×H×W and reshape it to RC×N . We perform a matrix multiplication between D and the transpose of S and reshape the result to RC×H×W . We multiply it by a scale parameter α and perform a element-wise sum operation with the feature A to obtain the final output E ∈ RC×H×W as shown in Equation 2 : E2j = α N∑ i=1 ( sjiDi ) +Aj ( 6 ) 3.4 AGGREGATION . In the final part of the network , all the sub-features are aggregated . We use a ” channel shuffle ” operator to enable cross-group information flow along the channel dimension . The final output of our module is the same size as that of input , making our attention module quite easy to integrate with other networks . The whole multi-scale pre-processed feature map can be obtained by a concatenation way as defined in Equation 7 : F = Concat ( [ E1j , E2j ] ) ( 7 ) where F ∈ RC×H×W is the obtained multi-scale feature map . Our attention module is used across channels to adaptively select different spatial scales which is guided by the feature descriptor . This operation is defined in Equation 8 : atti = Softmax ( Zi ) = exp ( Zi ) ∑S−1 i=0 exp ( Zi ) ( 8 ) Finally we multiply the re-calibrated weight of multi-scale channel attention atti with the feature map of the corresponding scale Fi as shown in Equation 9 : Yi = Fi atti i = 1 , 2 , 3 , · · ·S − 1 ( 9 ) 3.5 NETWORK ARCHITECTURE . We propose DMSA module with the goal to build more efficient and scalable architecture . The first part of our network borrows ideas from ( Li et al. , 2019 ) and ( Zhang and Yang , 2021 ) . An input feature mapX is splitted intoN parts along with the channel dimension . For each splitted parts , it has C0=CS number of common channels , and the ith feature map is Xi ∈ RC0×H×W . The individual features are fused before being passed to two different branches . These two branches are comprised of position attention module and channel attention module as proposed in ( Fu et al. , 2019 ) for semantic segmentation . The second part of our network does the following 1 ) Builds a spatial attention matrix which models the spatial relationship between any two pixels of the features , 2 ) A matrix multiplication between the attention matrix and the original features . 3 ) An element-wise sum operation is done on the resulting matrix and original features . The operators concat and sum are used to reshape the features . The features from the two parallel branches are aggregated to produce the final output . The complete network architecture is shown in Figure 2 : We compare our network architecture with Resnet ( Wang et al. , 2017 ) , SENet ( Hu et al. , 2018b ) and EPSANet ( Zhang et al. , 2021 ) in Figure 3 . We use our DMSA module in between 3× 3 convolution and 1× 1 convolution . Our network is able to extract features at various scales and aggregate those individual features before passing through the attention module . The architectural details our proposed attention network is shown in Table 1 :

Summary: In this paper, the authors proposed a new attention module, named Dual Multi-Scale Attention. Three commonly used methods are combined: multi-scale, channel-attention, and position attention. By introducing the new module (which is a simple combination without any novelty) to ResNet, it SIGNIFICANTLY improves the performance, e.g, 75.20 to 80.02  (and 76.83 to 81.54 for ResNet101) on ImageNet, 36.4 to 41.4 on MS COCO.