DMSANET: DUAL MULTI SCALE ATTENTION NETWORK

1 INTRODUCTION . The local receptive field of the human eye has led to the construction of convolutional neural networks which has powered much of the recent advances in computer vision . Multi scale architecture used in the famous InceptionNet ( Szegedy et al. , 2016 ) aggregates multi-scale information from different size convolutional kernels . Attention Networks has attracted a lot of attention recently as it allows the network to focus on only then essential aspects while ignoring the ones which are not useful ( Li et al. , 2019 ) , ( Cao et al. , 2019 ) and ( Li et al. , 2019 ) . A lot of problems have been successfully tackled using attention mechanism in computer vision like image classification , image segmentation , object detection and image generation . Most of the attention mechanisms can be broadly classified into two types channel attention and spatial attention , both of which strengthens the original features by aggregating the same feature from all the positions with different aggregation strategies , transformations , and strengthening functions ( Zhang et al. , 2021 ) . Some of the work combined both these mechanism together and achieved better results ( Cao et al. , 2019 ) and ( Woo et al. , 2018 ) . The computational burden was reduced by ( Wang et al. , 2020 ) using efficient channel attention and 1 × 1 convolution . The most popular attention mechanism is the Squeeze-and Excitation module ( Hu et al. , 2018b ) , which can significantly improve the performance with a considerably low cost . The “ channel shuffle ” operator is used ( Zhang and Yang , 2021 ) to enable information communication between the two branches . It uses a grouping strategy , which divides the input feature map into groups along the channel dimension . 2 RELATED WORK . There are two main problems which hinders the progress in this field : 1 ) Both spatial and channel attention as well as network using combination of two uses only local information while ignoring long range channel dependency , 2 ) The previous architectures fail to capture spatial information at different scales to be more robust and handle more complex problems . These two challenges were tackled by ( Duta et al. , 2020 ) and ( Li et al. , 2019 ) respectivly . The problem with these architectures is that the number of parameters increased considerably . Pyramid Split Attention ( PSA ) ( Zhang et al. , 2021 ) has the ability to process the input tensor at multiple scales . A multi-scale pyramid convolution structure is used to integrate information at different scales on each channel-wise feature map . The channel-wise attention weight of the multi-scale feature maps are extracted hence long range channel dependency is done . Non-Local block ( Wang et al. , 2018 ) is proposed to build a dense spatial feature map and capture the long-range dependency using non-local operations . ( Li et al. , 2019 ) used a dynamic selection attention mechanism that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input feature map . ( Fu et al. , 2019 ) proposed a network to integrate local features with their global dependencies by summing these two attention modules from different branches . Multi scale architectures have been used sucessfully for a lot of vision problems ( ? ) , ( Hu et al. , 2018b ) and ( Sagar and Soundrapandiyan , 2020 ) . ( Fu et al. , 2019 ) adaptively integrated local features with their global dependencies by summing the two attention modules from different branches . ( Hu et al. , 2018a ) used spatial extension using a depth-wise convolution to aggregate individual features . Our network borrows ideas from ( Gao et al. , 2018 ) which used a network to capture local cross-channel interactions . The performance ( in terms of accuracy ) vs computational complexity ( in terms of number of parameters ) of the state of art attention modules is shown in Figure 1 : Our main contributions can be summarized as follows : • A new attention module is proposed which aggregates feature information at various scales . Our network is scalable and can be easily plugged into various computer vision problems . • Our network captures more contextual information using both spatial and channel attention at various scales . • Our experiments demonstrate that our network outperforms previous state of the art with lesser computational cost . 3 METHOD . 3.1 FEATURE GROUPING . Shuffle Attention module divides the input feature map into groups and uses Shuffle Unit to integrate the channel attention and spatial attention into one block for each group . The sub-features are aggregated and a “ channel shuffle ” operator is used for communicating the information between different sub-features . For a given feature map X ∈ RC×H×W , where C , H , W indicate the channel number , spatial height , and width , respectively , shuffle attention module divides X into G groups along the channel dimension , i.e. , X = [ X1 , XG ] , Xk ∈ RC/G×H×W . An attention module is used to weight the importance of each feature . The input of Xk is split into two networks along the channel dimension Xk1 , Xk2 ∈ RC/2G×H×W . The first branch is used to produce a channel attention map by using the relationship of channels , while the second branch is used to generate a spatial attention map by using the spatial relationship of different features . 3.2 CHANNEL ATTENTION MODULE . The channel attention module is used to selectively weight the importance of each channel and thus produces best output features . This helps in reducing the number of parameters of the network . Let X ∈ RC×H×W denotes the input feature map , where the quantityH , W , C represent its height , width and number of input channels respectively . A SE block consists of two parts : squeeze and excitation , which are respectively designed for encoding the global information and adaptively recalibrating the channel-wise relationship . The Global Average Pooling ( GAP ) operation can be calculated by the as shown in Equation 1 : GAPc = 1 H ×W H∑ i=1 W∑ j=1 xc ( i , j ) ( 1 ) The attention weight of the cth channel in the SE block can be written as denoted in Equation 2 : wc = σ ( W1ReLU ( W0 ( GAPc ) ) ) ( 2 ) where W0 ∈ RC×Cr and W1 ∈ RCr×C represent the fully-connected ( FC ) layers . The symbol σ represents the excitation function where Sigmoid function is usually used . We calculate the channel attention map X ∈ RC×C from the original features A ∈ RC×H×W . We reshape A to RC×N , and then perform a matrix multiplication between A and the transpose of A . We then apply a softmax layer to obtain the channel attention map X ∈ RC×C as shown in Equation 3 : xji = exp ( Ai ·Aj ) ∑C i=1 exp ( Ai ·Aj ) ( 3 ) where xji measures the ith channel ’ s impact on the jth channel . We perform a matrix multiplication between the transpose of X and A and reshape their result to RC×H×W . We also multiply the result by a scale parameter β and perform an element-wise sum operation with A to obtain the final output E ∈ RC×H×W as shown in Equation 4 : E1j = β C∑ i=1 ( xjiAi ) +Aj ( 4 ) 3.3 SPATIAL ATTENTION MODULE . We use Instance Normalization ( IN ) over Xk2 to obtain spatial-wise statistics . A Fc ( ) operation is used to enhance the representation of Xk2 . The final output of spatial attention is obtained by where W2 and b2 are parameters with shape RC/2G×1×1 . After that the two branches are concatenated to make the number of channels equal to the number of input . A local feature denoted by A ∈ RC×H×W is fed into a convolution layer to generate two new feature maps B and C , respectively where B , C ∈ RC×H×W . We reshape them to RC×N , where N = H ×W is the number of pixels . Next a matrix multiplication is done between the transpose of C and B , and apply a softmax layer to calculate the spatial attention map S ∈ RN×N . This operation is shown in Equation 1 : sji = exp ( Bi · Cj ) ∑N i=1 exp ( Bi · Cj ) ( 5 ) where sji measures the ith position ’ s impact on jth position . Next we feed feature A into a convolution layer to generate a new feature map D ∈ RC×H×W and reshape it to RC×N . We perform a matrix multiplication between D and the transpose of S and reshape the result to RC×H×W . We multiply it by a scale parameter α and perform a element-wise sum operation with the feature A to obtain the final output E ∈ RC×H×W as shown in Equation 2 : E2j = α N∑ i=1 ( sjiDi ) +Aj ( 6 ) 3.4 AGGREGATION . In the final part of the network , all the sub-features are aggregated . We use a ” channel shuffle ” operator to enable cross-group information flow along the channel dimension . The final output of our module is the same size as that of input , making our attention module quite easy to integrate with other networks . The whole multi-scale pre-processed feature map can be obtained by a concatenation way as defined in Equation 7 : F = Concat ( [ E1j , E2j ] ) ( 7 ) where F ∈ RC×H×W is the obtained multi-scale feature map . Our attention module is used across channels to adaptively select different spatial scales which is guided by the feature descriptor . This operation is defined in Equation 8 : atti = Softmax ( Zi ) = exp ( Zi ) ∑S−1 i=0 exp ( Zi ) ( 8 ) Finally we multiply the re-calibrated weight of multi-scale channel attention atti with the feature map of the corresponding scale Fi as shown in Equation 9 : Yi = Fi atti i = 1 , 2 , 3 , · · ·S − 1 ( 9 ) 3.5 NETWORK ARCHITECTURE . We propose DMSA module with the goal to build more efficient and scalable architecture . The first part of our network borrows ideas from ( Li et al. , 2019 ) and ( Zhang and Yang , 2021 ) . An input feature mapX is splitted intoN parts along with the channel dimension . For each splitted parts , it has C0=CS number of common channels , and the ith feature map is Xi ∈ RC0×H×W . The individual features are fused before being passed to two different branches . These two branches are comprised of position attention module and channel attention module as proposed in ( Fu et al. , 2019 ) for semantic segmentation . The second part of our network does the following 1 ) Builds a spatial attention matrix which models the spatial relationship between any two pixels of the features , 2 ) A matrix multiplication between the attention matrix and the original features . 3 ) An element-wise sum operation is done on the resulting matrix and original features . The operators concat and sum are used to reshape the features . The features from the two parallel branches are aggregated to produce the final output . The complete network architecture is shown in Figure 2 : We compare our network architecture with Resnet ( Wang et al. , 2017 ) , SENet ( Hu et al. , 2018b ) and EPSANet ( Zhang et al. , 2021 ) in Figure 3 . We use our DMSA module in between 3× 3 convolution and 1× 1 convolution . Our network is able to extract features at various scales and aggregate those individual features before passing through the attention module . The architectural details our proposed attention network is shown in Table 1 :

Summary: In this paper, the authors propose a new attention module that shows better performance and lesser computation than most existing attention modules. Based on this module, the so-called Dual Multi Scale Attention Network is proposed. Several experiments are conducted to verify the performance on image classification, object detection and instance segmentation. Results shows that the proposed network has some advantages than previous works.