DMSANET: DUAL MULTI SCALE ATTENTION NET-
Year: 2021

Abstract:
Attention mechanism of late has been quite popular in the computer vision community. A lot of work has been done to improve the performance of the network, although almost always it results in increased computational complexity. In this paper, we propose a new attention module that not only achieves the best performance but also has lesser parameters compared to most existing models. Our attention module can easily be integrated with other convolutional neural networks because of its lightweight nature. The proposed network named Dual Multi Scale Attention Network (DMSANet) is comprised of two parts: the first part is used to extract features at various scales and aggregate them, the second part uses spatial and channel attention modules in parallel to adaptively integrate local features with their global dependencies. We benchmark our network performance for Image Classification on ImageNet dataset, Object Detection and Instance Segmentation both on MS COCO dataset.

heading: 1 INTRODUCTION
text: The local receptive field of the human eye has led to the construction of convolutional neural networks which has powered much of the recent advances in computer vision. Multi scale architecture used in the famous InceptionNet (Szegedy et al., 2016) aggregates multi-scale information from different size convolutional kernels. Attention Networks has attracted a lot of attention recently as it allows the network to focus on only then essential aspects while ignoring the ones which are not useful (Li et al., 2019), (Cao et al., 2019) and (Li et al., 2019).
A lot of problems have been successfully tackled using attention mechanism in computer vision like image classification, image segmentation, object detection and image generation. Most of the attention mechanisms can be broadly classified into two types channel attention and spatial attention, both of which strengthens the original features by aggregating the same feature from all the positions with different aggregation strategies, transformations, and strengthening functions (Zhang et al., 2021).
Some of the work combined both these mechanism together and achieved better results (Cao et al., 2019) and (Woo et al., 2018). The computational burden was reduced by (Wang et al., 2020) using efficient channel attention and 1 × 1 convolution. The most popular attention mechanism is the Squeeze-and Excitation module (Hu et al., 2018b), which can significantly improve the performance with a considerably low cost. The “channel shuffle” operator is used (Zhang and Yang, 2021) to enable information communication between the two branches. It uses a grouping strategy, which divides the input feature map into groups along the channel dimension.

heading: 2 RELATED WORK
text: There are two main problems which hinders the progress in this field: 1) Both spatial and channel attention as well as network using combination of two uses only local information while ignoring long range channel dependency, 2) The previous architectures fail to capture spatial information at different scales to be more robust and handle more complex problems. These two challenges were tackled by (Duta et al., 2020) and (Li et al., 2019) respectivly. The problem with these architectures is that the number of parameters increased considerably.
Pyramid Split Attention (PSA) (Zhang et al., 2021) has the ability to process the input tensor at multiple scales. A multi-scale pyramid convolution structure is used to integrate information at different scales on each channel-wise feature map. The channel-wise attention weight of the multi-scale feature maps are extracted hence long range channel dependency is done.
Non-Local block (Wang et al., 2018) is proposed to build a dense spatial feature map and capture the long-range dependency using non-local operations. (Li et al., 2019) used a dynamic selection attention mechanism that allows each neuron to adaptively adjust its receptive field size based on multiple scales of input feature map. (Fu et al., 2019) proposed a network to integrate local features with their global dependencies by summing these two attention modules from different branches.
Multi scale architectures have been used sucessfully for a lot of vision problems (?), (Hu et al., 2018b) and (Sagar and Soundrapandiyan, 2020). (Fu et al., 2019) adaptively integrated local features with their global dependencies by summing the two attention modules from different branches. (Hu et al., 2018a) used spatial extension using a depth-wise convolution to aggregate individual features. Our network borrows ideas from (Gao et al., 2018) which used a network to capture local cross-channel interactions.
The performance (in terms of accuracy) vs computational complexity (in terms of number of parameters) of the state of art attention modules is shown in Figure 1:
Our main contributions can be summarized as follows:
• A new attention module is proposed which aggregates feature information at various scales. Our network is scalable and can be easily plugged into various computer vision problems.
• Our network captures more contextual information using both spatial and channel attention at various scales.
• Our experiments demonstrate that our network outperforms previous state of the art with lesser computational cost.

heading: 3 METHOD

heading: 3.1 FEATURE GROUPING
text: Shuffle Attention module divides the input feature map into groups and uses Shuffle Unit to integrate the channel attention and spatial attention into one block for each group. The sub-features are aggregated and a “channel shuffle” operator is used for communicating the information between different sub-features.
For a given feature map X ∈ RC×H×W , where C,H,W indicate the channel number, spatial height, and width, respectively, shuffle attention module divides X into G groups along the channel dimension, i.e., X = [X1, XG], Xk ∈ RC/G×H×W . An attention module is used to weight the importance of each feature. The input of Xk is split into two networks along the channel dimension Xk1, Xk2 ∈ RC/2G×H×W . The first branch is used to produce a channel attention map by using the relationship of channels, while the second branch is used to generate a spatial attention map by using the spatial relationship of different features.

heading: 3.2 CHANNEL ATTENTION MODULE
text: The channel attention module is used to selectively weight the importance of each channel and thus produces best output features. This helps in reducing the number of parameters of the network. Let X ∈ RC×H×W denotes the input feature map, where the quantityH,W,C represent its height, width and number of input channels respectively. A SE block consists of two parts: squeeze and excitation, which are respectively designed for encoding the global information and adaptively recalibrating the channel-wise relationship. The Global Average Pooling (GAP) operation can be calculated by the as shown in Equation 1:
GAPc = 1
H ×W H∑ i=1 W∑ j=1 xc(i, j) (1)
The attention weight of the cth channel in the SE block can be written as denoted in Equation 2:
wc = σ (W1ReLU (W0 (GAPc))) (2)
where W0 ∈ RC×Cr and W1 ∈ RCr×C represent the fully-connected (FC) layers. The symbol σ represents the excitation function where Sigmoid function is usually used.
We calculate the channel attention map X ∈ RC×C from the original features A ∈ RC×H×W . We reshape A to RC×N , and then perform a matrix multiplication between A and the transpose of A. We then apply a softmax layer to obtain the channel attention map X ∈ RC×C as shown in Equation 3:
xji = exp (Ai ·Aj)∑C i=1 exp (Ai ·Aj)
(3)
where xji measures the ith channel’s impact on the jth channel. We perform a matrix multiplication between the transpose of X and A and reshape their result to RC×H×W . We also multiply the result by a scale parameter β and perform an element-wise sum operation with A to obtain the final output E ∈ RC×H×W as shown in Equation 4:
E1j = β C∑ i=1 (xjiAi) +Aj (4)

heading: 3.3 SPATIAL ATTENTION MODULE
text: We use Instance Normalization (IN) over Xk2 to obtain spatial-wise statistics. A Fc() operation is used to enhance the representation of Xk2. The final output of spatial attention is obtained by where
W2 and b2 are parameters with shape RC/2G×1×1. After that the two branches are concatenated to make the number of channels equal to the number of input.
A local feature denoted by A ∈ RC×H×W is fed into a convolution layer to generate two new feature maps B and C, respectively where B,C ∈ RC×H×W . We reshape them to RC×N , where N = H ×W is the number of pixels. Next a matrix multiplication is done between the transpose of C and B, and apply a softmax layer to calculate the spatial attention map S ∈ RN×N . This operation is shown in Equation 1:
sji = exp (Bi · Cj)∑N i=1 exp (Bi · Cj)
(5)
where sji measures the ith position’s impact on jth position. Next we feed feature A into a convolution layer to generate a new feature map D ∈ RC×H×W and reshape it to RC×N . We perform a matrix multiplication between D and the transpose of S and reshape the result to RC×H×W . We multiply it by a scale parameter α and perform a element-wise sum operation with the feature A to obtain the final output E ∈ RC×H×W as shown in Equation 2:
E2j = α N∑ i=1 (sjiDi) +Aj (6)

heading: 3.4 AGGREGATION
text: In the final part of the network, all the sub-features are aggregated. We use a ”channel shuffle” operator to enable cross-group information flow along the channel dimension. The final output of our module is the same size as that of input, making our attention module quite easy to integrate with other networks.
The whole multi-scale pre-processed feature map can be obtained by a concatenation way as defined in Equation 7:
F = Concat ([E1j , E2j ]) (7)
where F ∈ RC×H×W is the obtained multi-scale feature map. Our attention module is used across channels to adaptively select different spatial scales which is guided by the feature descriptor. This operation is defined in Equation 8:
atti = Softmax (Zi) = exp (Zi)∑S−1
i=0 exp (Zi) (8)
Finally we multiply the re-calibrated weight of multi-scale channel attention atti with the feature map of the corresponding scale Fi as shown in Equation 9:
Yi = Fi atti i = 1, 2, 3, · · ·S − 1 (9)

heading: 3.5 NETWORK ARCHITECTURE
text: We propose DMSA module with the goal to build more efficient and scalable architecture. The first part of our network borrows ideas from (Li et al., 2019) and (Zhang and Yang, 2021). An input feature mapX is splitted intoN parts along with the channel dimension. For each splitted parts, it has C0=CS number of common channels, and the ith feature map is Xi ∈ RC0×H×W . The individual features are fused before being passed to two different branches.
These two branches are comprised of position attention module and channel attention module as proposed in (Fu et al., 2019) for semantic segmentation. The second part of our network does the following 1) Builds a spatial attention matrix which models the spatial relationship between any
two pixels of the features, 2) A matrix multiplication between the attention matrix and the original features. 3) An element-wise sum operation is done on the resulting matrix and original features.
The operators concat and sum are used to reshape the features. The features from the two parallel branches are aggregated to produce the final output. The complete network architecture is shown in Figure 2:
We compare our network architecture with Resnet (Wang et al., 2017), SENet (Hu et al., 2018b) and EPSANet (Zhang et al., 2021) in Figure 3. We use our DMSA module in between 3× 3 convolution and 1× 1 convolution. Our network is able to extract features at various scales and aggregate those individual features before passing through the attention module.
The architectural details our proposed attention network is shown in Table 1:

heading: 3.6 IMPLEMENTATION DETAILS
text: We use Residual Networks (He et al., 2016) as the backbone which is widely used in literature for image classification on Imagenet dataset (Deng et al., 2009). Data augmentation is used for increasing the size of the dataset and the input tensor is cropped to size 224× 224. Stochastic Gradient Descent is used as the optimizer with learning rate of 1e−4, momentum as 0.9 and mini batch size of 64. The learning rate is initially set as 0.1 and is decreased by a factor of 10 after every 20 epochs for 50 epochs in total.
We use Residual Network along with FPN as the backbone network (Lin et al., 2017a) for object detection. The detectors we benchmark against are Faster RCNN (Ren et al., 2015), Mask RCNN(He et al., 2017) and RetinaNet (Lin et al., 2017b) on MS-COCO dataset (Lin et al., 2014). Stochastic Gradient Descent is used as the optimizer with a weight decay of 1e−4, momentum is 0.9, and the batch size is 16 per GPU for 10 epochs. The learning rate is set as 0.01 and is decreased by the factor of 10 every 10th epoch.
For instance segmentation we use Mask RCNN (He et al., 2017) with FPN (Lin et al., 2017a) as backbone. Stochastic Gradient Descent is used as the optimizer with a weight decay of 1e−4, momentum is 0.9, and the batch size is 4 per GPU for 10 epochs. The learning rate is set as 0.01 and is decreased by the factor of 10 every 10th epoch.

heading: 4 RESULTS

heading: 4.1 IMAGE CLASSIFICATION ON IMAGENET
text: We compare our network with previous state of the art on ResNet with 50 and 101 layers.
Our network shows the best performance in accuracy, achieving a considerable improvement compared with all the previous attention models along with lower computational cost. The comparision of our network against previous state of the art with ResNet50 as backbone is shown in Table 2:
The comparision of our network against previous state of the art with ResNet101 as backbone is shown in Table 3:

heading: 4.2 OBJECT DETECTION ON MS COCO
text: The comparision of our network using Faster RCNN detector against previous state of the art is shown in Table 4:
The comparision of our network using MASK RCNN detector against previous state of the art is shown in Table 5:
The comparision of our network using RetinaNet detector against previous state of the art is shown in Table 6:

heading: 4.3 INSTANCE SEGMENTATION ON MS COCO
text: We used Mask-RCNN (He et al., 2017) as benchmark on MS-COCO dataset (Lin et al., 2014). The comparision results of our network on instance segmentation using MS COCO dataset against previous state of the art is shown in Table 7:

heading: 4.4 ABLATION STUDY
text: The ablation studies of our architecture is shown in Table 8. The results are best obtained using instance normalization. Both removing Fc() and using 1× 1 Conv results in reduced performance as compared to the original network. The earlier is because Fc() is used to enhance the performance of individual features while latter is because number of channels in each sub-feature is too few, so it is not important to exchange information among different channels.

heading: 5 CONCLUSIONS
text: In this paper, we propose a novel Attention module named Dual Multi Scale Attention Network (DMSANet). Our network is comprised of two parts 1) first for aggregating feature information at various scales 2) second made up of position and channel attention modules in parallel for capturing global contextual information. After evaluating our network both qualitatively and quantitatively, we show that our network outperforms previous state of the art across image classification, object
detection and instance segmentation problems. The ablation experiments show that our attention module captures long-range contextual information effectively at various scales thus making it generalizable to other tasks. The best part of DMSANet attention module is that it is very lightweight and hence could be easily plugged into various custom networks as and when required.

heading: ACKNOWLEDGMENTS
text: We would like to thank Nvidia for providing the GPUs for this work.

Review summaries:
In this paper, the authors proposed a new attention module, named Dual Multi-Scale Attention. Three commonly used methods are combined: multi-scale, channel-attention, and position attention. By introducing the new module (which is a simple combination without any novelty) to ResNet, it SIGNIFICANTLY improves the performance, e.g, 75.20 to 80.02  (and 76.83 to 81.54 for ResNet101) on ImageNet, 36.4 to 41.4 on MS COCO.

In this paper, the authors propose a new attention module that shows better performance and lesser computation than most existing attention modules. Based on this module, the so-called Dual Multi Scale Attention Network is proposed. Several experiments are conducted to verify the performance on image classification, object detection and instance segmentation. Results shows that the proposed network has some advantages than previous works.