DMSANET: DUAL MULTI SCALE ATTENTION NETWORK
Authors: Abhinav Sagar
Venue: ICLR 2022 Submitted (2022)

Abstract: Attention mechanism of late has been quite popular in the computer vision community. A lot of work has been done to improve the performance of the network,
although almost always it results in increased computational complexity. In this paper, we propose a new attention module that not only achieves the best performance
but also has lesser parameters compared to most existing models. Our attention
module can easily be integrated with other convolutional neural networks because
of its lightweight nature. The proposed network named Dual Multi Scale Attention
Network (DMSANet) is comprised of two parts: the first part is used to extract
features at various scales and aggregate them, the second part uses spatial and
channel attention modules in parallel to adaptively integrate local features with
their global dependencies. We benchmark our network performance for Image
Classification on ImageNet dataset, Object Detection and Instance Segmentation
both on MS COCO dataset.

Under review as a conference paper at ICLR 2022
DMSANET: DUAL MULTI SCALE ATTENTION NET- WORK
Anonymous authors Paper under double-blind review
ABSTRACT
Attention mechanism of late has been quite popular in the computer vision com- munity. A lot of work has been done to improve the performance of the network, although almost always it results in increased computational complexity. In this pa- per, we propose a new attention module that not only achieves the best performance but also has lesser parameters compared to most existing models. Our attention module can casily integrated with other convolutional neural networks because of its lightweight nature. The proposed network named Dual Multi Scale Attention Network (DMSANet) is comprised of two parts: the first part used to extract features various scales and aggregate them, the second part uses spatial and channel attention modules in parallel to adaptively integrate local features with their global dependencies. We benchmark our network performance for Image Classification on ImageNet dataset, Object Detection and Instance Segmentation both on MS COCO dataset.
INTRODUCTION
The local receptive field the human eye has led to the construction convolutional neural networks which has powered much of the recent advances in computer vision. Multi scale architecture used in the famous InceptionNet (Szegedy et al., 2016) aggregates multi-scale information from different size convolutional kernels. Attention Networks has attracted a lot of attention recently as it allows the network to focus on only then essential aspects while ignoring the ones which are not useful (Li et al., 2019), (Cao et al., 2019) and (Li et al., 2019).
lot of problems have been successfully tackled using attention mechanism in computer vision like image classification, image segmentation, object detection and image generation. Most of the attention mechanisms can be broadly classified into two types channel attention and spatial attention, both of which strengthens the original features by aggregating the same feature from all the positions with different aggregation strategies, transformations, and strengthening functions (Zhang et al., 2021).
Some of the work combined both these mechanism together and achieved better results (Cao et al., 2019) and (Woo al., 2018). The computational burden was reduced by (Wang et al., 2020) using efficient channel attention and 1 convolution. The most popular attention mechanism is the Squeeze-and Excitation module (Hu al., 2018b), which can significantly improve the performance with a considerably low cost. The "channel shuffle" operator is used (Zhang and Yang. 2021) to enable information communication between the two branches. It uses a grouping strategy, which divides the input feature map into groups along the channel dimension.
RELATED WORK
There are two main problems which hinders the progress in this field: 1) Both spatial and channel attention as well as network using combination of two uses only local information while ignoring long range channel dependency, 2) The previous architectures fail to capture spatial information at different scales to be more robust and handle more complex problems. These two challenges were tackled by (Duta et al., 2020) and (Li et al., 2019) respectivly. The problem with these architectures that the number of parameters increased considerably.

---

Under review as a conference paper at ICLR 2022
Pyramid Split Attention (PSA) (Zhang al., 2021) has the ability process the input tensor at multiple scales. multi-scale pyramid convolution structure used to integrate information at different scales each channel-wise feature map. The channel-wise attention weight of the multi-scale feature maps are extracted hence long range channel dependency is done.
Non-Local block (Wang al., 2018) is proposed to build dense spatial feature map and capture the long-range dependency using non-local operations. (Li et al., 2019) used a dynamic selection attention mechanism that allows each neuron to adaptively adjust its receptive field size based multiple scales of input feature map. (Fu et al., 2019) proposed a network to integrate local features with their global dependencies by summing these two attention modules from different branches.
Multi scale architectures have been used sucessfully for lot of vision problems (?), (Hu et al., 2018b) and (Sagar and Soundrapandiyan, 2020). (Fu et al., 2019) adaptively integrated local features with their global dependencies by summing the two attention modules from different branches. (Hu et al., 2018a) used spatial extension using a depth-wing wise convolution to aggregate individual features. Our network borrows ideas from (Gao et al., 2018 8) which used network to capture local cross-channel interactions.
The performance (in terms of accuracy) vs computational complexity (in terms of number of parame- ters) of the state art attention modules shown Figure/
DMSANe150
D
79
FCANet 50
ESPANet50 Large
ESPANet100 Small
GCANet 50
ECANe150
SENet 101
77
ESPANet 50 Small
CBAM
ResNet 101
SENet50
76
ResNet50
75
Figure 1: Comparing the accuracy of different attention methods with ResNet-50 and ResNet-101 101 as backbone in terms of accuracy and network parameters. The circles reflects the network parameters and FLOPs of different models. Our proposed network achieves higher accuracy while having less model complexity.
Our main contributions be summarized as follows:
new attention module proposed which aggregates feature information at various scales. Our network is scalable and can be easily plugged into various computer vision problems.
Our network captures more contextual information using both spatial and channel attention at various scales.
Our experiments demonstrate that our network outperforms previous state of the art with lesser computational cost.
2

---

Under review as a conference paper at ICLR 2022
3 METHOD
3.1 FEATURE GROUPING
Shuffle Attention module divides the input feature map into groups and uses Shuffle Unit integrate the channel attention and spatial attention into one block for each group. The sub-features are aggregated and a "channel shuffle" operator is used for communicating the information between different sub-features.
For given feature map X E RC*HX where C, H. W indicate the channel number, spatial height, and width, respectively, shuffle attention module divides X into G groups along the channel dimension, i.c., X = 1, Xcl, X& RC/GxHXW An attention module is used to weight the importance of each feature. The imput of X split into two networks along the channel dimension XλI, Xx2 E DC/2GxHxW The first branch is used produce channel attention map by using the relationship of channels, while the second branch is used to generate a spatial attention map by using the spatial relationship of different features.
3.2 CHANNEL ATTENTION MODULE
The channel attention module is used to selectively weight the importance of each channel and thus produces best output features. This helps reducing the number parameters of the network. Let X E RC*H*W denotes the input feature map, where the quantity H, W, represent its height, width and number of imput channels respectively. A SE block consists two parts: squeeze and excitation, which are respectively designed for encoding the global information and adaptively recalibrating the channel-we -iise relationship. The Global Average Pooling (GAP) operation can be calculated by the as shown in Equation 1:
H
W
GAP
ΣΣ =1
x (i.j)
(I)
H
The attention weight of the eh channel in the SE block can be written denoted in Equation
We = (W₂ ReLU (Wo (GAP.))
where Wo E RCxOr and W1 E RCrC represent the fully-connected (FC) layers. The symbol represents the excitation function where Sigmoid function is usually used.
We calculate the channel attention map X E RCCC from the original features E CxxH We reshape A to C and then perform a matrix multiplication between and the transpose A. then apply softmax layer to obtain the channel attention map X E RCCC as shown in Equation 3:
exp (Ai A4) ES exp (A:-Ag)
JJj
where zijz measures the 7th channel's impact the jth channel. We perform a matrix multiplication between the transpose of X and A and reshape their result RC We also multiply the result by a scale parameter 3 and perform an element-wise wise sum operation with A to obtain the final output E E xx as shown in Equation 4:
E1 = 3 Σ Az) +
(4)
3.3 SPATIAL ATTENTION MODULE
We use Instance Normalization (IN) over X to obtain spat statistics. A Fe() operation is used enhance the representation of X12. The final output of spatial attention is obtained by where
3

---

Under review as a conference paper at ICLR 2022
W2 and by are parameters with shape RC/2GxIx1 After that the two branches concatenated to make the number of channels equal to the number of input.
A local feature denoted by A RC is fed into convolution layer generate two new feature maps and C, respectively where B, C E RCxHXW We reshape them to HC*N, where N H W is the number of pixels. Next matrix multiplication is done between the transpose of C' and B, and apply a softmax layer to calculate the spatial attention map S E RN*N, This operation shown in Equation
exp (B, Cÿ) 8j6 E exp (B, (B))
(5)
where Sys measures the hth position's impact on jzh position. Next we feed feature A into convo- lution layer generate new feature map D E RCxHxW and reshape it to RCXN We perform matrix multiplication between and the transpose of S and reshape the result to RC We multiply it by scale parameter and perform element-wist sum operation with the feature A to obtain the final output E E RCx xx as shown in Equation 2:
E2j a
(83s D.) + Aj
(6)
3.4 AGGREGATION
In the final part of the network, all the sub-features are aggregated. We use a "channel shuffle" operator to enable cross group information flow along the channel dimension. The final output of our module is the same size as that of input, making our attention module quite easy to integrate with other networks.
The whole multi-scale pre-processed feature map can be obtained by concatenation way as defined in Equation 7:
F Concal ([E1j, E25])
where F HXx is the obtained multi-scale feature map. Our attention module used across channels to adaptively select different spatial scales which is guided by the feature descriptor. This operation defined in Equation
exp (Zi)
all, Softmax (2,)
i- exp (Z,)
Finally we multiply the re-calibrated weight of multi-scale channel attention with the feature map of the corresponding scale F; as shown in Equation 9:
Y = F O att, i = 1,2,3...... ..5-
(9)
3.5 NETWORK ARCHITECTURE
We propose DMSA module with the goal to build more efficient and scalable architecture. The first part of our network borrows ideas from (Li al., 2019) and (Zhang and Yang, 2021). An input feature map is splitted into parts along with the channel dimension. For each splitted parts, has Co=Cs number of common channels, and the 4h feature map is X, RCxx HxW The individual features are fused before being passed to two different branches.
These two branches comprised of position attention module and channel attention module as proposed (Fu et al., 2019) for semantic segmentation. The second part our network does the following Builds spatial attention matrix which models the spatial relationship between any
4

---

Under review as a conference paper at ICLR 2022
two pixels of the features, 2) A matrix multiplication between the attention matrix and the original features. 3) An element-wiss wise sum operation is done on the resulting matrix and original features.
The operators concat and sum used reshape the features. The features from the two parallel branches are aggregated to produce the final output. The complete network architecture is shown in Figure 2:
Position Attention
(НXX))((НXX)
Split
Rechape
Reshape
CXHXX
Input
$2
fusion
concet
Oulput
CXHXW
Spltt
Reshape
Reshape
CXC
Channel Attention Module
Figure 2: A detailed Illustration of DMSANet
We compare our network architecture with Resnet (Wang et al., 2017), SENet (Hu et al., 20) and EPSANet (Zhang et al., 2021) in Figure 3. We use DMSA module in between 3 x 3 convolution and 1 1 convolution. Our network is able to extract features at various scales and aggregate those individual features before passing through the attention module.
ResNer block
SENet
EPSANet
DMSAWet
Impur
Ingui
Inpur
Impux
camv
conv
conv
cory
Ukd
PSA module
GUEy 3x3
cuny 1x1.
Dorv 1xt
cony 1xi
DMSA modile
Module
Figure 3: Illustration and comparison ResNet, SENet, EPSANet and our proposed DMSANet blocks.
The architectural details our proposed attention network is shown Table 1:
3.6 IMPLEMENTATION DETAILS
We use Residual Networks (He et al., 2016) as the backbone which is widely used in literature for image classification on Imagenet dataset (Deng al., 2009). Data augmentation is used for increasing the size of the dataset and the input tensor is cropped to size 224 x 224. Stochastic Gradient Descent used the optimizer with learning rate of le momentum 0.9 and mini batch size of 64. The learning rate is initially set as 0.1 and is decreased by a factor of 10 after every 20 epochs for 50 epochs in total.
5

---

Under review as a conference paper at ICLR 2022
Table 1: Network design of the proposed DMSANet.
Output
ResNet-50
DMSANet
112x112
7x7, 64
7x7, 64
56x56
3x3 max pool
3x3 max pool
[xx1, 64
1x1,
64
56x56
3x3, 64
x3
DMSA,
64
x 3
[xx1, 256
1x1,
256
[x11 128
1x1. 1287
28x28
3x3, 128
x4
DMSA,
128
1x1, 512
1x1, 512
1x1, 256
1x1, 256
14x14
3x3, 256
X6
DMSA,
256
x 6
1x1, 1024
1x1, 1024
[1x1, 512
1x1, 512
7x7
3x3, 512
x
DMSA, 512
x 3
1x1, 2048
1x1, 2048
1x1
7x7
GAP,1000-d fc
7x7
GAP,1000-d fc
We use Residual Network along with FPN the backbone network (Lin et al., 2017a) for object detection. The detectors we benchmark against are Faster RCNN (Ren al., 2015), Mask RCNN(He et al., 2017) and RetinaNet (Lin et al., 2017b) Gradient Descent is used as the optimizer with a weight decay of le momentum is 0.9, and the batch size is 16 per GPU for 10 epochs. The learning rate is set 0.01 and is decreased by the factor of 10 every 10th epoch.
MS-COCO dataset (Lin al., 2014). Stochastic
Mask RCNN (He al., 2017) with FPN (Lin et al., 2017a) as backbone. Stochastic Gradient Descent used as the optimizer with weight decay of le momentum is 0.9, and the batch size is per GPU for 10 epochs. The learning rate is 0.01 and decreased by the factor of 10 every 10th epoch.
For instance segmentation
4 RESULTS
4.1 IMAGE CLASSIHCATION ON IMAGENET
We compare our network with previous state of the art on ResNet with and 101 layers.
Our network shows the best performance in accuracy, achieving considerable improvement com- pared with all the previous attention models along with lower computational cost. The comparision of our network against previous state of the art with ResNet50 backbone is shown in Table 2:
The comparision of our network against previous state of the art with ResNet101 as backbone is shown in Table 3:
4.2
ОВJЕCТ DETECTION ON MS COCO
The comparision of our network using Faster RCNN detector against previous state of the art is shown in Table 4:
The comparision of our network using MASK RCNN detector against previous state of the art is shown in Table 5:
The comparision of our network using RetinaNet detector against previous state of the art is shown in Table 6:

---

Network Under review as a conference paper at ICLR 2022
Table 2: Comparison of various attention methods ImageNet with ResNet50 as backbone in terms of network pn millions), floating point operations per second (FLOPs), Top-1 and Top-5 Validation Accuracy(%). The best records are marked in bold.
Params
FLOPs
Top-1 Acc (%)
Top-5 Acc(%)
ResNet
25.56
4.12G
75.20
92.52
SENet
28.07
4.13G
76.71
93.38
CBAM
28.07
4.14G
77.34
93.69
ABN
43.59
7.18G
76.90
GCNet
28.11
4.13G
77.70
93.66
AANet
25.80
4.15G
77.70
93.80
ECANet
25.56
4.13G
77.48
93.68
FcaNet
28.07
4.13G
78.52
94.14
EPSANct(Small)
22.56
3.62G
77.49
93.54
(PSaNe((Larre)
27.90
4.72G
78.64
94.18
DMSANet 26.25
3,44G
80.02
94.27
Table 3: Comparison of various attention methods on ImageNet with ResNet 101 as backbone in terms of network parameters(in millions), floating point operations per second (FLOPs), Top-1 and Top-5 Validation Accuracy($).. The best records are marked in bold.
Network
Params
FLOPs
Top-1 Acc (%)
Top-5 Acc(%)
ResNet
44.55
7.85G
76.83
93.48
SENet
49.29
7.86G
77.62
93.93
BAM
44.91
7.93G
77.56
93.71
CBAM
49.33
7.88G
78.49
94.31
SRM
44.68
7.95G
78.47
94.20
ECANet
44.55
7.86G
78.65
94.34
AANet
45.40
8.05G
78.70
94.40
(PSmaN)(Small)
38.90
6.82G
78.43
94.11
EPsreet((aagge)
49.59
8.97G
79.38
94.58
DMSANet
42.29
7.11G
81.54
94.93
Table 4: Comparison of object detection results on COCO va12017 using Faster RCNN detector. The best records are marked in bold.
Backbone
Params(M)
GFLOPs
AP
APsu
APrs
APs
APM
APL
ResNer-50
41.53
207.07
36.4
58.2
39.5
21.8
40.0
46.2
SENet-50
44.02
207.18
37.7
60.1
40.9
22.9
41.9
48.2
ECANet-50
41.53
207.18
38.0
60.6
40.9
23.4
42.1
48.0
SANet-50
41.53
207.35
38.7
61.2
41.4
22.3
42.5
49.8
FcaNet-50
44.02
215.63
39.0
61.1
42.3
23.7
42.8
49.6
(PSANet-50(Small)
38.56
197.07
39.2
60.3
42.3
22.8
42.4
51.1
EPSaNett-0(Laage)
43.85
219.64
40.9
62.1
44.6
23.6
44.5
54.0
DMSANet
44.17
222.31
41.4
61.9
46.2
25.8
44.7
55.3
4.3 INSTANCE SEGMENTATION ON MS COCO
We used Mask-RCNN (He et al., 2017) benchmark MS-COCO dataset (Lin et al., 2014).
The comparision results of our network on instance segmentation using MS COCO dataset against previous state of the art is shown in Table 7:

---

Under review as a conference paper at ICLR 2022
Table 5: Comparison of object detection results COCO val2017 using Mask RCNN detector. The best records are marked in bold.
Backbone
Params(M)
GFLOPs
AP
APso
AP7s
AP's
APM
APL
ResNet-50
44.18
275.58
37.2
58.9
40.3
22.2
40.7
48.0
SENet-50
46.67
275.69
38.7
60.9
42.1
23.4
42.7
50.0
Non-local
46.50
288.70
38.0
59.8
41.0
GCNet-50
46.90
279.60
39.4
61.6
42.4
RCANet-50
44.18
275.69
39.0
61.3
42.1
24.2
42.8
49.9
SANet-50
44.18
275.86
39.4
61.5
42.6
23,4
42.8
51.1
FeaNet-50
46.66
261.93
40.3
62.0
44.1
25.2
43.9
52.0
EPSANe:--0(Smmll)
41.20
248.53
40.0
60.9
43.3
22.3
43.2
52.8
EPSANet:-5(Laare)
46.50
271.10
41.4
62.3
45.3
23.6
45.1
54.6
DMSANet 47.23
279.26
43.1
61.6
47.5
24.1
46.9
56.5
Table 6: Comparison of object detection results on COCO val2017 using RetinaNet detector. The best records are marked in bold.
Backbone
Params(M)
GFL.OPs
AP
A APso
A AP75
APs
APM
APL
ResNet-50
37.74
239.32
35.6
55.5
38.2
20.0
39.6
46.8
SENet-50
40.25
239.43
37.1
57.2
39.9
21.2
40.7
49.3
SANet-50
37.74
239.60
37.5
58.5
39.7
21.3
41.2
45.9
EPSAANet-5(Smma)
34.78
229.32
38.2
58.1
40.6
21.5
41.5
51.2
EPSAANet-5((Large)
40.07
251.89
39.6
59.4
42.3
21.2
43.4
52.9
DMSANet 41.63
270.17
40.2
59.8
44.1
23.5
44.8
54.8
Table Instance segmentation results different attention networks by using the Mask R-CNN on COCO. The best records are marked in bold.
Network
AP
APto
AP1s
APs
APM
APL
ResNet-50
34.1
55.5
36.2
16.1
36.7
50.0
SENet-50
35.4
57.4
37.8
17.1
38.6
51.8
GCNet
35.7
58.4
37.6
ECANet
35.6
58.1
37.7
17.6
39.0
51.8
FeaNet
36.2
58.6
38.1
SANet
36.1
58.7
38.2
19.4
39.4
49.0
EPSA 50t-ll(
35.9
57.7
38.1
18.5
38.8
49.2
37.1
59.0
39.5
19.6
40.4
50.4
DMSANet
37.4
61.1
40.7
19.3
40.9
51.7
4.4 ABLATION STUDY
The ablation studies of our architecture is shown Table 8. The results are best obtained using instance normalization. Both removing Fe() and using x Conv results in reduced performance as compared to the original network. The earlier because Fo() is used to enhance the performance of individual features while latter is because number of channels in each sub-feature is too few, so it is not important to exchange information among different channels.
5 CONCLUSIONS
In this paper, we propose novel Attention module named Dual Multi Scale Attention Network (DMSANet). Our network is comprised of two parts 1) first for aggregating feature information at various scales 2) second made up of position and channel attention modules in parallel for capturing global contextual information. After evaluating our network both qualitatively and quantitatively. show that network outperforms previous state of the art across image classification, object
8

---

Under review as a conference paper at ICLR 2022
Table 8: Performance comparisons of network using ResNet 50 backbone with four options (i.c., original, using Batch Normalization, using Group Normalization, using shuffle normalization, eliminating FcO and using Conv to replace Fe(( on ImageNet- Ik in terms of GFLOPs and Top-1/Top-5 accuracy (in %).. The best records are marked in bold.
Methods
GFLOPs
Top-1 Aco(%)
Top-5 Aco(%)
origin
3,44
80.02
94.27
W BN
3.82
77.37
93.80
W GN
3.56
77.61
92.89
SN
3.51
78.16
93.48
W/O F.O)
4.07
77.64
93.18
x Conv
3.55
78.69
93.71
detection and instance segmentation problems. The ablation experiments show that attention module captures long-range contextual information effectively at various scales thus making generalizable other tasks. The best part of DMSANet attention module is that it is very lightweight and hence could be easily plugged into various custom networks as and when required.
ACKNOWLEDGMENTS
We would like to thank Nvidia for providing the GPUs for this work.
REFERENCES
I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. Le. Attention augmented convolutional networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3286-3295, 2019.
Y. Cao, Xu, S. Lin, F. Wei, and H. Hu. Genet: Non-local networks meet squeeze-excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icce, 2009.
I. C. Duta, L. Liu, F. Zhu, and I.. Shao. Pyramidal convolution: rethinking convolutional neural networks for visual recognition. arXiv preprint arXiv:2006. S0t.::2000..13388, 2020.
J. Fu, J. Liu, H. Tian, Li Y. Bao, Z. Fang, and H. Lu. Dual attention network for scene segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pages 3146-315 3154, 2019.
H. Fukui, T. Hirakawa, Yamashita, and H. Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 10705-10714, 2019.
H. Gao, Z. Wang, and S. Ji. Channelnets: Compact and efficient convolutional neural networks via channel-wise convolutions. arXiv preprint 1809 1800.0139, 2018.
S. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. H. Torr. Res2net: new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference compuler vision and pattern recognition, pages 770-778, 2016.

---

Under review as a conference paper at ICLR 2022
K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. arXiv preprint arXiv:1810.122 12348 2018a.
J. Hu, L. Shen, and G. Sun. Squceze-and--xcitatten networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7132-7141, 2018b.
X. Li, W. Wang, X. Hu, and J. Yang. Selective kernel networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 510-519, 2019.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017a.
T.-Y. Lin, Goyal, R. Girshick, K. He, and Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pages 2980-2988. 2017b.
S. Ren, K. He, R. Girshick, and Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015.
A. Sagar. Aa3dnet: Attention augmented real time 3d object detection. arXiv preprint arXiv:2107.12137, 2021.
A. Sagar and R. Soundrapandiyan. Semantic segmentation with multi scale spatial attention for self driving cars. arXiv preprint arXiv:2007. 12685, 2020.
H. Sang, Q. Zhou, and Zhao. Peanet: Pyramid convolutional attention network for semantic segmentation. Image and Vision Computing, 103:103997, 2020.
C. Szegedy, V. Vanhoucke, S. loffe, Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
Wang, M. Jiang, Qian, Yang, C. Li, H. Zhang, Wang, and Tang. Residual attention network for image classification. In Proceddings of the IEEE conference computer vision and pattern recognition, pages 3156-3164, 2017.
Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu. Eca-net: Efficient channel attention for deep convolutional neural networks, 2020.
X. Wang, R. Girshick, A. Gupta, and He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 7794-7803, 2018.
S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
Y. Wu and K. He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pages 3-19, 2018.
H. Zhang, C. Wu, Z. Zhang, Zhu, H. Lin, Z. Zhang, Y. Sun, T. He, J. Mueller, R. Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004. 08955, 2020.
H. Zhang, Zu, J. Lu, Zou, and D. Meng. Epsanet: An efficient pyramid split attention block on convolutional neural network. arXiv preprint arXiv:2 105. / 2021.
Q. Zhang and Y.-B. Yang. Sa-net: Shuffle attention for deep convolutional neural networks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2235-2239 IEEE, 2021.
10

---

Under review as a conference paper at ICLR 2022
H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
Z. Zhong, Z. Q. Lin, R. Bidart, X. Hu, I. B. Daya, Z. Li, W.-. Zheng, J. Li, and A. Wong. Squeeze- and-attention networks for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13065-13074, 2020.


Reviews:
[{"review_id": "VIuifvJngVe", "reviewer": "Reviewer_gSEr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the papers are as follow :\n\n1-  Clarity of claims\n\n2-  Extensive experiments\n\nThe weaknesses  are as follow : \n\n1- Similarities and dissimilarities with most common related works are not enumerated. Namely, comparison with dual attention network for scene segmentation\n\n2- One reference is not well formated. Absence of date and proceeding title.\n\n3-  The paper claims that the proposed model captures well multi-scale information. However, no mention and comparison have been conducted w.r.t U-net models which are targeted to achieve multi-scale representation and trading-off local and global information.\n\n4-  Figures and tables lack of information in the captions. Moreover, State-of-the-art methods are not referenced in the captions.\n\n5-  Evaluation metrics for each task are not described.\n\n6- Lack of analysis and discussion of the results. In the ablation study the table are not well discussed and analyzed. This the major drawback of the section\n", "summary_of_the_paper": "The paper proposes a kind of multi-scale attention module capable at capturing spatial and channel attention informations considering long-range dependencies.\n\n The claims are corroborated with experimental studies for computer vision tasks, including Image classification and regression tasks.", "summary_of_the_review": "Following the aforementioned consideration, l recommend to reject the paper \"3: reject, not good enough\" except if the authors make major changes, mainly providing an extensive analysis and discussion of the empirical results and considering U-net in the ablation study.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AMJW4tuAvF8", "reviewer": "Reviewer_QHkT", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Methodology**\n* The novelty is very limited. The proposed method is a simple combination of previous methods, and basically, no substantial new technique is introduced.\n\n**(Unconvincing) Experiments**\n* The experimental results are too good to be true. While the idea is simple and has been explored by many related works, authors still achieve unbelievable improvements over all these vision tasks. \n* Also, the baseline is significantly lower than the original results: the image classification result of ResNet101 only achieves 76.83% in this paper, while the standard number should be around 77.37% (see torchvision models: https://pytorch.org/vision/stable/models.html)\n* The results in Table 8 are confusing.  Why does removing FC result in larger GFLOPs? \n* Could authors also report the training/inference efficiency? I assume it would be super slow.\n\n**Presentation**\n* The writing is rather poor, as well as the presentation. Too many typos. \"combined both these mechanism\", \"respectivly\", \"The channel-wise attention weight of the multi-scale feature maps are extracted ...\", \"comparision\", etc. These simple mistakes show this paper is far from well-prepared.", "summary_of_the_paper": "This paper proposes a new method, Dual Multi Scale Attention (DMSA), as a plug-in module for CNN backbone. With the help of DMSA, authors claim to achieve huge improvements in image classification, object detection, and segmentation task.\n\n\n", "summary_of_the_review": "\nStrong reject given the poor writing, limited novelty, and unconvincing experimental results of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "3RddyLH2-P_", "reviewer": "Reviewer_wSEj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. No novelty, not limited.  This is a simple combination of multi-scale and DANet.\n2. Extremely poor paper organization, see Fig.1, Fig. 2, and Fig. 3.\n3. Bad presentation, e.g, \"Multi scale architectures have been used sucessfully for a lot of vision problems (?), (Hu et al., 2018b) and (Sagar and Soundrapandiyan, 2020).\"  Even in this one sentence, there are a lot of mistakes. What is \"(?)\"? \"sucessfully\" -> \"successfully\"\n4. Could authors explain why a simple \"multi-scale + dual attention\" idea could achieve such big improvements over all vision tasks? Based on my experience, that is impossible.\n\n", "summary_of_the_paper": "In this paper, the authors proposed a new attention module, named Dual Multi-Scale Attention. Three commonly used methods are combined: multi-scale, channel-attention, and position attention. By introducing the new module (which is a simple combination without any novelty) to ResNet, it SIGNIFICANTLY improves the performance, e.g, 75.20 to 80.02  (and 76.83 to 81.54 for ResNet101) on ImageNet, 36.4 to 41.4 on MS COCO.", "summary_of_the_review": "I would recommend strong reject for this paper, which introduces NO novelty and the experimental results are suspect.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "0v1upQ4OfkM", "reviewer": "Reviewer_4y42", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nMost of the proposed techniques are based on previous works, thus it is easy to follow. Three kinds of experimental verifications are provided, including image classification, object detection and instance segmentation. Better results are achieved on typical benchmarks.\n\nWeaknesses:\n1. Limited technical contributions \nIn fact, the proposed module largely borrows ideas from existing works, such as SENet, ResNet, shuffle attention, (Fu et al., 2019),  (Li et al., 2019) and (Zhang and Yang, 2021). I think this work is a combination of previous works. Most of the proposed modules have already been used in previous attention methods. The authors also admit these facts. Overall, I think this work is very limited in its novelty.\n\n2. Insufficient evaluations\nIn this work, DMSA consists of many sub-modules. The authors list the ablations in Tab.8. However, there are no visual comparisons. Why the sub-modules are worked? How about their effects on typical image examples? In facts, the authors do not provide any visual examples to clarify the effects of the proposed attention module. Besides, there are no examples on image classification, object detection, instance segmentation. The performance improvements are very small when compared with other attention modules.", "summary_of_the_paper": "In this paper, the authors propose a new attention module that shows better performance and lesser computation than most existing attention modules. Based on this module, the so-called Dual Multi Scale Attention Network is proposed. Several experiments are conducted to verify the performance on image classification, object detection and instance segmentation. Results shows that the proposed network has some advantages than previous works.", "summary_of_the_review": "This work provides a new attention module to improve the performance of backbone networks. The main concerns are the limited techinical contributions and insufficient experiments. In my view, this work is much lower than the standard the ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is proposed to improve base CNN models by dual multi-scale attention module. To achieve a better feature representational ability, authors consider the multi-scale mechanism from both channel-dimension and spatial-dimension. The proposed method has been verified on several benchmarks, including ImageNet and MS COCO. However, all reviewers consider rejecting this paper because this work lacks novelty, the results are suspicious, and the writing is poor. No responses are submitted by authors to address the reviewers' concerns.
Decision: Reject