AASEG: ATTENTION AWARE NETWORK FOR REAL TIME SEMANTIC SEGMENTATION
Authors: Abhinav Sagar
Venue: ICLR 2022 Submitted (2022)

Abstract: In this paper, we present a new network named Attention Aware Network (AASeg)
for real time semantic image segmentation. Our network incorporates spatial and
channel information using Spatial Attention (SA) and Channel Attention (CA)
modules respectively. It also uses dense local multi-scale context information
using Multi Scale Context (MSC) module. The feature maps are concatenated
individually to produce the final segmentation map. We demonstrate the effectiveness of our method using a comprehensive analysis, quantitative experimental
results and ablation study using Cityscapes, ADE20K and Camvid datasets. Our
network performs better than most previous architectures with a 74.4% Mean IOU
on Cityscapes test dataset while running at 202.7 FPS.

Under review as a conference paper at ICLR 2022
AASEG: ATTENTION AWARE NETWORK FOR REAL TIME SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we present new network named Attention Aware Network (AASeg) for real time semantic image segmentation. Our network incorporates spatial and channel information using Spatial Attention (SA) and Channel Attention (CA) modules respectively. also uses dense local multi-scale context information using Multi Scale Context (MSC) module. The feature maps concatenated individually to produce the final segmentation map. We demonstrate the effee- tiveness of our method using comprehensive analysis, quantitative experimental results and ablation study using Cityscapes, ADE20K and Camvid datasets. Our network performs better than most previous architectures with 74.4% Mean IOU on Cityscapes test dataset while running at 202.7 FPS.
INTRODUCTION
Semantic Segmentation is a fundamental problem in computer vision where the goal is to label each and every pixel in the image to its appropriate class. Since it is required real world settings like robots and autonomous vehicles, hence there is need to balance the speed vs performance tradeoff. Fully Convolutional Network (FCN) (Long et al., 2015) composed of convolutional layers was one of the first works to get strong semantic representation.
be deployed in
However, this method was not able to capture boundary information accurately. Atrous convolutions (Yu and Koltun, 2015) at the last several stages of their network was used to give feature maps with strong semantic representation, thus solving the problem with FCN based architectures. However, this comes at at the cost of increased computational complexity. For real world navigation tasks like autonomous driving, there is a need to improve the Frame Per Second(FPS).
Chen al. (2017) and (Yu and Koltun, 2015) used dilated convolutions to increase the recpetive field while maintaining the number of parameters. SegNet (Badrinarayanan et al., 2017) utilizes small network structure and the skip-connected method to achieve improved FPS. (Mehta et al., 2018), (Paszke et al., 2016) and (Poudel et al., 2019) proposed unique approaches to tackle real time semantic segmentation problem.
RELATED WORK
(Fu et al., 2019) introduces spatial-wise and channelwise attention modules to enhance the reepetive field (Paszke et al., 2016) trims lot of convolution filters to reduce computation. ICNet (Zhao et al., 8a) proposed an image cascade network using multiresolution branches. (landola et al., 2016) allows the neural network to find the critical channels of the feature map and select the most suitable channels by itself.
ESPNet (Mehta et al., 2018) introduces efficient spatial pyramid (ESP), which brings great improvement both speed and performance. Bilateral Segmentation Network (BiSeNet) (Yu et al., 2018a) used two parts: Spatial Path (SP) is used to get with the loss of spatial information and Context Path (CP) for compressing the receptive field. (Yu et al., 2020) used multi-path framework to combine the low-level details and high-level semantics.
(Li et al., 2019b) utilizes a light backbone to speed up its network and a multi scale feature aggregation to improve accuracy. SwiftNet (Orsic et al., 2019) used lateral connections restore

---

Under review as a conference paper at ICLR 2022
the prediction resolution while maintaining the speed. (Lin et al., 2017) multipath refinement network to refine the feature but ignores the global context feature.
The speed-Accuracy performance comparison of state of the methods on the Cityscapes test set is shown in Figure
78
STDG2-Seg75
76
MASH
Moan IDU (9)
BiSeNet V2
74
STDCI-Sag7E Seg75
HMERg
STOC2- SegS0
CABNet
70
DTANGI
FasterSeg
STECI-Segg0
68
BiSeNetvi vd
FASTSONN
66
DFANst
20
100
120
200
220
240
Inference Speed (FPS)
Figure 1: Speed- performance comparison on the Cityscapes test set. Our approach achieves higher mean IOU while still being faster than most existing methods.
We summarize our main contributions as follows:
We propose novel Attention Aware Network (AASeg) for real time semantic segmentation.
Our network is comprised of three parts: Spatial Attention(SA) module to capture the spatial dependencies of the feature maps, Channel Attention(CA) module extract high level semantic information and Multi Scale ConteXt(MSSC) module to learn the information flow between feature maps of consecutive levels.
Detailed experiments and analysis indicate the efficacy of our proposed network in not only improving the performance but also FPS. We achieve results on par with previous state of the using Cityscapes, Camvid and on ADE20K datasets.
BACKGROUND
SPATIAL INFORMATION
The spatial information of the image is important to predict the detailed output for semantic segmen- tation. Modern existing approaches uses encoding of spatial information. DUC (Wang et al., 2018), PSPNet (Zhao et al., 2017), DeepLab v2 (Chen al., 2017) use the dilated convolution preserve the spatial size the feature map.
3.2 CONTEXT INFORMATION
Semantic segmentation requires context information to generate a high-quality result. Most of the used methods enlarge the receptive field fuse different contextual information. (Chen et al., 2017), (Wang et al., 2018) and (Yu and Koltun, 2015) use different dilation rates in convolution layers to capture different scale contextual information. In (Chen et al., 2017), an "ASPP" module is used to capture context information different receptive field. PSPNet (Zhao et al., 2017) applics a "PSP" module which contains several different scales of average pooling layers.
2

---

Under review as a conference paper at ICLR 2022
3.3 ATTENTION MECHANISM
(Hu et al., 2018) applied channel attention for image recognition and achieve the state-of-the-art. (Yu al., 2018b) proposed a network that learns the global context attention and revise the features. Multi scale network along with a custom attention module was proposed by (Sagar and Soundrapandiyan, 2020) for semantic segmentation.
3.4 FEATURE FUSION
The spatial information captured by the Spatial Path consists of rich detailed information. The output feature of the Context Path is made up of contextual information. Feature fusion was used to achieve state of the art results on image classification, object detection and instance segmentation using DMSANet (Sagar, 2021b).
4 PROPOSED METHOD
4.1 DATASET
The following datasets have been used to benchmark our results:
Cityscapes is used for urban street segmentation. The 5000 annotated images are used in our experiments which are divided into 2975, 500 and 1525 images for training, validation, and testing respectively.
2. ADE20K This dataset contains labels of 150 object categories. The dataset includes 20k,2k and 3k images for training, validation and testing respectively.
3. Cam Vid This dataset is used for semantic segmentation for autonomous driving scenarios. It is composed of 701 densely annotated images.
4.2 NETWORK ARCHITECTURE
The fundamental goal in semantic segmentation is to map an RGB image semantic map Y E #RRxxxx XxC with the same spatial resolution H W, where C is the number of classes of objects present in image. The input image is converted to a set of feature maps Fi where 1=,,..,,3 from each network stage, where Fj RRI is Ci-dimensional feature map.
RHxWx3
Our network dosen't t use any backbone to extract features from the input image unlike many previous architectures. The input image is first passed through block comprising of convolutional, batch normalization and ReLU activation function.
We express convolution layer (²(x)) as follows:
W" (a) Waxn or+b
where @ represents the convolution operator, Wnxx represents the n x n convolutional kernel, represents the input data and b represents the bias vector.
4.3 MULTI SCALE CONTEXT MODULE
The feature map after passing through convolutional block is split to three different parts with x convolution, 3x3 convolution and 5x5 convolution respectively. The individual feature maps are then fused together. The output fused feature map is first convolved with 1 x convolution to reduce the number of channels from 2048 to 256. The feature map produced of size H W x ne- where H. W are the height and width of the feature map, and n. denotes the number of channels.
The input feature map is convolved with dilated convolution layers with increasing dilation rates 3, 6 and 12. The dilated convolution layer input every stage is formed by concatenating the input feature map with the outputs from previous convolutions. At the final step, the outputs from the three dilated convolutions concatenated with the input feature map.

---

Under review as a conference paper at ICLR 2022
4.4 SPATIAL ATTENTION MODULE
The spatial attention module is used for capturing the spatial dependencies of the feature maps. The spatial attention (SA) module in network is defined below:
fSA(#) Ssigmoid (W2 (fReLy (#))))))
where Wi and W2 denotes the first and second 1 convolution layer respectively, æ denotes the input data, Sigmoid denotes the sigmoid function, fRLL denotes the ReLu activation function.
The spatial attention module used in this work is shown in Figure
Spetial Attention Module
(HXW))X(HXM)
Inpur feature map
Dutput feature map
Reshaape
Reshappe
Figure 2: Details of our spatial attention module
4.5 CHANNEL ATTENTION MODULE
The channel attention module is used for extracting high level multi-scale semantic information. The channel attention (CA) module our network is defined below:
fCA(z) = faigmord (W2(fRel (W1f (Wif WifhugPoot (z))))
(3)
where Wi and W2 denotes the first and second x convolution layer, x denotes the input data. Pogg denotes the global average pooling function, J Siigmoi denotes the Sigmoid function, LU denotes ReLU activation function.
The channel attention module used in this work is shown in Figure 3:
Reshape
Rechape
Uutput
map
map
Channel Attention Module
Figure 3: Details of our channel attention module
4.6 AGGREGATION
We denote the concatenation operation as follows:
Feoneat
.72 4
(4)
where @ represents the concatenation operator and æı, and Ø3 represents the features of the two
branches. The AASeg module can be denoted follow:
I AAASeg = ((fsA (Xconcat) & Ieoncat) @ (feA (Fconent) @ Deaneat) @ (fMSC (Dconcat) & Feoncat)
(5)
4

---

Under review as a conference paper at ICLR 2022
where represents the concatenation operator, fcA represents the channel attention module men- tioned in Equation 2, fSA represents the spatial attention module, faise represents the multi scale attention and Fconeat represents the combined feature.
we use ConvX; to denote the operations of 7th block. Therefore, the output of ht block is calculated follows:
2; ConeX; (
where Cone includes one of each convolutional layer, batch normalization layer and ReLU activation layer, k; is the kernel size of convolutional layer, 3i-1 and æ; are the input and output of 7t block. Fusion operation is used to combine high-level features with low-level features using Equation 7:
Toutpat
(I1,
The overall structure of our proposed AASeg network is shown in Figure 4.
Spatial Attention
Input Image
corv relu
Fusion
1x1
d=3
d=
d=12
Concat
Output Image
S3
Channel Attention
Figure Architecture of proposed AASeg network architecture. "c" denotes concatenation.
4.7 LOss FUNCTIONS
We use cross-entropy loss function to weigh the difference between the forward propagation result of the network and the ground truth of the samples. The cross-entropy loss is calculated defined in Equation 8:
N (yn log Hn + (1 yn) log (1 .n.)
Leee
where N denotes the total number of samples, Vn, denotes the probability that the forward propagation result true, and 1-y/, denotes the probability that the forward propagation result is false.
We also use the auxiliary supervision loss Lawx improve the model performance thus making easier to optimize. The auxiliary loss can be defined as:
K
exp (piNz)
Lava
Σ
Σ
(g) = ke) log
(9)
BN
k Σm= exp (w)mm)
1=1
1,
g k
7 = =k)= =
(10)
0, otherwise
5

---

Under review as a conference paper at ICLR 2022
where B is the mini batch size, N is the number of pixels in every batch; K is the number of categories; Pijk is the prediction of the j pixel in the 7t sample for the hthh class, [gs = k) is function which is defined in Equation 10.
The class attention loss Lts from channel attention module also used. The class attention loss is defined as follows:
B N
K I (o)
exp
C
Ld
k) log
(1I)
BN
Σ exp (a).m)
1=1
where aijk is the value generated of the class attention map of the jth pixel in the 7th sample for the hthh class. We combine the three terms to balance final loss term as follows:
C = >aCee + A2Lels + AgLauz
(12)
where X1, X2 and X3 are 1, 0.5 and 0.5 balance the loss.
4.8 IMPLEMENTATION DETAILS
PyTorch deep learning framework is used carry out experiments. Our neural networks is trained with stochastic gradient descent (SGD) as optimizer, batch size of 16, momentum of 0.9 and weight decay of 54 The network is trained for 20K iterations with initial learning rate value of 0.01. The, "poly" learning rate policy is used to decay the initial learning rate while training the model to reduce over-fitting.
Data augmentation operations like random horizontal flip, random resizing with scale range of [1.0, 2.0], and random cropping with crop size of 1024 1024 was done.
4.9 EVALUATION METRICS
For quantitative evaluation of the performance of our network, mean of class-wise intersection-over- union (mloU) is used for bench-marking the performance, number of float-point operations (FLOPs) and frames per second (FPS) used for bench-marking the speed.
RESULTS
We present the segmentation accuracy and inference speed of our proposed method on Cityscapes validation and test set in Table 1. We use the training set and validation set to train models before submitting to Cityscapes online server
The qualitative results Cityscapes validation set presented in Figure 5:

---

Under review as a conference paper at ICLR 2022
Table 1: Comparisons with other state-of-the-art methods using Cityscapes dataset. no indicates the method do not have any backbone for training and testing. Best results are highlighted in bold.
Model
Resolution
Backbone
mloU val(%)
mloU test(())
FPS
DeANet B
1024 x 1024
Xception
67.1
120
DFANet A
1024 1024
Xception A
71.3
100
BiSeNetVI
768 x 1536
Xception39
69.0
68.4
105.8
BiSeNetV1
768 1536
ResNet18
74.8
74.7
65.5
SFNet
1024 x 2048
DFI
74.5
121
BiSeNetV2
512 x 1024
no
73.4
72.6
156
B1S
512 x 1024
75.8
75.3
47.3
FasterSeg
1024 2048 no
73.1
71.5
163.9
STDCI-Seg50
512 x 1024
STDCI
72.2
71.9
250.4
STDC2-Seg50
512 x 1024
STDC2
74.2
73.4
188.6
STDCI-Seg75
768 x 1536
STDCI
74.5
75.3
126.7
STDC2-Seg75
768 x 1536
STDC2
77.0
76.8
97.0
AASeg
512 x 1024
74.8
74.4
202.7
Figure Visualized segmentation results on Cityscapes validation set. The three columns left-to-right refer input image, ground truth, prediction from network.
We evaluate our method on Cam dataset. The comparison results with other methods is shown in Table 2:
7

---

Under review as a conference paper at ICLR 2022
Table 2: Comparisons with other state-of-the-ar methods Cam indicates the method do not have a backbone for both training and testing. Best results are highlighted in bold.
Model
Resolution
Backbone
mloU(%)
FPS
DFANet A
720 x 960
no
64.7
120
DFANet B
720 x 960
59.3
160
BiSeNetV1
720 x 960
Xception39 ResNet18
65.6
175
BiSeNetV1
720 960
68.7
116.3
BiSeNetV2
720 x 960 no
72.4
124.5
BiSeNetV2-L
720 x 960
73.2
32.7
SFNet
720 x 960
DF2
70.4
134.1
SFNet
720 x 960
ResNet-18
73.8
35.5
SFNet
720 x 960
DF2
70.4
134.1
SFNet
720 x 960
ResNet-18
73.8
35.5
STDCI-Seg
720 x 960
STDCI
73.0
197.6
STDC2-Seg
720 x 960
STDC2
73.9
152.2
AASeg
720 x 960
73.5
188.7
The comparison of our network with previous state of the art methods for semantic segmentation using ADE20K validation is shown Table
Table 3: Performance comparison with results reported on ADE20K validation set. Best results are highlighted in bold.
Method
Mean loU(%)
GFLOPs
PSPNet50
42.78
167.6
PSPNet101
43.29
238.4
SFNet
42.81
75.7
SFNet
44.67
94.0
DCANet
45.49
AASeg
46.29
80.26
ABLATION STUDY
We explore the effect of 3 different upsamling operations: bilinear upsampling, deconvolution and nearest neighbor upsampling as shown in Table 4. The best results are obtained using bilincar upsampling but there isn't considerable difference.
Table 4: Ablation study on Upsampling operation in our network using Cityscapes validation set. Best results are highlighted in bold.
Method
mloU (%)
bilinear upsampling
79,2
deconvolution
78.5
nearest neighbor
78.3
We also try the various kernel size in Table 5. Larger kernel size of 7 x is also tried which results in a similar performance but increases computational complexity.
We use increasing sequence of dilation rates in network, which is 0, 2, 3. It seems large dilation rates increases the performance but comes at the cost of increased number of parameters, hence decrease in FPS. This effect is shown in Table 6:
8

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation study on kernel size k in our network using Cityscapes validation set. Best results are highlighted in bold.
Method
mloU (%)
GFlops
k 1
79.2
118.2
k 3
79.4
120.8
k 5
79.5
127.5
k
79.5
136.1
Table 6: Ablation study results on Cityscapes validation set. FPS are estimated for an input image of resolution 512 1024. Best results are highlighted in bold.
Model
mloU (%)
FPS
Parameters (M)
AASeg-baseline
79.2
118.2
0.76
AASeg-w/o dilation
77.4
121.3
0.80
AASeg-(r=1)
79.8
115.5
0.79
A AASeg-(r=2)
80.1
102.7
0.84
A AASeg-(r33)
80.2
88.6
0.90
CONCLUSIONS
In this paper, present network named Attention Aware Network (AASeg) for real time semantic segmentation. We use Spatial Attention(SA) and Channel Attention(CA) modules to enhance features of objects without adding extra computational cost. The fused spatial-channel attention along with Multi Scale Context(MSC) module enables network to extract discriminative and robust features targets background. We test our network using Cityscapes, ADE20K and Camvid datasets. Our network balances speed-perte preed-performance tradeoff at par with previous state of the art network architectures. the future, we would like to use multi scale attention module for instance segmentation problem.
ACKNOWLEDGMENTS
We would like to thank Nvidia for providing the GPUs.
REFERENCES
V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder archi- tecture for image segmentation. IEEE Iransactions on paltern analysis and machine intelligence, 39(12):2481-2995 2017.
L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 100.:00677.5.7 2017.
W. Chen, X. Gong, Liu, Q. Zhang, Y. Li, and Z. Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv preprint arXiv: 1et.: 10917, 2019.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and
B. Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-28 258.255.200 Icce, 2009.
M. Fan, S. Lai, J. Huang. X. Wei, Z. Chai, J. Luo, and X. Wei. Rethinking bisenet for real-time semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9716-9725, 2021.

---

Under review as a conference paper at ICLR 2022
J. Fu, Liu, H. Tian, Li, Y. Bao, Z. Fang, and H. Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
K. Han, Wang, Q. Tian, Guo, C. Xu, and C. Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1580-1589, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016a.
K. He, X. Zhang, S. Ren, and Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016b.
Hu, L. Shen, and G. Sun. Squeeze-and--xciittion networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7132-7141, 2018.
Hu, F. Caba, O Wang, Z. Lin, Sclaroff, and F. Perazzi. Temporally distributed networks for fast video semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8818-88 8827, 2020.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu. Ccnet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 603 612,,,1122220 2019.
N. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016.
L. Jing, Y. Chen, and Y. Tian. Coarse-to-fing semantic segmentation from image-level labels. IEEE Transactions on Image Processing, 29:225 236, 2019.
G. Li, Yun, J. Kim, and Kim. Dabnet: Depth-wise asymmetric bottleneck for real-time semantic segmentation. arXiv preprint arXiv:
2019a.
H. Li, P. Xiong, H. Fan, and Sun. Dfanet: Deep feature aggregation for real-time semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9522-9531, 2019b.
P. Li, X. Dong, X. Yu, and Y. Yang. When humans meet machines: Towards efficient segmentation networks. Proceedings of the British Machine Vision Conference (BMVC), 2020a.
X. Li, A. You, Z. Zhu, H. Zhao, M. Yang, K. Yang, S. Tan, and Y. Tong. Semantic flow for fast and accurate scene parsing. In European Conference Computer Vision, pages 775-793. Springer, 2020b.
C.-Y. Lin, Y.-C. Chiu, H.-F. Ng, T. K. Shih, and K.-H. Lin. Global-and-local context network for semantic segmentation of street view images. Sensors, 20(10):2007, 2020.
G. Lin, A. Milan, C. Shen, and I. Reid. Refinenet: Multi-path refinement networks for high- nigh-resolution semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1925-1934, 2017.
Y. Liu, C. Xu, and X. Jin. Deanet: Dense context-aware network for semantic segmentation. arXiv preprint arXiv:2104.02533 2021.
J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
10

---

Under review as a conference paper at ICLR 2022
S. Mehta, M. Rastegari, A. Caspi, L. Shapiro, and H. Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In Proceedings of the european conference on computer vision (ECCV), pages 552-568, 2018.
R. Niu, X. Sun, Y. Tian, W. Diao, Chen, and Fu. Hybrid multiple attention network for semantic segmentation in aerial images. IEEE Transactions Geoscience and Remote Sensing, 2021.
M. Orsic, I. Kreso, P. Bevandic, and S. Segvie. In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 12607-12616, 2019.
A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint arXiv: 1606. 0477, 2016.
R. Poudel, S. Liwicki, and R. Cipolla. Fast-senn: fast semantic segmentation network. arXiv preprint arXiv:1902.044 04502, 2019.
A. Sagar. Aa3dnet: Attention augmented real time 3d object detection. arXiv:2107.12137, 2021a.
arXiv preprint
A. Sagar. Dmsanet: Dual multi scale attention network. arXiv preprint arXiv:2106.08382 2021b.
A. Sagar and R. Soundrapandiyan. Semantic segmentation with multi scale spatial attention for self driving cars, arXiv preprint arXiv:2007.12685, 2020.
M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 6105-6114, PMLR, 2019.
Q Tang, Liu, T. Zhang, Jiang, and Y. Zhang. Attention-guided chained context aggregation for semantic segmentation. arXiv preprint wrXv:22002.12411, 2020.
Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and Cottrell. Understanding convolution for semantic segmentation. In 2018 IEEE winter conference applications of computer vision (WACV), pages 1451-1460. IEEE, 2018.
X. Yang, S. Li, Z. Chen, J. Chanussot, X. Jia, B. Zhang, B. Li, and P. Chen. An attention-fused network for semantic segmentation of very-hight very-high-resolution remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 177:238-2 262, 2021.
C. Yu, Wang, C. Peng, C. Gao, G. real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018a.
Yu, and N. Sang. Bisenet: Bilateral segmentation network for
C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang. Learning a discriminative feature network for semantic segmentation. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1857-1866, 2018b.
C. Yu, C. Gao, J. Wang, G. Yu. C. Shen, and N. Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. arXiv preprint arXiv:2004.02147, 2020.
F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:
2015.
H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal. Context encoding for semantic segmentation. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7151-7160, 2018
Y. Zhang, Z. Qiu, J. Liu, T. Yao, D. Liu, and T. Mei. Customizable architecture search for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11641-16550. 2019.
H. Zhao, J. Shi. X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2881-2890 2017.
11

---

Under review as a conference paper at ICLR 2022
H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia. lenet for real-time semantic segmentation high- resolution images. Proceedings of the European conference on computer vision (ECCV), pages 405-420, 2018a.
H. Zhao, Y. Zhang, S. Liu, Shi, C. C. Loy, D. Lin, and J. Jia. Psanet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 267-283, 2018b.
12


Reviews:
[{"review_id": "kEFkocuT7R", "reviewer": "Reviewer_og3N", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has some issues as follows:\n(1)\tThe novelty is quite limited. This paper proposed an attention based method for semantic segmentation, but the proposed spatial attention module and channel attention module are same with [1], and they are widely used in semantic segmentation. \n(2)\tOrganization. The organization of this paper should be improved. The contribution of this paper should be in the Introduction section, rather than the related work. The introduction of datasets should be the in experiments section, rather than method section. It will be better to put the section of background in related work.\n(3)\tThe motivation of this work is unclear. The authors should clarify the motivation in the introduction section.\n(4)\tThe draws of the existing methods should be analyzed in the related work section.\n(5)\tThe authors claim that atrous convolutions are able to capture boundary information, why?\n(6)\tThe proposed modules are not evaluated in ablation study.\n(7)\tWrong claims: “DUC (Wang et al., 2018), PSPNet (Zhao et al., 2017), DeepLab v2 (Chen et al., 2017) use the dilated convolution…”, “Our network dosen’t use any backbone”\n(8)\tBad representation: “It also uses dense local multi-scale context information using Multi Scale Context (MSC) module.” \n\n[1] Dual Attention Network for Scene Segmentation, CVPR 2019\n", "summary_of_the_paper": "This paper proposed an attention aware network for real time semantic segmentation. Spatial attention and channel attention modules are used in the proposed method, as well as a multi scale context module. The proposed method is evaluated on some public datasets.", "summary_of_the_review": "Mainly because of the weak novelty of the proposed method and poor organization of the paper, I will reject this paper.    ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WpBes0go01Z", "reviewer": "Reviewer_dXYM", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is not well written. It feels like more of the school project report. It doesn’t explain what are the novelties and why the current approach outperforms other competing methods. \n\nThe related works are incomplete and mostly outdated. Between Intro and Related Works there is only a citation to a 2020 segmentation approach. Also, there are many references at the bottom that are not cited in the paper.\n\nThe background section quickly summarizes other segmentation works. However, it does not explain what distinguishes these approaches from the proposed method.\n\nNone of the components like architecture and losses is novel or used in a novel way. The combination thereof is also not novel. \n\nThe analysis lacks comparisons with other approaches like TDNet that is using channel attention.", "summary_of_the_paper": "The paper proposes an approach to perform video semantic segmentation using the established non-local attention mechanism to fuse spatio-temporal features.", "summary_of_the_review": "Reject due to lack of novelty, claims not explained and supported.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "P4KE4ADX7Oj", "reviewer": "Reviewer_2HPE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. The idea to omit heavy backbones is worth exploring and the proposed model manages to achieve a good performance with a lightweight architecture.\n2. The models runs fast and addresses the need of real-time application.\n\nWeakness\n1. The submodules are not well explained. For example, there are addition ops in Fig.2&3, but they only take one input so I don't understand how the addition is done. The reshape in addition to the addition is also unclear.\n2. The motivation of the proposed loss is unclear. \nThe CE loss (Eq. 8) is a binary cross entropy loss, where the groundtruth is \"whether the forward propagation result is true\". I don't quite understand what is \"forward propagation result\" for a semantic segmentation task. \nEq.(9), I believe, is in fact the standard pixel-wise multi-class cross entropy loss for semantic segmentation tasks and the authors call it \"auxiliary\" loss, which I find very confusing. \nEq.(10) is a loss applied to \"the class attention map\", which is not defined in previous sections. I assume it's channel attention from Sec. 4.5. Eq.(10) and Eq.(9) are essentially the same, but applied to the outputs from different modules.  Therefore, from my perspective, the \"class attention map\" can be treated as class predictions of each pixel, but it's called \"attention\", which seems confusing, too.\n3. The ablation studies are poorly designed. \nThe first ablation is about upsampling op, which is not discussed in previous manuscript. I don't even know where those upsampling ops are in the whole architecture. \nI expect the authors to ablate the three components, MSC, SA, CA, so that we can understand the contribution from each module, but only some detailed designs, such as kernel sizes and dilation rates, are ablated. In Table 6, the baseline architecture is unclear.\nI also expect the authors to oblate the losses, e.g. what if there's no CE loss (Eq.8).\n4. Comparing with previous real-time methods, the performance gain is marginal.\n5. Limited novelty. Attention-based modules and skipping heavy backbones are not new for segmentation tasks.", "summary_of_the_paper": "This paper proposes a real-time semantic segmentation. To save computation, there is no heavy backbone to extract features but just simple block containing conv/relu/bn. Then there three modules applied in parallel on top of the image features: multi-scale context module (conv with different dilation rate), spatial attention module and channel attention module.\nExperiments were done on three datasets and the proposed method can achieve a reasonable performance with higher FPS than previous methods.", "summary_of_the_review": "The paper made some contribution by proposing a lightweight architecture and achieving real-time inference. \nHowever, the design is not well explained (figures and texts sometimes don't match). The experiments didn't explain the function of each module and focused on too-low-level ops. The performance gain and speed gain are marginal comparing with other real-time methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ICw8ulgSWs4", "reviewer": "Reviewer_kwfp", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths:\n\n(1) The network is simple and effective. The achieved trade-off between accuracy and inference speed is impressive.\n\n2. Weakness:\n\n(1)  The motivation is not clear. The introduction is so short, which didn't illustrate the clear motivation. \n\n(2) The novelty is limited. The attention module is over-explored in this community. The proposed network has nothing new. The main structures of Fig.2 and Fig.3 has no difference with DANet. The multi-scale context module is like the DenseASPP. The simple combination of the off-the-shelf technologies far from reaches the level of ICLR.\n\n(3) The gaps of mIoU and FPS between Tab. 1(test set) and Tab. 6 (validation set) are too large, which are beyond our common sense. Please  give detailed reasons.\n\n(4) In the spatial/channel attention module, Eq. 2 and Eq. 3 indicate that the input features simply passed through two 1x1 convolutions. Why the output shape of Eq. 2 is (HxW)x(HxW)? Why the output shape of Eq. 3 is CxC?\n\n(5) \"x1, x2 and x3 represents the features of the two branches.\" Two branches produce three features?\n\n(6) Please refine the typesetting. Page 7 has too much blank space.\n\n(7) Why the performance on ADE20K dataset is so high? Did you pre-train on the ImageNet dataset?\n\n(8) \"Our network dosen’t use any backbone to extract features from the input image unlike many previous\narchitectures. \" This claim is wrong. What's the meaning of the backbone? Could your network be called a backbone? Maybe you mean the pre-trained model.\n\n(9) The whole organization is poor.\n+ Please give the rigorous formulations for the spatial/context information, attention mechanism, feature fusion. Or move the background part to the related work.\n+ Please summarize different methods in the related work, instead of just enumerating them.\n+ Please move the section 4.1 (dataset part), 4.8 (implementation details), 4.9 (metrics) to the experiment part. They are not related to the method part directly.\n+ The section 4.2 \"network architecture\" doesn't illustrate the whole architecture, just introduce a convolution layer.\n+ Figure 4 can be illustrated in Section 4.2.\n\n(10) The English writing is also poor.\n+ Chaotic tense.\n+ The second paragraph demonstrates the dilated convolution increases the computation complexity, which motivated improving the model speed. However, the third paragraph still introduces the dilated convolution method. The logic doesn't make sense. \n+ The second paragraph introduce that \"(Mehta et al., 2018), (Paszke et al., 2016) and (Poudel et al., 2019) proposed unique approaches\". What are the unique approaches? Please summarize them.\n+ \"and ON ADE20K dataset\" in the second contribution\n+ The \"use\" is USED too much in the whole manuscript, which can be replaced by \"perform\", \"conduct\", \"apply\", \"adapt\", etc.\n+ The symbol of \"l\" in Line 4, Paragraph 1, Section 4.2 is not correct.\nTypos:\n+ Duplicated \"at\" in Line 4 of the second paragraph\n+ The format of the reference of \"Chen et al. (2017) \" in the third paragraph is incorrect.\n+ The comma is required before \"respectively\".\n\n", "summary_of_the_paper": "This manuscript proposes an Attention Aware Network for real-time semantic segmentation. This network is designed from scratch, which contains a spatial attention module, a channel attention module and a multi scale context module. It achieves an impressive tradeoff between accuracy and inference speed on three datasets. However, this manuscript seems like an unfinished tech report. The motivation is not clear. The proposed method is not novel. The details are missing. The writing and organization are poor. There are so many problems in this manuscript.", "summary_of_the_review": "This manuscript needs careful refinement. This version far from reaches the level of ICLR. Maybe the authors can improve it according to the weakness part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended reject, and there were no responses from authors.
Decision: Reject