Year: 2021

Abstract:
In this paper, we present a new network named Attention Aware Network (AASeg) for real time semantic image segmentation. Our network incorporates spatial and channel information using Spatial Attention (SA) and Channel Attention (CA) modules respectively. It also uses dense local multi-scale context information using Multi Scale Context (MSC) module. The feature maps are concatenated individually to produce the final segmentation map. We demonstrate the effectiveness of our method using a comprehensive analysis, quantitative experimental results and ablation study using Cityscapes, ADE20K and Camvid datasets. Our network performs better than most previous architectures with a 74.4% Mean IOU on Cityscapes test dataset while running at 202.7 FPS.

heading: 1 INTRODUCTION
text: Semantic Segmentation is a fundamental problem in computer vision where the goal is to label each and every pixel in the image to its appropriate class. Since it is required to be deployed in real world settings like robots and autonomous vehicles, hence there is a need to balance the speed vs performance tradeoff. Fully Convolutional Network (FCN) (Long et al., 2015) composed of convolutional layers was one of the first works to get strong semantic representation.
However, this method was not able to capture boundary information accurately. Atrous convolutions (Yu and Koltun, 2015) at the last several stages of their network was used to give feature maps with strong semantic representation, thus solving the problem with FCN based architectures. However, this comes at at the cost of increased computational complexity. For real world navigation tasks like autonomous driving, there is a need to improve the Frame Per Second(FPS).
Chen et al. (2017) and (Yu and Koltun, 2015) used dilated convolutions to increase the recpetive field while maintaining the number of parameters. SegNet (Badrinarayanan et al., 2017) utilizes a small network structure and the skip-connected method to achieve improved FPS. (Mehta et al., 2018), (Paszke et al., 2016) and (Poudel et al., 2019) proposed unique approaches to tackle real time semantic segmentation problem.

heading: 2 RELATED WORK
text: (Fu et al., 2019) introduces spatial-wise and channelwise attention modules to enhance the recpetive field (Paszke et al., 2016) trims a a lot of convolution filters to reduce computation. ICNet (Zhao et al., 2018a) proposed an image cascade network using multiresolution branches. (Iandola et al., 2016) allows the neural network to find the critical channels of the feature map and select the most suitable channels by itself.
ESPNet (Mehta et al., 2018) introduces an efficient spatial pyramid (ESP), which brings great improvement in both speed and performance. Bilateral Segmentation Network (BiSeNet) (Yu et al., 2018a) used two parts: Spatial Path (SP) is used to get with the loss of spatial information and Context Path (CP) for compressing the receptive field. (Yu et al., 2020) used multi-path framework to combine the low-level details and high-level semantics.
(Li et al., 2019b) utilizes a light-weight backbone to speed up its network and a multi scale feature aggregation to improve accuracy. SwiftNet (Orsic et al., 2019) used lateral connections to restore
the prediction resolution while maintaining the speed. (Lin et al., 2017) uses a multipath refinement network to refine the feature but ignores the global context feature.
The speed-Accuracy performance comparison of state of the art methods on the Cityscapes test set is shown in Figure 1:
.
We summarize our main contributions as follows:
• We propose a novel Attention Aware Network (AASeg) for real time semantic segmentation.
• Our network is comprised of three parts: Spatial Attention(SA) module to capture the spatial dependencies of the feature maps, Channel Attention(CA) module to extract high level semantic information and Multi Scale Context(MSC) module to learn the information flow between feature maps of consecutive levels.
• Detailed experiments and analysis indicate the efficacy of our proposed network in not only improving the performance but also FPS. We achieve results on par with previous state of the art using Cityscapes, Camvid and on ADE20K datasets.

heading: 3 BACKGROUND

heading: 3.1 SPATIAL INFORMATION
text: The spatial information of the image is important to predict the detailed output for semantic segmentation. Modern existing approaches uses encoding of spatial information. DUC (Wang et al., 2018), PSPNet (Zhao et al., 2017), DeepLab v2 (Chen et al., 2017) use the dilated convolution to preserve the spatial size of the feature map.

heading: 3.2 CONTEXT INFORMATION
text: Semantic segmentation requires context information to generate a high-quality result. Most of the used methods enlarge the receptive field or fuse different contextual information. (Chen et al., 2017), (Wang et al., 2018) and (Yu and Koltun, 2015) use different dilation rates in convolution layers to capture different scale contextual information. In (Chen et al., 2017), an “ASPP” module is used to capture context information of different receptive field. PSPNet (Zhao et al., 2017) applies a “PSP” module which contains several different scales of average pooling layers.

heading: 3.3 ATTENTION MECHANISM
text: (Hu et al., 2018) applied channel attention for image recognition and achieve the state-of-the-art. (Yu et al., 2018b) proposed a network that learns the global context as attention and revise the features. Multi scale network along with a custom attention module was proposed by (Sagar and Soundrapandiyan, 2020) for semantic segmentation.

heading: 3.4 FEATURE FUSION
text: The spatial information captured by the Spatial Path consists of rich detailed information. The output feature of the Context Path is made up of contextual information. Feature fusion was used to achieve state of the art results on image classification, object detection and instance segmentation using DMSANet (Sagar, 2021b).

heading: 4 PROPOSED METHOD

heading: 4.1 DATASET
text: The following datasets have been used to benchmark our results:
1. Cityscapes It is used for urban street segmentation. The 5000 annotated images are used in our experiments which are divided into 2975, 500 and 1525 images for training, validation, and testing respectively.
2. ADE20K This dataset contains labels of 150 object categories. The dataset includes 20k,2k and 3k images for training, validation and testing respectively.
3. CamVid This dataset is used for semantic segmentation for autonomous driving scenarios. It is composed of 701 densely annotated images.

heading: 4.2 NETWORK ARCHITECTURE
text: The fundamental goal in semantic segmentation is to map an RGB image X ∈ RH×W×3 to a semantic map Y ∈ RH×W×C with the same spatial resolution H ×W , where C is the number of classes of objects present in image. The input image X is converted to a set of feature maps Fl where l=1,...,3 from each network stage, where Fl ∈ RHlWlCl is a Cl-dimensional feature map. Our network dosen’t use any backbone to extract features from the input image unlike many previous architectures. The input image is first passed through a block comprising of convolutional, batch normalization and ReLU activation function.
We express a convolution layer Wn(x) as follows:
Wn(x) = Wn×n x+ b (1)
where represents the convolution operator, Wn×n represents the n × n convolutional kernel, x represents the input data and b represents the bias vector.

heading: 4.3 MULTI SCALE CONTEXT MODULE
text: The feature map after passing through convolutional block is split to three different parts with 1×1 convolution, 3×3 convolution and 5×5 convolution respectively. The individual feature maps are then fused together. The output fused feature map is first convolved with a 1 × 1 convolution to reduce the number of channels from 2048 to 256. The feature map produced is of size H ×W × nc, where H,W are the height and width of the feature map, and nc denotes the number of channels.
The input feature map is convolved with dilated convolution layers with increasing dilation rates of 3, 6 and 12. The dilated convolution layer input at every stage is formed by concatenating the input feature map with the outputs from previous convolutions. At the final step, the outputs from the three dilated convolutions are concatenated with the input feature map.

heading: 4.4 SPATIAL ATTENTION MODULE
text: The spatial attention module is used for capturing the spatial dependencies of the feature maps. The spatial attention (SA) module in our network is defined below:
fSA(x) = fsigmoid (W2 (fReLU (W1(x)))) (2)
where W1 and W2 denotes the first and second 1× 1 convolution layer respectively, x denotes the input data, fSigmoid denotes the sigmoid function, fReLU denotes the ReLu activation function.
The spatial attention module used in this work is shown in Figure 2:

heading: 4.5 CHANNEL ATTENTION MODULE
text: The channel attention module is used for extracting high level multi-scale semantic information. The channel attention (CA) module in our network is defined below:
fCA(x) = fsigmoid(W2(fReLU (W1f 1 AvgPool(x)))) (3)
where W1 and W2 denotes the first and second 1 × 1 convolution layer, x denotes the input data. f1AvgPool denotes the global average pooling function, fSigmoid denotes the Sigmoid function, fReLU denotes ReLU activation function.
The channel attention module used in this work is shown in Figure 3:

heading: 4.6 AGGREGATION
text: We denote the concatenation operation as follows:
xconcat = x1 ⊕ x2 ⊕ x3 (4)
where ⊕ represents the concatenation operator and x1, x2 and x3 represents the features of the two branches. The AASeg module can be denoted as follow:
xAASeg = ((fSA (xconcat)⊗ xconcat)⊕ (fCA (xconcat)⊗ xconcat)⊕ (fMSC (xconcat)⊗ xconcat)) (5)
where ⊕ represents the concatenation operator, fCA represents the channel attention module mentioned in Equation 2, fSA represents the spatial attention module, fMSC represents the multi scale attention and xconcat represents the combined feature.
we use ConvXi to denote the operations of ith block. Therefore, the output of ith block is calculated as follows:
xi = ConvXi (xi−1, ki) (6)
where ConvX includes one of each convolutional layer, batch normalization layer and ReLU activation layer, ki is the kernel size of convolutional layer, xi−1 and xi are the input and output of ith block. Fusion operation is used to combine high-level features with low-level features using Equation 7:
xoutput = F (x1, x2, . . . , xn) (7)
The overall structure of our proposed AASeg network is shown in Figure 4.

heading: 4.7 LOSS FUNCTIONS
text: We use cross-entropy loss function to weigh the difference between the forward propagation result of the network and the ground truth of the samples. The cross-entropy loss is calculated as defined in Equation 8:
Lce = 1
N N∑ n=1 [yn log ŷn + (1− yn) log (1− ŷn)] (8)
where N denotes the total number of samples, yn denotes the probability that the forward propagation result is true, and 1-yn denotes the probability that the forward propagation result is false.
We also use the auxiliary supervision loss Laux to improve the model performance thus making it easier to optimize. The auxiliary loss can be defined as:
Laux = − 1
BN B∑ i=1 N∑ j=1 K∑ k=1 I ( gij = k ) log
 exp ( pij,k ) ∑K
m=1 exp ( pij,m
)  (9)
I ( gij = k ) = { 1, gij = k 0, otherwise
(10)
where B is the mini batch size, N is the number of pixels in every batch; K is the number of categories; pijk is the prediction of the jth pixel in the ith sample for the kth class, I(gij = k) is a function which is defined in Equation 10.
The class attention loss Lcls from channel attention module is also used. The class attention loss is defined as follows:
Lcls = − 1
BN B∑ i=1 N∑ j=1 K∑ k=1 I ( gij = k ) log
 exp ( aij,k ) ∑K
m=1 exp ( aij,m
)  (11)
where aijk is the value generated of the class attention map of the jth pixel in the ith sample for the kth class. We combine the three terms to balance final loss term as follows:
L = λ1Lce + λ2Lcls + λ3Laux (12)
where λ1, λ2 and λ3 are set as 1, 0.5 and 0.5 to balance the loss.

heading: 4.8 IMPLEMENTATION DETAILS
text: PyTorch deep learning framework is used to carry out our experiments. Our neural networks is trained with stochastic gradient descent (SGD) as optimizer, batch size of 16, momentum of 0.9 and weight decay of 5e−4. The network is trained for 20K iterations with an initial learning rate value of 0.01. The, “poly” learning rate policy is used to decay the initial learning rate while training the model to reduce over-fitting.
Data augmentation operations like random horizontal flip, random resizing with scale range of [1.0, 2.0], and random cropping with crop size of 1024× 1024 was done.

heading: 4.9 EVALUATION METRICS
text: For quantitative evaluation of the performance of our network, mean of class-wise intersection-overunion (mIoU) is used for bench-marking the performance, number of float-point operations (FLOPs) and frames per second (FPS) are used for bench-marking the speed.

heading: 5 RESULTS
text: We present the segmentation accuracy and inference speed of our proposed method on Cityscapes validation and test set in Table 1. We use the training set and validation set to train our models before submitting to Cityscapes online server.
The qualitative results on Cityscapes validation set are presented in Figure 5:
.
We evaluate our method on CamVid dataset. The comparison results with other methods is shown in Table 2:
The comparison of our network with previous state of the art methods for semantic segmentation using ADE20K validation set is shown in Table 3:

heading: 6 ABLATION STUDY
text: We explore the effect of 3 different upsamling operations: bilinear upsampling, deconvolution and nearest neighbor upsampling as shown in Table 4. The best results are obtained using bilinear upsampling but there isn’t considerable difference.
We also try the various kernel size in Table 5. Larger kernel size of 7× 7 is also tried which results in a similar performance but increases computational complexity.
We use increasing sequence of dilation rates in our network, which is 0, 1, 2, 3. It seems large dilation rates increases the performance but comes at the cost of increased number of parameters, hence decrease in FPS. This effect is shown in Table 6:

heading: 7 CONCLUSIONS
text: In this paper, we present a new network named Attention Aware Network (AASeg) for real time semantic segmentation. We use Spatial Attention(SA) and Channel Attention(CA) modules to enhance features of objects without adding extra computational cost. The fused spatial-channel attention along with Multi Scale Context(MSC) module enables network to extract discriminative and robust features of targets or background. We test our network using Cityscapes, ADE20K and Camvid datasets. Our network balances speed-performance tradeoff at par with previous state of the art network architectures. In the future, we would like to use multi scale attention module for instance segmentation problem.

heading: ACKNOWLEDGMENTS
text: We would like to thank Nvidia for providing the GPUs.

Review summaries:
This manuscript proposes an Attention Aware Network for real-time semantic segmentation. This network is designed from scratch, which contains a spatial attention module, a channel attention module and a multi scale context module. It achieves an impressive tradeoff between accuracy and inference speed on three datasets. However, this manuscript seems like an unfinished tech report. The motivation is not clear. The proposed method is not novel. The details are missing. The writing and organization are poor. There are so many problems in this manuscript.

This paper proposes a real-time semantic segmentation. To save computation, there is no heavy backbone to extract features but just simple block containing conv/relu/bn. Then there three modules applied in parallel on top of the image features: multi-scale context module (conv with different dilation rate), spatial attention module and channel attention module. Experiments were done on three datasets and the proposed method can achieve a reasonable performance with higher FPS than previous methods.

This paper proposed an attention aware network for real time semantic segmentation. Spatial attention and channel attention modules are used in the proposed method, as well as a multi scale context module. The proposed method is evaluated on some public datasets.