Enhancing feature fusion with spatial aggregation and channel fusion for semantic segmentation

IET Computer VisionVolume 15, Issue 6 p. 418-427 ORIGINAL RESEARCH PAPER Open Access Enhancing feature fusion with spatial aggregation and channel fusion for semantic segmentation Jie Hu, School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this authorHuifang Kong, Corresponding Author konghuifang@hfut.edu.cn School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, China Correspondence Huifang Kong; School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, China. Email: konghuifang@hfut.edu.cnSearch for more papers by this authorLei Fan, School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this authorJun Zhou, orcid.org/0000-0001-7569-5589 School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this author Jie Hu, School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this authorHuifang Kong, Corresponding Author konghuifang@hfut.edu.cn School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, China Correspondence Huifang Kong; School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, China. Email: konghuifang@hfut.edu.cnSearch for more papers by this authorLei Fan, School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this authorJun Zhou, orcid.org/0000-0001-7569-5589 School of Electrical Engineering and Automation, Hefei University of Technology, Hefei, ChinaSearch for more papers by this author First published: 04 May 2021 https://doi.org/10.1049/cvi2.12026 AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinked InRedditWechat Abstract Semantic segmentation is crucial to the autonomous driving, as an accurate recognition and location of the surrounding scenes can be provided for the street scenes understanding task. Many existing segmentation networks usually fuse high-level and low-level features to boost segmentation performance. However, the simple fusion may impose a limited performance improvement because of the gap between high-level and low-level features. To alleviate this limitation, we respectively propose spatial aggregation and channel fusion to bridge the gap. Our implementation, inspired by the attention mechanism, consists of two steps: (1) Spatial aggregation relies on the proposed pyramid spatial context aggregation module to capture spatial similarities to enhance the spatial representation of high-level features, which is more effective for the latter fusion. (2) Channel fusion relies on the proposed attention-based channel fusion module to weight channel maps on different levels to enhance the fusion. In addition, the complete network with U-shape structure is constructed. A series of ablation experiments are conducted to demonstrate the effectiveness of our designs, and the network achieves mIoU score of 81.4% on Cityscapes test dataset and 84.6% on PASCALVOC 2012 test dataset. 1 INTRODUCTION Recent years, there is an increasing interest in autonomous driving cars and autonomous driving assistance systems. An essential aspect of autonomous driving is to acquire an accurate understanding of the semantic entities usually found in street scenes, such as the cars, pedestrians, road or sidewalks in the Figure 1. Semantic segmentation, a kind of dense pixel-wise prediction task, can provide abundant information (e.g. object category, location and the shape of each element) for comprehensive scene description. Thanks to the excellent scene recognition performance of semantic segmentation, it has become an important tool for the street scenes understanding task in autonomous driving [1, 2]. FIGURE 1Open in figure viewerPowerPoint Illustration of the comprehensive semantic entities in the street scenes Semantic segmentation aims at assigning each pixel in the input image with corresponding semantic labels, which pose strong demand for both semantic and spatial information. The semantic information plays a crucial role in achieving high performance of recognition (i.e. what is visible?) and spatial information is important to boost localization accuracy (i.e. where precisely is something located?). In this study, we pursue the goal of achieving high-quality semantic segmentation by enhancing the correlation between semantic information and spatial information. Currently, based on the pioneering fully convolutional network (FCN) [3] that takes image as input and outputs a probability map for each class, many state-of-the-art segmentation networks [4-19] have achieved remarkable progress. When performing some form of FCN to boost segmentation performance, the U-shape structure [7, 16, 17, 19] is demonstrated to an effective framework to capture information from both semantic and spatial levels. As shown in Figure 2(a), the U-shape network uses one contracting path to compute high-level features for capturing the semantic information and one symmetric path to fuse low-level features for recovering the spatial information. Low-level and high-level features are complementary by nature, where low-level features are rich in spatial details but lack semantic information and vice versa [18]. Although the networks based on U-shape structure can improve segmentation performance, they fuse the features between low-level and high-level by combining channel maps simply. This simple fusion would result in the limited performance improvement because of the gap between low-level and high-level features. Several methods exist to overcome this challenge and obtain remarkable progress. Zhang et al. [18] proposes a new network, named ExFuse, to bridge the gap between low-level features and high-level features. PAN [15] and DFN [19] adopt high-level features to weight low-level features for selecting precise spatial details. FIGURE 2Open in figure viewerPowerPoint Detailed structure for the U-shape network. EM: encoder module. (a) U-shape network captures context information from both semantic and spatial levels. (b) our network enhances the feature fusion to boost segmentation performance In contrast to above methods, we rethink the feature fusion from a more macroscopic point of view. In this way, we regard the feature fusion as a task to explore the correlation between high-level features and low-level features rather than a simple fusion of channel maps. Considering that the high-level features are rich in semantic information details but lack spatial details and the gap between high-level features and low-level features is mainly caused by the imbalance between semantic information and spatial information, we propose to enhance the feature fusion from two aspects: (1) Spatial aggregation, aggregating more spatial information into the high-level features for enhancing spatial correlation between the high-level features and low-level features, which are more effective for the latter channel fusion. (2) Channel fusion, weighting channel maps on different levels to enhance the feature fusion. Specifically, we construct two attention-based modules for the implementation of spatial aggregation and channel fusion. The attention mechanism [20-24], an important mechanism to select more distinct and informative features, has proven to be an effective method in computer vision tasks. For spatial aggregation, the pyramid spatial context aggregation (PSCA) module is designed, which exploits the attention mechanism to capture the spatial similarities between any two positions. Guided by the spatial similarities, a spatial attention map is generated to update each position of the high-level features. That is to say, spatial representation of high-level features can be enhanced without losing semantic representation. For channel fusion, we build a so-called attention-based channel fusion (ACF) module to generate a weight vector for changing the weights of channel maps on each level, through which low-level and high-level features can be adaptively fused. This effective feature fusion can provide rich feature representations from both semantic and spatial levels for the final prediction. Based on PSCA and ACF modules, we build our segmentation network following the U-shape structure, as shown in Figure 2(b). In summary, the main contributions of our study include three-fold: We introduce the spatial aggregation and channel fusion to emphasize the spatial correlation and channel interdependence between low-level and high-level features, respectively for enhancing the feature fusion. We propose two attention-based modules (PSCA and ACF modules) for spatial aggregation and channel fusion, and design a network based on U-shape structure. We conduct a series of ablation experiments to demonstrate the effectiveness of our method. And our network achieves comparable performance to some state-of-the-art works. The remainder of this study is organized as follows. In Section 2, we review some works that have far-reaching effects for semantic segmentation. Section 3 presents our proposed modules and network. In Section 4, we conduct ablation experiments. In Section 5, the conclusion is drawn. 2 RELATED WORKS 2.1 Semantic segmentation The last years, semantic segmentation has achieved great progress with FCN [3], which was the earliest approach to introduce full convolution into semantic segmentation. Later, many networks based on FCN were proposed to capture multiscale information to alleviate the problem caused by multiscale objects. PSPNet [12] designed a pyramid pooling module to collect the effective contextual prior, containing information of different scales. DenseASPP [14] brought dense connections into atrous spatial pyramid pooling (ASPP) to generate features with various scale. To further enhance feature representation, some works exploited contextual dependencies to generate dense and pixel-wise contextual information. PSANet [26] learnt to aggregate contextual information for each position via a predicted attention map. Liu et al. [27] and Visin et al. [28] utilized RNNs to capture long-range contextual dependency information. In addition, the encoder-decoder framework was widely used in existing works to capture context information from both semantic and spatial levels. For instance, RefineNet [7], U-net [16], SegNet [29] and ParseNet [30] utilized multipath and symmetrical subnetworks (U-shape structure) to fuse the low-level feature for refining prediction results. ENet [31] and ERFNet [32] explored more efficient encoder-decoder framework and multiple optimization methods for semantic segmentation task. Motivated by above methods, our approach investigates the spatial details in high-level features to enhance contextual dependencies and utilizes the multipath and symmetrical subnetworks to adaptively fuse low-level feature and high-level feature. 2.2 Attention mechanism In computer vision tasks, attention mechanism can be considered as an important mechanism to weight features for selecting more distinct and informative features. SENet [33] enhanced the representational power of the network by modelling channel-wise relationship, which obtained significant improvement on image classification. Woo et al. [34] established a convolutional block attention module to select feature maps along with channel and spatial axes respectively. Wang et al. [35] proposed the nonlocal neural network to calculate the context at one position as a weighted sum of all positions for guiding dense contextual information aggregation. GCNet [36] simplified the nonlocal network and abstracted this simplified version to a global context modelling framework, which can effectively model long-range dependency. For scene understanding, attention mechanism, as a selector not only was used to select discriminative features, but also was adopted to model spatial dependencies and channel dependencies. PAN [15] and DFN [19] suggested that high-level features can be used as guidance to weight low-level features for selecting precise spatial details. EncNet [37] introduced an encoding layer to capture the semantic information. OCNet [38] proposed an object context pooling (OCP) to represent each pixel by exploiting the set of pixels that belong to the same object category. CCNet [39] proposed the criss-cross attention module on the criss-cross path to aggregate contextual information. DANet [40] explored long-range dependencies over local features across spatial dimensions and channel dimensions simultaneously. 3 METHOD In this section, we present the details of our study. First, we specifically introduce the two attention-based modules for spatial aggregation and channel fusion. Then, we present the complete architecture of our network, which adopts U-shape structure. 3.1 Spatial aggregation To encode a wider range of spatial information into high-level features, spatial aggregation is introduced with considering two crucial factors simultaneously: global context dependence and multilevel features. Specifically, the spatial context aggregation (SCA) block is designed to capture global context dependence over the spatial locations, and the PSCA module is further proposed to exploit SCA block to scan the whole image over both positions and pyramid levels. Next, we elaborate the implementation details of SCA block and PSCA module. 3.1.1 Spatial context aggregation block The intuition of spatial aggregation is to utilize the spatial similarities to guide global contextual information aggregation, which helps enhance the spatial representation of the high-level features. As illustrated in Figure 3(a), given a local feature X, we denote x = { x i } i = 1 N p as the feature map of one input image, where Np = H×W is the number of positions in the feature map. We first feed X into a convolution layer with batch normalization and ReLU layers to generate two feature spaces f and g. After that we perform a matrix multiplication between the transpose of f(xi) and g(xj), and apply a softmax layer to calculate the spatial attention mapS: s j i = exp ( β i j ) ∑ i = 1 N p exp ( β i j ) , where β i j = f ( x i ) T g ( x j ) (1) FIGURE 3Open in figure viewerPowerPoint The details of spatial context aggregation (SCA) block and pyramid spatial context aggregation (PSCA) module are illustrated in (a) and (b) where the spatial attention map Sji is a representation vector for the global position relationship, βij indicates the extent to which the module attends to the ith position when synthesizing the jth position, f() is the query-position transform function and g() is the key-position transform function. Note that the more similar spatial representations of the two positions ith and jth, the greater correlation between them. Finally, we construct the spatial aggregation for each key-position by performing a matrix multiplication between the spatial attention mapS and the original feature map, as below: y j = ∑ i = 1 N p s j i × h ( x i ) (2)where h() is the value transform function following the self-attention. Therefore, the final feature representations achieve mutual compensation. 3.1.2 Pyramid spatial context aggregation module As illustrated in Figure 3(b), we further implement SCA for the feature maps at multiple levels respectively by introducing a feature pyramid (FP) [41], which has been heavily used in the era of object detection. Then we construct the PSCA module, which involves a bottom-up pathway, lateral connections and a top-down pathway. The bottom-up pathway is the feed forward computation based on the backbone ResNet-101 [42], which takes a single-scale image as input and computes a pyramidal hierarchy consisting of feature maps at several levels with the pooling operation. For the FP, we denote the outputs of each pyramid level as (P1, P2, P3, P4), which have the resolution of (32×32, 16×16, 8×8, 4×4). We choose the outputs of each pyramid level as our reference set of feature maps, which are integrated to the lateral connections for enriching spatial context information. The lateral connections utilize the SCA block to compute spatial aggregation for the feature maps at different pyramidal hierarchies, respectively, which generate a multilevel feature representation. This design is natural since the higher levels of FP should have the stronger features. Note that we do not apply SCA block toP1 due to its large memory footprint. The top-down pathway is an extended path symmetrical to the bottom-up pathway, feature maps from higher pyramid levels and then merges feature maps of the same spatial size from each lateral connection. This process of up-sampling and merging is iterated until the finest resolution maps from P1 are merged. We append a 3×3 convolution on each merged map for reducing the aliasing effect of up-sampling. We fix the feature dimension in all the feature maps, and the final convolutional layer has 512-channel outputs. Benefit from the simple design, PSCA module can aggregate spatial context information at different semantic levels, which strengthen the spatial representation for the high-level features without losing the semantic representation. 3.2 Channel fusion Channel maps located in low-level and high-level feature maps can be regarded as the response of spatial and semantic information, respectively, and different responses are associated with each other. Based on the spatial aggregation proposed above, we exploit the interdependent channel maps on different levels to enhance the feature fusion. Specifically, the ACF module is proposed to generate a weight vector for changing the weights of channel maps between high-level and low-level feature maps, through which low-level and high-level features can be adaptively fused. In general, a common way of feature fusion in the U-shape structure is to formulate as a residual form: y l = Upsample ( y l + 1 ) + x l (3)where yl is the obtained feature by fusing l-th level feature, yl + 1 stands for the l + 1-th feature generated by the encoder module, which has higher semantic but lower spatial information, and xl is the feature extracted from the backbone, which has higher spatial resolution but lower semantic level. Equation (3) implicitly indicates that the fusion weights between channels of different levels are equal, which is considered to be simple feature fusion. However, due to the features in different levels have different degrees of discrimination, simple feature fusion results in the semantic and spatial gap between low-level and high-level features. Therefore, we build the ACF module to explicitly model interdependencies between channels for enhancing the channel fusion by changing the weights of the feature maps on each levels, which are inspired by the attention mechanism [15, 19]. As illustrated in Figure 4, two input feature maps of ACF module are denoted as high-level features Fh and low-level features Fl respectively, and we directly calculate the feature maps Ff by the simple feature fusion of Fh and Fl. Then, the feature maps Ff pass through the channel attention block to generate channel attention vector Mc. Specifically, we exploit channel dependencies by applying both average and max pooling operators on channel feature maps Ff, producing two different channel description vectors: V max c and V a v e c . Both vectors are then forwarded to a shared network composed of multilayer perceptron (MLP) with one hidden layer. After the shared network is applied to each descriptor, we merge two output attention vectors using element-wise summation to generate our channel attention vector Mc, which is calculated as Equation (4): M c = δ ( MLP ( 1 s 2 ∑ i , m ∈ [ i , i + s ] Ρ ( F f ) a v e ) ⊕ MLP ( max i , m ∈ [ i , i + s ] Ρ ( F f ) max ) ) (4)where δ is the sigmoid activation function, ⊕ is element-wise addition, Ρ denotes the pooling function and s is the total number of pixels in feature maps Ff. Finally, Mc is used to weight the channel maps of high level and low level to obtain final output Fo, as given in Equation (5): F o = F f × M c (5)With this design, the ACF module is used to emphasize interdependent features stage-wise to enhance channel fusion, which is vital to bridge the gap between low-level and high-level features. FIGURE 4Open in figure viewerPowerPoint Attention-based channel fusion (ACF) module structure 3.3 Network architecture With the PSCA and ACF modules, we design the complete network, which is given in Figure 5. Our network based on the U-shape structure consists of two paths, one contracting path to compute the high-level features, and the other symmetric expanding path to fuse low-level features of adjacent stages. FIGURE 5Open in figure viewerPowerPoint Overview of the proposed network for semantic segmentation First, in the contraction path, we use pretrained ResNet-101 [42] as our backbone, similar to the PAN [15]. The ResNet-101 is widely used to extract the low-level features. Our approach feeds the input image to the ResNet-101, outputting a feature map X of size W×H×C. In detail, we divide the ResNet-101 into five blocks Res2 to Res5. In order to retain more details, we remove the last two down-sampling operations and use atrous convolutions [48] in the Res4 and Res5 blocks, thus enlarging the feature map size of the Res5 to one-eighth of the input image. Next, the feature maps from Res5 are fed into our proposed PSCA module. The PSCA module aggregates spatial context information, which spans level from low to high. Through this path, more spatial information is embedded into the high-level features without losing semantic information, which is better for the latter channel fusion. Then, in the symmetric expanding path, the proposed ACF module is used to adaptively fuse high-level and low-level features from Res2 to Res4 layer by layer. In this process, attention mechanism is used to change the weights of channel maps on each level for enhancing the channel fusion. Finally, through the U-shape structure, we get our final prediction map. With the support of this network, the feature fusion of the low-level and high-level features is enhanced to boost the segmentation performance. 4 EXPERIMENT In this section, we carried out comprehensive experiments on Cityscapes dataset [25] and PASCAL VOC 2012 [50] to evaluate the proposed method. First, we introduced the dataset and implementation details, then the contributions of each module were investigated in ablation experiments. Finally, we performed comprehensive experiments on Cityscapes and PASCAL VOC 2012 benchmark and the test results are submitted to a dedicated evaluation server for comparison to other methods. For quantitative evaluation, mean of class-wise Intersection over Union (mIoU) are used. 4.1 Experimental settings 4.1.1 Cityscapes dataset The Cityscapes dataset [25] is composed of a large, diverse set of high resolution (2048 × 1024) images recorded in street scenes, where 5000 of these images have high-quality pixel-level labels of 19 classes and results 9.43 × 109 labelled pixels in total. High quality images include 2975 images for training, 500 images for validation, as well as 1525 images for testing. 4.1.2 PASCAL VOC 2012 dataset This PASCAL VOC 2012 [50] benchmark contains 20 foreground object classes and one background class. The original dataset contains 1464 (training), 1449 (validation) and 1456 (testing) pixel-level annotated images. 4.1.3 Implementation details We implemented our experiments on the public platform PyTorch [43] with four NVIDIA GTX1080ti cards. For data augmentation, common data augmentations were used as preprocessing, including mean subtraction, random flipping horizontally and random scaling in the range of [0.5, 2]. For training, following the previous studies [30, 44], we used the 'poly' learning rate strategy where the learning rate is shown in Equation (6): l r = l r b a s e ∗ ( 1 − e p o c h max e p o c h ) p o w e r (6)where lrbase denoted the base learning rate, power denoted decayed index and max_epoch denoted the number of total epochs. Specifically, we empirically set lrbase to 0.01, power to 0.9 and weight decay to 0.0001. The training was performed by using minibatch stochastic gradient descent [45] with batch size eight and momentum 0.99. We used 60 K training iterations. 4.2 Ablation studies for attention-based modules We used the ResNet-101 as our baseline, and directly up-sampled the output. First, we evaluated the performance of the baseline on the validation set, as shown in Table 1. Then we extended baseline to U-shape structure mentioned in Section 1 with our proposed PSCA and ACF modules, and conducted a set of ablation experiments for the two modules. Finally, we experimented the complete network. TABLE 1. Performance of ablation studies on Cityscapes val set Method Setting mIoU (%) Road Sidewalk Building Wall Fence Pole Traffic light Traffic sign Vegetation Terrain Sky Person Rider Car Truck Bus Train Motorcycle Bicycle Baseline 71.05 97.9 82.4 90.0 52.5 51.2 54.2 58.7 70.2 90.3 65.1 92.1 77.5 48.9 92.2 63.2 73.3 65.2 55.1 70.1 PSCA SCA 74.89 98.3 83.5 91.4 48.4 53.5 55.2 66..2 74.4 92.6 70.9 94.5 82.3 57.9 94.3 68.3 77.6 71.3 61.1 71.0 FP 73.23 98.1 83.7 92.3 50.1 53.2 57.2 59.2 69.1 92.4 70.2 94.2 77.2 54.1 93.7 67.9 79.2 67.2 59.1 70.7 FP&SCA 76.65 98.5 85.4 92.7 49.1 52.9 59.1 71.2 76.3 93.1 71.2 95.3 84.3 67.2 95.4 67.4 80.9 68.4 63.1 72.2 ACF NOR 77.01 98.6 86.1 92.8 57.2 54.1 63.2 70.8 76.8 93.4 72.2 95.4 84.9 67.8 95.7 68.6 77.5 69.4 65.3 74.5 AVE 78.23 98.6 86.6 93.0 58.9 56.9 67.2 74.3 78.9 93.6 73.1 95.3 86.2 69.7 96.1 65.8 82.1 76.2 66.7 75.7 MAX 78.72 98.5 86.4 92.8 57.3 59.7 63.8 73.2 77.2 93.2 72.7 95.5 85.3 70.1 95.7 75.4 84.1 75.1 67.2 74.2 AVE + MAX 79.91 98.5 86.6 93.1 59.1 61.1 67.2 76.5 77.3 93.3 72.3 95.2 86.0 70.5 96.5 74.1 90.6 77.1 67.4 75.7 (AVE + MAX)&MLP 81.16 98.6 86.8 93.4 61.9 60.7 68.5 75.8 78.9 93.7 73.5 95.6 86.9 71.6 97.9 78.9 92.1 77.2 68.7 75.5 Network (PSCA&ACF) 81.31 98.7 87.5 93.5 61.6 62.6 66.7 77.5 78.6 93.7 73.7 95.4 86.2 72.5 97.8 81.2 92.8 78.6 70.8 75.3 4.2.1 Ablation studies for PSCA module Based on the observation in Section 3.2, we performed different settings for PSCA module to verify the validity of each component. As shown the second row block in Table 1, using SCA or FP alone yielded a result of 74.89% and 73.23% in mIoU, respectively. When we integrated SCA and FP together by embedding SCA into each level of the FP, the performance further was improved to 76.65%, which was an obvious improvement comparing with the baseline. Noted that, FP&SCA significantly outperformed the baseline in recognizing the objects with diverse scales, for example the 'rider' and 'person'. 4.2.2 Ablation studies for ACF module As shown the third row block in Table 1, we set five experiments (i.e. NOR, AVE, MAX, AVE + MAX and (AVE + MAX)&MLP) to explore the best design for ACF module. Noted that the five experiments denoted different designs used to generate the channel attention vector. NOR: the simple fusion, which obtained 77.01% mIoU. AVE: using global average pooling to generate channel attention vector, which obtained 78.23% mIoU. MAX: using global max pooling to generate channel attention vector, which obtained 78.72% mIoU. AVE + MAX: combing global max and average pooling to generate channel attention vector, which obtained 79.91% mIoU. (AVE + MAX)&MLP: combing global max and average pooling to generate channel attention vector firstly and using MLP to generate the final attention vector, which obtained 81.16% mIoU. Comparing with NOR, four other designs all obtained performance improvement. Because the global average and max pooling can utilize global information to guide the channel fusion. We experimentally verified that utilizing MLP helps integrate the attention vector from AVE and MAX, which can bring superior performance improvement for the ACF module. So we adopted (AVE + MAX)&MLP in our final module. 4.2.3 Complete network Combined our best setting on PSCA and ACF modules, we experimented the complete network. In evaluation, we applied the multiscale inputs (with scales {0.5, 0.75, 1.0, 1.5, 1.75, 2.0}) and also left-right flipped the images. Finally, our network achieved performance of 81.3% in mIoU without MS-COCO [46]. More performance details were listed in the fourth row block of Table 1. 4.3 Comparison with state-of-the-art methods In t